PyPy 7.3.2+ (Python 3.6) weird unicode/utf-8 file reading/decoding problem
Pypy 7.3.2
and 7.3.3
seems to be dropping valid characters when reading utf-8
files - but pypy 7.3.1
has no problem with them.
- Affected PyPy versions:
-
7.3.3
has this weird behavior. -
7.3.2
has this weird behavior. -
7.3.1
is OK.
-
- OS versions:
- Reproduced on WSL1
Ubuntu 18.04
(Windows build 20262). - Reproduced on ordinary native
Ubuntu 20.04
.
- Reproduced on WSL1
Steps to Reproduce:
- Download pypy_utf_error.zip
- Run
python parsetest.py
(python
is pointing atpypy3
binary inside active venv) with both pypy7.3.1
and7.3.3
and see these results:
Update
Seems like for certain environments (native Linux?) a larger sample XML file is needed for this problem to reproduce.
pypy 7.3.3|7.3.2 (dropping valid characters)
...
█ XML ITEM 14
- ITEM_ID: 'item_id ěščřžýáíé' (length 17)
- PRODUCTNAME: 'productname éíáýžřčšě' (length 21)
- DESCRIPTION: 'description íáýžřčšě nebo ěščřžýáíé' (length 35)
...
pypy 7.3.1 (OK)
...
█ XML ITEM 14
- ITEM_ID: 'item_id ěščřžýáíé' (length 17)
- PRODUCTNAME: 'productname éíáýžřčšě' (length 21)
- DESCRIPTION: 'description éíáýžřčšě nebo ěščřžýáíé' (length 36)
...
errors
parameter values.
Different # For XML ITEM 14:
# 1/ errors='ignore'
# DESCRIPTION: 'description íáýžřčšě nebo ěščřžýáíé' (length 35)
# 2/ errors='strict'
# DESCRIPTION: 'description éíáýžřčšě nebo ěščřžýáíé' (length 36)
# 3/ errors='replace'
# DESCRIPTION: 'description ��íáýžřčšě nebo ěščřžýáíé' (length 37)
buffered = open(filename, 'rb')
buffered = TextIOWrapper(buffered, encoding='UTF-8', errors='ignore')
The problem seems to go away when TextIOWrapper
's parameter errors=strict
, but the behavior is still different between 7.3.1
and 7.3.3
.
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information