Skip to content

GitLab

  • Projects
  • Groups
  • Snippets
  • Help
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
pypy
pypy
  • Project overview
    • Project overview
    • Details
    • Activity
    • Releases
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 594
    • Issues 594
    • List
    • Boards
    • Labels
    • Service Desk
    • Milestones
  • Merge Requests 10
    • Merge Requests 10
  • CI / CD
    • CI / CD
    • Pipelines
    • Jobs
    • Schedules
  • Operations
    • Operations
    • Incidents
    • Environments
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Analytics
    • Analytics
    • CI / CD
    • Repository
    • Value Stream
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Members
    • Members
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar

This instance will be upgraded to the latest Heptapod 0.20rc on 2021-02-26 at 11:00 UTC+1

  • PyPy
  • pypypypy
  • Issues
  • #3348

Closed
Open
Opened Nov 24, 2020 by Přemysl Karbula@smuuf

PyPy 7.3.2+ (Python 3.6) weird unicode/utf-8 file reading/decoding problem

Pypy 7.3.2 and 7.3.3 seems to be dropping valid characters when reading utf-8 files - but pypy 7.3.1 has no problem with them.

  • Affected PyPy versions:
    • 7.3.3 has this weird behavior.
    • 7.3.2 has this weird behavior.
    • 7.3.1 is OK.
  • OS versions:
    • Reproduced on WSL1 Ubuntu 18.04 (Windows build 20262).
    • Reproduced on ordinary native Ubuntu 20.04.

Steps to Reproduce:

  • Download pypy_utf_error.zip
  • Run python parsetest.py (python is pointing at pypy3 binary inside active venv) with both pypy 7.3.1 and 7.3.3 and see these results:

Update

Seems like for certain environments (native Linux?) a larger sample XML file is needed for this problem to reproduce.

  • pypy_utf_error_larger_sample.zip

pypy 7.3.3|7.3.2 (dropping valid characters)

...
█ XML ITEM 14
- ITEM_ID: 'item_id ěščřžýáíé' (length 17)
- PRODUCTNAME: 'productname éíáýžřčšě' (length 21)
- DESCRIPTION: 'description íáýžřčšě nebo ěščřžýáíé' (length 35)
...

pypy 7.3.1 (OK)

...
█ XML ITEM 14
- ITEM_ID: 'item_id ěščřžýáíé' (length 17)
- PRODUCTNAME: 'productname éíáýžřčšě' (length 21)
- DESCRIPTION: 'description éíáýžřčšě nebo ěščřžýáíé' (length 36)
...

Different errors parameter values.

# For XML ITEM 14:
# 1/ errors='ignore'
#   DESCRIPTION: 'description íáýžřčšě nebo ěščřžýáíé' (length 35)
# 2/ errors='strict'
#   DESCRIPTION: 'description éíáýžřčšě nebo ěščřžýáíé' (length 36)
# 3/ errors='replace'
#   DESCRIPTION: 'description ��íáýžřčšě nebo ěščřžýáíé' (length 37)
buffered = open(filename, 'rb')
buffered = TextIOWrapper(buffered, encoding='UTF-8', errors='ignore')

The problem seems to go away when TextIOWrapper's parameter errors=strict, but the behavior is still different between 7.3.1 and 7.3.3.

Edited Nov 24, 2020 by Přemysl Karbula
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Assignee
Assign to
None
Milestone
None
Assign milestone
Time tracking
None
Due date
None
Reference: pypy/pypy#3348