Skip to content

GitLab

  • Menu
Projects Groups Snippets
    • Loading...
  • Help
    • Help
    • Support
    • Community forum
    • Submit feedback
    • Contribute to GitLab
  • Sign in / Register
  • pypy pypy
  • Project information
    • Project information
    • Activity
    • Labels
    • Members
  • Repository
    • Repository
    • Files
    • Commits
    • Branches
    • Tags
    • Contributors
    • Graph
    • Compare
  • Issues 654
    • Issues 654
    • List
    • Boards
    • Service Desk
    • Milestones
  • Merge requests 13
    • Merge requests 13
  • CI/CD
    • CI/CD
    • Pipelines
    • Jobs
    • Schedules
  • Deployments
    • Deployments
    • Environments
    • Releases
  • Monitor
    • Monitor
    • Incidents
  • Packages & Registries
    • Packages & Registries
    • Container Registry
  • Analytics
    • Analytics
    • Value stream
    • CI/CD
    • Repository
  • Wiki
    • Wiki
  • Snippets
    • Snippets
  • Activity
  • Graph
  • Create a new issue
  • Jobs
  • Commits
  • Issue Boards
Collapse sidebar
  • PyPy
  • pypypypy
  • Issues
  • #2389

Closed
Open
Created Sep 01, 2016 by Bitbucket Importer@bitbucket_importerMaintainer

Different behavior of bytes.decode('utf8', 'custom_replace')

Created originally on Bitbucket by kostialopuhin (Konstantin Lopukhin)

The following program:

import codecs

codecs.register_error('custom_replace', lambda exc: (u'\ufffd', exc.start+1))

s1 = b"WORD\xe3\xab"
print(repr(s1.decode('utf8', 'custom_replace')))
print(repr(s1.decode('utf8', 'replace')))

s2 = b"\xef\xbb\xbfWORD\xe3\xabWORD2"
print(repr(s2.decode('utf8', 'custom_replace')))
print(repr(s2.decode('utf8', 'replace')))

produces different results on CPython 2.7 (I tried 2.7.6 and 2.7.12) and on PyPy 5.4.0:

$ pypy test.py 
u'WORD\ufffd'
u'WORD\ufffd'
u'\ufeffWORD\ufffd\ufffdWORD2'
u'\ufeffWORD\ufffdWORD2'
$ python test.py 
u'WORD\ufffd\ufffd'
u'WORD\ufffd'
u'\ufeffWORD\ufffd\ufffdWORD2'
u'\ufeffWORD\ufffdWORD2'

And I think CPython is more consistent here: with a custom replace function, it replaces each invalid byte with given symbol, but PyPy in some cases does a different thing.

The context: this code is used in w3lib here https://github.com/scrapy/w3lib/blob/v1.14.2/w3lib/encoding.py#L176 (the CPython bug reference might be slightly misleading here) to emulate browser behavior for invalid utf8 handling, and CPython with custom_replace agrees with browser behavior here.

To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information
Assignee
Assign to
Time tracking