Skip to content
Snippets Groups Projects
  1. Nov 02, 2022
  2. Oct 31, 2022
  3. Jun 08, 2022
  4. Jun 02, 2022
  5. Mar 09, 2022
  6. Feb 21, 2022
  7. Mar 03, 2022
  8. Mar 25, 2021
  9. Oct 22, 2020
    • Matt Harbison's avatar
      demandimport: don't raise AttributeError if `exec_module` is missing · 8ed69bd42f10
      Matt Harbison authored
      I assume this was meant to do the check gracefully.  After shoveling a bunch of
      modules into the ignore list in order to get keyring to work out of the box on
      CentOS 8, I hit the following error accessing the password, which the change
      fixes.  Now the SecretStorage backend works out of the box, without any edits to
      the ignore list.
      
          ** Unknown exception encountered with possibly-broken third-party extension mercurial_keyring
          ** which supports versions unknown of Mercurial.
          ** Please disable mercurial_keyring and try your action again.
          ** If that fixes the bug please report it to https://foss.heptapod.net/mercurial/mercurial_keyring/issues
          ** Python 3.6.8 (default, Apr 16 2020, 01:36:27) [GCC 8.3.1 20191121 (Red Hat 8.3.1-5)]
          ** Mercurial Distributed SCM (version 5.5.2)
          ** Extensions loaded: evolve, topic, rebase, absorb, mercurial_keyring
          Traceback (most recent call last):
            File "/home/mharbison/hg_py3.6.8_venv/lib64/python3.6/site-packages/mercurial_keyring.py", line 230, in _read_password_from_keyring
              password = keyring.get_password(KEYRING_SERVICE, pwdkey)
            File "/home/mharbison/hg_py3.6.8_venv/lib64/python3.6/site-packages/keyring/core.py", line 53, in get_password
              return _keyring_backend.get_password(service_name, username)
            File "/home/mharbison/hg_py3.6.8_venv/lib64/python3.6/site-packages/keyring/backends/chainer.py", line 51, in get_password
              password = keyring.get_password(service, username)
            File "/home/mharbison/hg_py3.6.8_venv/lib64/python3.6/site-packages/keyring/backends/SecretService.py", line 79, in get_password
              return item.get_secret().decode('utf-8')
            File "/home/mharbison/hg_py3.6.8_venv/lib64/python3.6/site-packages/secretstorage/item.py", line 105, in get_secret
              decryptor = Cipher(aes, modes.CBC(aes_iv), default_backend()).decryptor()
            File "/home/mharbison/hg_py3.6.8_venv/lib64/python3.6/site-packages/cryptography/hazmat/backends/__init__.py", line 15, in default_backend
              from cryptography.hazmat.backends.openssl.backend import backend
            File "<frozen importlib._bootstrap>", line 971, in _find_and_load
            File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
            File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
            File "/home/mharbison/hg_py3.6.8_venv/lib64/python3.6/site-packages/hgdemandimport/demandimportpy3.py", line 53, in exec_module
              self.loader.exec_module(module)
            File "/home/mharbison/hg_py3.6.8_venv/lib64/python3.6/site-packages/cryptography/hazmat/backends/openssl/__init__.py", line 7, in <module>
              from cryptography.hazmat.backends.openssl.backend import backend
            File "<frozen importlib._bootstrap>", line 971, in _find_and_load
            File "<frozen importlib._bootstrap>", line 955, in _find_and_load_unlocked
            File "<frozen importlib._bootstrap>", line 665, in _load_unlocked
            File "/home/mharbison/hg_py3.6.8_venv/lib64/python3.6/site-packages/hgdemandimport/demandimportpy3.py", line 53, in exec_module
              self.loader.exec_module(module)
            File "/home/mharbison/hg_py3.6.8_venv/lib64/python3.6/site-packages/cryptography/hazmat/backends/openssl/backend.py", line 14, in <module>
              from six.moves import range
            File "<frozen importlib._bootstrap>", line 971, in _find_and_load
            File "<frozen importlib._bootstrap>", line 951, in _find_and_load_unlocked
            File "<frozen importlib._bootstrap>", line 894, in _find_spec
            File "/home/mharbison/hg_py3.6.8_venv/lib64/python3.6/site-packages/hgdemandimport/demandimportpy3.py", line 117, in find_spec
              and getattr(spec.loader, "exec_module")
          AttributeError: '_SixMetaPathImporter' object has no attribute 'exec_module'
      
      Differential Revision: https://phab.mercurial-scm.org/D9243
      8ed69bd42f10
  10. Sep 29, 2020
  11. Sep 09, 2020
  12. Jul 08, 2020
    • Manuel Jacob's avatar
      demandimport: ignore `lzma` module for demandimport · 40120de810ba
      Manuel Jacob authored
      This makes importing the module fail if the `_lzma` module is not present.
      
      This makes e.g. tarfile correctly recognize if LZMA support is not present. It
      changes the exception
      
          File "/usr/lib/python3.6/tarfile.py", line 1694, in xzopen
            fileobj = lzma.LZMAFile(fileobj or name, mode, preset=preset)
        AttributeError: module 'lzma' has no attribute 'LZMAFile'
      
      to the more correct exception
      
          File "/usr/lib/python3.6/tarfile.py", line 1692, in xzopen
            raise CompressionError("lzma module is not available")
        tarfile.CompressionError: lzma module is not available
      
      Also, it prevents that the error "abort: No module named '_lzma'!" is shown when
      a development warning is to be shown. The reason why that happened is that for
      showing the warning, we get information about the stack frames from the inspect
      module, which accesses the `__file__` attribute of all modules in `sys.modules`
      to build some cache, causing all modules (including `lzma`) to be imported.
      40120de810ba
  13. May 11, 2020
    • Manuel Jacob's avatar
      demandimport: fix compatibility with meta path finders w/o find_spec() method · a6e12d477595
      Manuel Jacob authored
      Meta path finders got a find_spec() method in Python version 3.4. The
      sys.meta_path documentation says that the deprecated find_module() method is
      used as a fallback.
      
      Setuptool’s VendorImporter still doesn’t have the find_spec() method,
      which resulted in a crash e.g. within a virtual environment. For reference, I
      opened an issue for that: https://github.com/pypa/setuptools/issues/2104.
      
      An alternative implementation would have been to implement a wrapper for
      find_module() itself and raise an AttributeError when accessing find_spec() if
      the wrapped finder doesn’t have it.
      a6e12d477595
  14. Jan 21, 2020
    • Gregory Szorc's avatar
      hgdemandimport: apply lazy module loading to sys.meta_path finders · f81c17ec303c
      Gregory Szorc authored
      Python's `sys.meta_path` finders are the primary objects whose job it
      is to find a module at import time. When `import` is called, Python
      iterates objects in this list and calls `o.find_spec(...)` to find
      a `ModuleSpec` (or None if the module couldn't be found by that
      finder). If no meta path finder can find a module, import fails.
      
      One of the default meta path finders is `PathFinder`. Its job is to
      import modules from the filesystem and is probably the most important
      importer. This finder looks at `sys.path` and `sys.path_hooks` to do
      its job.
      
      The `ModuleSpec` returned by `MetaPathImporter.find_spec()` has a
      `loader` attribute, which defines the concrete module loader to use.
      `sys.path_hooks` is a hook point for teaching `PathFinder` to
      instantiate custom loader types.
      
      Previously, we injected a custom `sys.path_hook` that told `PathFinder`
      to wrap the default loaders with a loader that creates a module object
      that is lazy.
      
      This approach worked. But its main limitation was that it only applied
      to the `PathFinder` meta path importer. There are other meta path
      importers that are registered. And in the case of PyOxidizer loading
      modules from memory, `PathFinder` doesn't come into play since
      PyOxidizer's own meta path importer was handling all imports.
      
      This commit changes our approach to lazy module loading by proxying
      all meta path importers. Specifically, we overload the `find_spec()`
      method to swap in a wrapped loader on the `ModuleSpec` before it
      is returned. The end result of this is all meta path importers should
      be lazy.
      
      As much as I would have loved to utilize .__class__ manipulation to
      achieve this, some meta path importers are implemented in C/Rust
      in such a way that they cannot be monkeypatched. This is why we
      use __getattribute__ to define a proxy.
      
      Also, this change could theoretically open us up to regressions in
      meta path importers whose loader is creating module objects which
      can't be monkeypatched. But I'm not aware of any of these in the
      wild. So I think we'll be safe.
      
      According to hyperfine, this change yields a decent startup time win of
      5-6ms:
      
      ```
      Benchmark #1: ~/.pyenv/versions/3.6.10/bin/python ./hg version
        Time (mean ± σ):      86.8 ms ±   0.5 ms    [User: 78.0 ms, System: 8.7 ms]
        Range (min … max):    86.0 ms …  89.1 ms    50 runs
      
        Time (mean ± σ):      81.1 ms ±   2.7 ms    [User: 74.5 ms, System: 6.5 ms]
        Range (min … max):    77.8 ms …  90.5 ms    50 runs
      
      Benchmark #2: ~/.pyenv/versions/3.7.6/bin/python ./hg version
        Time (mean ± σ):      78.9 ms ±   0.6 ms    [User: 70.2 ms, System: 8.7 ms]
        Range (min … max):    78.1 ms …  81.2 ms    50 runs
      
        Time (mean ± σ):      73.4 ms ±   0.6 ms    [User: 65.3 ms, System: 8.0 ms]
        Range (min … max):    72.4 ms …  75.7 ms    50 runs
      
      Benchmark #3: ~/.pyenv/versions/3.8.1/bin/python ./hg version
        Time (mean ± σ):      78.1 ms ±   0.6 ms    [User: 70.2 ms, System: 7.9 ms]
        Range (min … max):    77.4 ms …  80.9 ms    50 runs
      
        Time (mean ± σ):      72.1 ms ±   0.4 ms    [User: 64.4 ms, System: 7.6 ms]
        Range (min … max):    71.4 ms …  74.1 ms    50 runs
      ```
      
      Differential Revision: https://phab.mercurial-scm.org/D7954
      f81c17ec303c
    • Gregory Szorc's avatar
      hgdemandimport: disable on Python 3.5 · c5e0a9b97b8a
      Gregory Szorc authored
      The demand importer functionality isn't working at all on Python 3.5.
      I'm not sure what's wrong.
      
      Since it isn't working, let's disable it completely.
      
      ```
      $ HGRCPATH= hyperfine -w 1 -r 50 -- "~/.pyenv/versions/3.5.9/bin/python ./hg version" \
        "HGDEMANDIMPORT=disable ~/.pyenv/versions/3.5.9/bin/python ./hg version"
      Benchmark #1: ~/.pyenv/versions/3.5.9/bin/python ./hg version
        Time (mean ± σ):     163.7 ms ±   2.2 ms    [User: 148.5 ms, System: 15.7 ms]
        Range (min … max):   161.0 ms … 170.2 ms    50 runs
      
      Benchmark #2: HGDEMANDIMPORT=disable ~/.pyenv/versions/3.5.9/bin/python ./hg version
        Time (mean ± σ):     164.3 ms ±   1.4 ms    [User: 148.2 ms, System: 16.6 ms]
        Range (min … max):   161.4 ms … 169.8 ms    50 runs
      ```
      
      Differential Revision: https://phab.mercurial-scm.org/D7953
      c5e0a9b97b8a
  15. Dec 01, 2019
  16. Nov 08, 2019
    • Augie Fackler's avatar
      cleanup: remove pointless r-prefixes on single-quoted strings · 9f70512ae2cf
      Augie Fackler authored
      This is the promised second step on single-quoted strings. These had
      existed because our source transformer didn't turn r'' into b'', so we
      had tagged some strings as r-strings to get "native" strings on both
      Pythons. Now that the transformer is gone, we can dispense with this
      nonsense.
      
      Methodology:
      
      I ran
      
          hg locate 'set:added() or modified() or clean()' | egrep '.*\.py$'  | xargs egrep --color=never -n  -- \[\^b\]\[\^a-z\]r\'\[\^\'\\\\\]\*\'\[\^\'\
      
      in an emacs grep-mode buffer, and then used a keyboard macro to
      iterate over the results and remove the r prefix as needed.
      
      # skip-blame removing unneeded r prefixes left over from Python 3 migration.
      
      Differential Revision: https://phab.mercurial-scm.org/D7306
      9f70512ae2cf
  17. Nov 07, 2019
    • Augie Fackler's avatar
      cleanup: remove pointless r-prefixes on double-quoted strings · 313e3a279828
      Augie Fackler authored
      This is only double-quoted strings. I'll do single-quoted strings as a
      second step. These had existed because our source transformer didn't
      turn r"" into b"", so we had tagged some strings as r-strings to get
      "native" strings on both Pythons. Now that the transformer is gone, we
      can dispense with this nonsense.
      
      Methodology:
      
      I ran
      
          hg locate 'set:added() or modified() or clean()' | egrep '.*\.py$'  | xargs egrep --color=never -n  -- \[\^a-z\]r\"\[\^\"\\\\\]\*\"\[\^\"\]
      
      in an emacs grep-mode buffer, and then used a keyboard macro to
      iterate over the results and remove the r prefix as needed.
      
      # skip-blame removing unneeded r prefixes left over from Python 3 migration.
      
      Differential Revision: https://phab.mercurial-scm.org/D7305
      313e3a279828
  18. Nov 02, 2019
    • Gregory Szorc's avatar
      demandimportpy3: only use lazy extension loader on Python 3.6+ · 2d31ef3fb494
      Gregory Szorc authored
      There was an inline comment denoting a bug in the lazy extension
      loader on Python 3.5 which prevents it from working there. But the
      code was not conditional on the Python version.
      
      The result of this was a myriad of failures on Python 3.5 due to
      getattr() and friends not working on lazy extension modules.
      
      By making extension modules non-lazy on Python 3.5, we reduce the
      number of test failures from 48 to 22 on that Python version.
      2d31ef3fb494
  19. Oct 06, 2019
  20. Aug 08, 2019
  21. Jun 12, 2019
  22. Sep 01, 2018
  23. Aug 21, 2018
  24. May 06, 2018
    • Gregory Szorc's avatar
      demandimport: make module ignores a set (API) · 670eb4fa1b86
      Gregory Szorc authored
      The list of modules to ignore is used for membership testing. Yet
      it is defined as a list.
      
      Sets are more efficient for membership testing. So this commit converts
      the module list to a set.
      
      Since we took an API hit, I renamed the variable to further clarify
      the change.
      
      This appears to reduce the CPU time for running 300 invocations of
      `hg log -r. -T '{rev}'` on my i7-6700K:
      
      before: 18.64s
      after:  18.44s
      
      Differential Revision: https://phab.mercurial-scm.org/D3440
      670eb4fa1b86
  25. Jan 05, 2018
  26. Aug 22, 2017
  27. Aug 16, 2017
    • Jun Wu's avatar
      demandimport: disable if chg is being used · 3cfc9070245f
      Jun Wu authored
      In chg's case, making modules lazily loaded could actually slow down things
      since chg pre-imports them. Therefore disable demandimport if chg is being
      used.
      
      This is not done by setting `HGDEMANDIMPORT` chg client-side because that
      has side-effects on child processes (hooks, etc).
      
      Differential Revision: https://phab.mercurial-scm.org/D351
      3cfc9070245f
  28. Aug 18, 2017
  29. Jul 16, 2017
  30. Apr 27, 2017
  31. Apr 28, 2017
  32. May 01, 2017
Loading