Skip to content
Snippets Groups Projects
  1. Dec 15, 2022
    • Matt Harbison's avatar
      windows: drop some py2 registry module importing · 7a80a614
      Matt Harbison authored
      The comment was actually backwards- `winreg` is importable on py3, and is
      already imported by mercurial/windows.py.
      7a80a614
    • Matt Harbison's avatar
      typing: add type hints to the platform specific scm modules · 7a414342
      Matt Harbison authored
      Surprisingly, pytype struggled to figure out the return types in the posix
      functions.
      7a414342
    • Matt Harbison's avatar
      typing: add type hints to most mercurial/pycompat.py functions · c5a06cc3
      Matt Harbison authored
      The `rapply` methods are left out because it's not `rapply(f, xs: _T0) -> _T0`
      as I first thought- it's used somewhere to walk a collection and convert between
      bytes and str.
      
      Also, the `open()` call is partially untyped because I'm not sure what its
      purpose is at this point- both the name and mode can be either bytes or str as
      it is currently constituted.  It might make sense to assert that the file is
      being opened in binary mode (like `namedtempfile()`) and cast the result to
      `BinaryIO`, but that shouldn't be smuggled in with these other changes.  The
      return is currently typed as `Any` because something suddenly got smarter and a
      few uses in util.py (like readfile()) suddenly think it returns `IO[str]`
      instead of `IO[bytes]` (BinaryIO), and it flags the type mismatch there.
      c5a06cc3
    • Matt Harbison's avatar
      statprof: don't pass str `sys.argv` to a function expecting bytes · 9eb69fa5
      Matt Harbison authored
      Found by typing the global functions in mercurial.pycompat.
      9eb69fa5
    • Matt Harbison's avatar
      typing: drop an unnecessary warning disabling comment in match.py · a06a503e
      Matt Harbison authored
      This stopped being necessary in d2e1dcd4490d, when the exception stopped being
      subscripted.
      a06a503e
    • Matt Harbison's avatar
      scmposix: don't subscript IOError · a87338fe
      Matt Harbison authored
      This warning disabling has been in place since late 2019 in 667f56d73ceb.  We
      should have had some py3 support at the time, but both pytype complains and
      subscripting a real FileNotFoundError generated in `hg debugshell` crashed, so
      maybe this fixes a problem.  It looks like all other instances of subscripting
      exceptions have been replaced (at least as far as greping for `== errno.`
      revealed).
      a87338fe
  2. Dec 14, 2022
    • Matt Harbison's avatar
      typing: add type hints to pycompat.bytestr · 55d45d0d
      Matt Harbison authored
      The problem with leaving pytype to its own devices here was that for functions
      that returned a bytestr, pytype inferred `Union[bytes, int]`.  It now accepts
      that it can be treated as plain bytes.
      
      I wasn't able to figure out the arg type for `__getitem__`- `SupportsIndex`
      (which PyCharm indicated is how the superclass function is typed) got flagged:
      
          File "/mnt/c/Users/Matt/hg/mercurial/pycompat.py", line 236, in __getitem__:
              unsupported operand type(s) for item retrieval: bytestr and SupportsIndex [unsupported-operands]
            Function __getitem__ on bytestr expects int
      
      But some caller got flagged when I marked it as `int`.
      
      There's some minor spillover problems elsewhere- pytype doesn't seem to
      recognize that `bytes.startswith()` can optionally take a 3rd and 4th arg, so
      those few places have the warning disabled.  It also flags where the tar API is
      being abused, but that would be a tricky refactor (and would require typing
      extensions until py3.7 is dropped), so disable those too.
      55d45d0d
    • Matt Harbison's avatar
      pycompat: explicitly prefix builtin attr usage with `builtins.` · f3f33980
      Matt Harbison authored
      It doesn't seem like this would fix any bug, because the wrapped functions that
      take bytes instead of str are defined after these calls.  But PyCharm was
      flagging the second and third uses, saying "Type 'str' doesn't have expected
      attribute 'decode'".  It wasn't flagging the first, but I changed it for
      consistency.
      f3f33980
    • Matt Harbison's avatar
      typing: add type hints to global variables in mercurial/pycompat.py · 9cd32750
      Matt Harbison authored
      The way `osaltsep` and `sysexecutable` were defined, pytype determined them to
      be `Union[bytes, str]`.  This was a problem because that cascaded to all of the
      callers, and also because it couldn't be annotated as bytes on the initial
      assignment.  Therefore, we use a ternary operator.
      
      The documentation says that `sys.executable` can either be None or an empty
      string if the value couldn't be determined.  We opt for an empty string here
      because there are places that blindly pass it to `os.path.xxx()` functions,
      which crash if given None.  Other places test `if pycompat.sysexecutable`, so
      empty string works for both.
      9cd32750
  3. Dec 13, 2022
  4. Dec 12, 2022
  5. Dec 11, 2022
    • Matt Harbison's avatar
      typing: add trivial type hints to mercurial/ui.py · 0449fb77
      Matt Harbison authored
      There's not really a pattern here; it's mostly obvious return types and in a few
      cases, obvious parameter types.  Some other "obvious" functions are left out
      because of quirks in how the return value for the various config() functions are
      inferred cause pytype to complain.
      0449fb77
  6. Dec 10, 2022
  7. Nov 25, 2022
    • Matt Harbison's avatar
      pytype: stop excluding mercurial/ui.py · 8147abc0
      Matt Harbison authored
      ui.extractchoices() is perhaps making assumptions that it shouldn't about the
      pattern always matching, but presumably we have test coverage for that.
      
      PyCharm flags the updated classes with a warning "Class xxx must implement all
      abstract methods", and suggests adding `abc.ABC` to the superclasses.  I'm not
      sure why, unless it doesn't recognize the `__getattr__()` delegation pattern.
      Additionally, we can't unconditionally subclass `typing.BinaryIO` because that
      defeats the `__getattr__` delegation to the wrapped object at runtime.  Instead,
      it has to only subclass during the type checking phase[1].
      
      In any event, this fixes:
      
          File "/mnt/c/Users/Matt/hg/mercurial/ui.py", line 1518, in _runpager:
              Function subprocess.Popen.__new__ was called with the wrong arguments [wrong-arg-types]
                   Expected: (cls, args, bufsize, executable, stdin,
                              stdout: Optional[Union[IO, int]] = ..., ...)
            Actually passed: (cls, args, bufsize, stdin,
                              stdout: Union[mercurial.utils.procutil.WriteAllWrapper,
                                      mercurial.windows.winstdout], ...)
          File "/mnt/c/Users/Matt/hg/mercurial/ui.py", line 1798, in extractchoices:
              No attribute 'group' on None [attribute-error]
            In Optional[Match[bytes]]
          File "/mnt/c/Users/Matt/hg/mercurial/ui.py", line 1799, in extractchoices:
              No attribute 'group' on None [attribute-error]
            In Optional[Match[bytes]]
      
      [1] https://stackoverflow.com/q/71365594
      8147abc0
  8. Dec 07, 2022
    • Pierre-Yves David's avatar
      bundle: emit full snapshot as is, without doing a redelta · e1953a34
      Pierre-Yves David authored
      With the new `forced` delta-reused policy, it become important to be able to
      send full snapshot where full snapshot are needed. Otherwise, the fallback delta
      will simply be used on the client side… creating monstrous delta chain, since
      revision that are meant as a reset of delta-chain chain becoming too complex are
      simply adding a new full delta-tree on the leaf of another one.
      
      In the `non-forced` cases, client process full snapshot from the bundle
      differently from deltas, so client will still try to convert the full snapshot
      into a delta if possible. So this will no lead to pathological storage
      explosion.
      
      I have considered making this configurable, but the impact seems limited enough
      that it does not seems to be worth it. Especially with the current
      sparse-revlog format that use "delta-tree" with multiple level snapshots, full
      snapshot are much less frequent and not that different from other intermediate
      snapshot that we are already sending over the wire anyway.
      
      CPU wise, this will help the bundling side a little as it will not need to
      reconstruct revisions and compute deltas. The unbundling side might save a tiny
      amount of CPU as it won't need to reconstruct the delta-base to reconstruct the
      revision full text. This only slightly visible in some of the benchmarks. And
      have no real impact on most of them.
      
      ### data-env-vars.name      = pypy-2018-08-01-zstd-sparse-revlog
        # benchmark.name          = perf-bundle
        # benchmark.variants.revs = last-40000
      before:          11.467186 seconds
      just-emit-full:  11.190576 seconds   (-2.41%)
      with-pull-force: 11.041091 seconds   (-3.72%)
        # benchmark.name          = perf-unbundle
        # benchmark.variants.revs = last-40000
      before:     16.744862
      just-emit-full:: 16.561036 seconds   (-1.10%)
      with-pull-force: 16.389344 seconds   (-2.12%)
        # benchmark.name          = pull
        # benchmark.variants.revs = last-40000
      before:     26.870569
      just-emit-full:  26.391188 seconds   (-1.78%)
      with-pull-force: 25.633184 seconds   (-4.60%)
      
      
      Space wise (so network-wise) the impact is fairly small. When taking compression into
      account.
      
      Below are tests the size of `hg bundle --all` for a handful of benchmark repositories
      (with bzip, zstd compression and without it)
      
      This show a small increase in the bundle size, but nothing really significant
      except maybe for mozilla-try (+12%) that nobody really pulls large chunk of anyway.
      Mozilla-try is also the repository that benefit the most for not having to
      recompute deltas client size.
      
      ### mercurial:
      bzip-before:     26 406 342 bytes
      bzip-after:      26 691 543 bytes  +1.08%
      
      zstd-before:     27 918 645 bytes
      zstd-after:      28 075 896 bytes  +0.56%
      
      none-before:     98 675 601 bytes
      none-after:     100 411 237 bytes  +1.76%
      
      
      ### pypy
      bzip-before:    201 295 752 bytes
      bzip-after:     209 780 282 bytes  +4.21%
      
      zstd-before:    202 974 795 bytes
      zstd-after:     205 165 780 bytes  +1.08%
      
      none-before:    871 070 261 bytes
      none-after:     993 595 057 bytes +14.07%
      
      
      ### netbeans
      bzip-before:    601 314 330 bytes
      bzip-after:     614 246 241 bytes  +2.15%
      
      zstd-before:    604 745 136 bytes
      zstd-after:     615 497 705 bytes  +1.78%
      
      none-before:  3 338 238 571 bytes
      none-after:   3 439 422 535 bytes  +3.03%
      
      
      ### mozilla-central
      bzip-before:  1 493 006 921 bytes
      bzip-after:   1 549 650 570 bytes  +3.79%
      
      zstd-before:  1 481 910 102 bytes
      zstd-after:   1 513 052 415 bytes  +2.10%
      
      none-before:  6 535 929 910 bytes
      none-after:   7 010 191 342 bytes  +7.26%
      
      
      ### mozilla-try
      bzip-before:  6 583 425 999 bytes
      bzip-after:   7 423 536 928 bytes +12.76%
      
      zstd-before:  6 021 009 212 bytes
      zstd-after:   6 674 922 420 bytes +10.86%
      
      none-before: 22 954 739 558 bytes
      none-after:  26 013 854 771 bytes +13.32%
      e1953a34
  9. Dec 06, 2022
    • Pierre-Yves David's avatar
      bundle: when forcing acceptance of incoming delta also accept snapshot · acdb9a15
      Pierre-Yves David authored
      Snapshot where never considered reusable and the unbundling side always tried
      to find a delta from them. In the `forced` mode this is counter-productive
      because it will either connect two delta-tree that should not be connected or
      it will spend potentially a lot of time because creating a full snapshot
      anyway.
      
      So in this mode, we accept the full snapshot as is.
      
      This changeset is benchmarked with its children so please do not split them
      apart when landing.
      acdb9a15
  10. Dec 07, 2022
  11. Nov 29, 2022
  12. Nov 08, 2022
  13. Dec 03, 2022
  14. Dec 05, 2022
    • Pierre-Yves David's avatar
      delta-find: add a test checking various simple behavior · e300f445
      Pierre-Yves David authored
      There are enough work happening in this area that it is worth having a dedicated
      test for it. So far we add to small test checking that the "best" parent is
      picked as the delta base and that this behavior can be controlled during commit
      and unbundle.
      e300f445
  15. Dec 02, 2022
  16. Dec 03, 2022
  17. Dec 02, 2022
  18. Dec 03, 2022
  19. Dec 02, 2022
Loading