Skip to content
Snippets Groups Projects
  1. Nov 21, 2022
  2. Nov 07, 2022
  3. Nov 23, 2022
    • Pierre-Yves David's avatar
      delta-find: set the default candidate chunk size to 10 · 523cacdfd324
      Pierre-Yves David authored
      I ran performance and storage tests on repositories of various sizes and shapes
      for the following values of the config : 5, 10, 20, 50, 100, no-chunking
      
      The performance tests do not show any statistical impact on computation
      times for large pushes and pulls.
      
      For searching for an individual delta, this can provide a significant
      performance improvement with a minor degradation of space-quality on the
      result. (see data at the end of the commit).
      
      For overall store size, the change :
      - does not have any impact on many small repositories,
      - has an observable, but very negligible impact on most larger repositories.
      - One private repository we use for testing sees a small increase in size
        (1%) in the narrower version.
        We will try to get more numbers on a larger version of that repository to
        make sure nothing pathological happens.
      
      We pick "10" as the limit as "5" seems a bit more risky.
      
      There are room to improve the current code, by using more aggressive filtering
      and better (i.e any) sorting of the candidates. However this is already a large
      improvement for pathological cases, with little impact in the common
      situations.
      
      The initial motivation for this change is to fix performance of delta
      computation for a file where the previous code ended up testing 20 000 possible
      candidate-bases in one go, which is… slow. This affected about ½ of the file
      revisions leading to atrocious performance, especially during some push/pull
      operations.
      
      
      Details about individual delta finding timing:
      ----------------------------------------------
      
      The vast majority of benchmark cases are unchanged but the three below. The first
      two do not see any impact on the final delta. The last one sees a change in
      delta-size that is negligible compared to the full text size.
      
      
      ### data-env-vars.name     = mozilla-try-2019-02-18-zstd-sparse-revlog
        # benchmark.name         = perf-delta-find
        # benchmark.variants.rev = manifest-snapshot-many-tries-a (revision 756096)
      ∞:   5.844783
      5:   4.473523  (-23.46%)
      10:  4.970053  (-14.97%)
      20:  5.770386   (-1.27%)
      50   5.821358
      100: 5.834887
      
      MANIFESTLOG: rev = 756096: (no-limit)
          delta-base = 301840
          search-rounds = 6
          try-count = 60
          delta-type = snapshot
          snap-depth = 7
          delta-size = 179
      
      MANIFESTLOG: rev=756096: (limit = 10)
          delta-base=301840
          search-rounds=9
          try-count=51
          delta-type=snapshot
          snap-depth=7
          delta-size=179
      
      ### data-env-vars.name     = mozilla-try-2019-02-18-zstd-sparse-revlog
        # benchmark.name         = perf-delta-find
        # benchmark.variants.rev = manifest-snapshot-many-tries-d (revision 754060)
      ∞:   5.017663
      5:   3.655931  (-27.14%)
      10:  4.095436  (-18.38%)
      20:  4.828949   (-3.76%)
      50   4.987574
      100: 4.994889
      
      MANIFESTLOG: rev=754060: (no limit)
          delta-base=301840
          search-rounds=5
          try-count=53
          delta-type=snapshot
          snap-depth=7
          delta-size = 179
      
      MANIFESTLOG: rev=754060: (limite = 10)
          delta-base=301840
          search-rounds=8
          try-count=45
          delta-type=snapshot
          snap-depth=7
          delta-size = 179
      
      ### data-env-vars.name     = mozilla-try-2019-02-18-zstd-sparse-revlog
        # benchmark.name         = perf-delta-find
        # bin-env-vars.hg.flavor = rust
        # benchmark.variants.rev = manifest-snapshot-many-tries-e (revision 693368)
      ∞:   4.869282
      5:   2.039732  (-58.11%)
      10:  2.413537  (-50.43%)
      20:  4.449639   (-8.62%)
      50   4.865863
      100: 4.882649
      
      MANIFESTLOG: rev=693368:
          delta-base=693336
          search-rounds=6
          try-count=53
          delta-type=snapshot
          snap-depth=6
          full-test-size=131065
          delta-size=199
      
      MANIFESTLOG: rev=693368:
          delta-base=278023
          search-rounds=5
          try-count=21
          delta-type=snapshot
          snap-depth=4
          full-test-size=131065
          delta-size=278
      
      
      Raw data for store size (in bytes) for various chunk size value below:
      ----------------------------------------------------------------------
      
      
         440 134 384          5 pypy/.hg/store/
         440 134 384         10 pypy/.hg/store/
         440 134 384         20 pypy/.hg/store/
         440 134 384         50 pypy/.hg/store/
         440 134 384        100 pypy/.hg/store/
         440 134 384        ... pypy/.hg/store/
      
         666 987 471          5 netbsd-xsrc-2022-11-15/.hg/store/
         666 987 471         10 netbsd-xsrc-2022-11-15/.hg/store/
         666 987 471         20 netbsd-xsrc-2022-11-15/.hg/store/
         666 987 471         50 netbsd-xsrc-2022-11-15/.hg/store/
         666 987 471        100 netbsd-xsrc-2022-11-15/.hg/store/
         666 987 471        ... netbsd-xsrc-2022-11-15/.hg/store/
      
         852 844 884          5 netbsd-pkgsrc-2022-11-15/.hg/store/
         852 844 884         10 netbsd-pkgsrc-2022-11-15/.hg/store/
         852 844 884         20 netbsd-pkgsrc-2022-11-15/.hg/store/
         852 844 884         50 netbsd-pkgsrc-2022-11-15/.hg/store/
         852 844 884        100 netbsd-pkgsrc-2022-11-15/.hg/store/
         852 844 884        ... netbsd-pkgsrc-2022-11-15/.hg/store/
      
       1 504 227 981           5 netbeans-2018-08-01-sparse-zstd/.hg/store/
       1 504 227 871         10 netbeans-2018-08-01-sparse-zstd/.hg/store/
       1 504 227 813         20 netbeans-2018-08-01-sparse-zstd/.hg/store/
       1 504 227 813         50 netbeans-2018-08-01-sparse-zstd/.hg/store/
       1 504 227 813        100 netbeans-2018-08-01-sparse-zstd/.hg/store/
       1 504 227 813        ... netbeans-2018-08-01-sparse-zstd/.hg/store/
      
       3 875 801 068          5 netbsd-src-2022-11-15/.hg/store/
       3 875 696 767         10 netbsd-src-2022-11-15/.hg/store/
       3 875 696 757         20 netbsd-src-2022-11-15/.hg/store/
       3 875 696 653         50 netbsd-src-2022-11-15/.hg/store/
       3 875 696 653        100 netbsd-src-2022-11-15/.hg/store/
       3 875 696 653        ... netbsd-src-2022-11-15/.hg/store/
      
       4 531 441 314          5 mozilla-central/.hg/store/
       4 531 435 157         10 mozilla-central/.hg/store/
       4 531 432 045         20 mozilla-central/.hg/store/
       4 531 429 119         50 mozilla-central/.hg/store/
       4 531 429 119        100 mozilla-central/.hg/store/
       4 531 429 119        ... mozilla-central/.hg/store/
      
       4 875 861 390          5 mozilla-unified/.hg/store/
       4 875 855 155         10 mozilla-unified/.hg/store/
       4 875 852 027         20 mozilla-unified/.hg/store/
       4 875 848 851         50 mozilla-unified/.hg/store/
       4 875 848 851        100 mozilla-unified/.hg/store/
       4 875 848 851        ... mozilla-unified/.hg/store/
      
      11 498 764 601          5 mozilla-try/.hg/store/
      11 497 968 858         10 mozilla-try/.hg/store/
      11 497 958 730         20 mozilla-try/.hg/store/
      11 497 927 156         50 mozilla-try/.hg/store/
      11 497 925 963        100 mozilla-try/.hg/store/
      11 497 923 428        ... mozilla-try/.hg/store/
      
      10 047 914 031          5 private-repo
       9 969 132 101         10 private-repo
       9 944 745 015         20 private-repo
       9 939 756 703         50 private-repo
       9 939 833 016        100 private-repo
       9 939 822 035        ... private-repo
      523cacdfd324
  4. Nov 06, 2022
  5. Nov 24, 2022
    • Matt Harbison's avatar
      setup: include vendored 3rd party type stubs · 54421ef8a423
      Matt Harbison authored
      While pytype may not support PEP 561, PyCharm does, so having the stubs
      available means it can determine `foo = attr.ib(type=int)` means `foo` is an
      int.  This only applies when using Mercurial as a library, like with TortoiseHg
      development- PyCharm is already smart enough to use the *.pyi files in the
      Mercurial source tree when hacking on Mercurial itself.
      
      I left the mercurial.cext stubs out because it seems very low level, that 3rd
      parties shouldn't be using directly.
      54421ef8a423
    • Matt Harbison's avatar
      ci: bump pytype to 2022.11.18 · 86e398a3d598
      Matt Harbison authored
      No particular reason, other than the current build is fairly old.  It flagged a
      few more things (that weren't errors based on the logic around them), but OTOH,
      some of the pyi stubs it generates are less specific.
      86e398a3d598
    • Matt Harbison's avatar
      ci: run the script to add vendored type stubs to typeshed · 556efeea6dbf
      Matt Harbison authored
      Since CI runs from docker images, it doesn't matter that this mucks with the
      typeshed bundled with pytype.
      556efeea6dbf
    • Matt Harbison's avatar
      contrib: add a script for adding vendored type stubs to typeshed · 03792c1ed341
      Matt Harbison authored
      I really hate this, but pytype doesn't support PEP 561 and doesn't seem to have
      the equivalent of `MYPYPATH` to point to custom stubs.  Ignoring the vendored
      stubs isn't necessarily harmful, but pytype has been choking on the vendored
      attr package after pytype 2022.03.29 with errors like this:
      
          File "/mnt/c/Users/Matt/hg/mercurial/linelog.py", line 52, in __iter__: Built-in function iter was called with the wrong arguments [wrong-arg-types]
                   Expected: (collection: bytearray)
            Actually passed: (collection: mercurial.thirdparty.attr._make._CountingAttr)
      
          File "/mnt/c/Users/Matt/hg/mercurial/dirstateutils/v2.py", line 143, in pack: Built-in function len was called with the wrong arguments [wrong-arg-types]
                   Expected: (obj: Sized)
            Actually passed: (obj: mercurial.thirdparty.attr._make._CountingAttr)
            Attributes of protocol Sized are not implemented on mercurial.thirdparty.attr._make._CountingAttr: __len__
          File "/mnt/c/Users/Matt/hg/mercurial/dirstateutils/v2.py", line 144, in pack: No attribute 'rfind' on mercurial.thirdparty.attr._make._CountingAttr [attribute-error]
          File "/mnt/c/Users/Matt/hg/mercurial/dirstateutils/v2.py", line 146, in pack: Built-in function len was called with the wrong arguments [wrong-arg-types]
                   Expected: (obj: Sized)
            Actually passed: (obj: mercurial.thirdparty.attr._make._CountingAttr)
            Attributes of protocol Sized are not implemented on mercurial.thirdparty.attr._make._CountingAttr: __len__
          File "/mnt/c/Users/Matt/hg/mercurial/dirstateutils/v2.py", line 152, in pack: No attribute 'v2_data' on mercurial.thirdparty.attr._make._CountingAttr [attribute-error]
      
          File "/mnt/c/Users/Matt/hg/mercurial/util.py", line 2817, in go: unsupported operand type(s) for /: 'count: mercurial.thirdparty.attr._make._CountingAttr' and 'float: float' [unsupported-operands]
            No attribute '__truediv__' on 'count: mercurial.thirdparty.attr._make._CountingAttr' or '__rtruediv__' on 'float: float'
          Called from (traceback):
            line 2981, in __bytes__
      
      This is essentially the same hack we've been using in TortoiseHg to add the
      vendored PyQt5 stubs.  What I don't understand is pytype *still* generates *.pyi
      files under .pytype/pyi/mercurial/thirdparty/attr, even when the package is
      explicitly ignored in the pytype command line args.  But it avoids the errors,
      which means we aren't stuck on pytype==2022.03.29.
      
      https://github.com/google/pytype/issues/151
      03792c1ed341
    • Matt Harbison's avatar
      contrib: update check-pytype.sh to list stubs that caused pytype to crash · 3e1869751cfe
      Matt Harbison authored
      The same logic is in the TortoiseHg tests for running pytype, and it's useful to
      know if a new version of pytype is better or worse.
      3e1869751cfe
  6. Nov 23, 2022
  7. Nov 21, 2022
    • Matt Harbison's avatar
      typing: add type hints to mercurial/help.py · 2a70d1fc70c4
      Matt Harbison authored
      Was hoping to find more issues like f09bc2ed9100, but it may be that nothing
      checks the args to that operation.  In any event, the work is done and pytype
      doesn't do a very good job inferring the types.  A few of th emore complicated
      things like the command table are left untyped, because they come from modules
      that aren't typed yet.
      2a70d1fc70c4
    • Matt Harbison's avatar
      configitems: add a default value for "merge-tools.xxx.regappend" · 5744ceeb9067
      Matt Harbison authored
      When trying to figure out how `hg help -v` took the Set interpolation path in
      f09bc2ed9100, I turned on devel warnings and noticed this (unrelated) warning:
      
          devel-warn: specifying a mismatched default value for a registered config
              item: 'merge-tools.beyondcompare4.regappend' ''
              at: c:\Users\Matt\hg\mercurial\filemerge.py:46 (_toolstr)
      
      The previous default value for this config was `None`, but that slightly
      complicates the code at the only site it is used, referenced above.
      5744ceeb9067
    • Matt Harbison's avatar
      attr: vendor 22.1.0 · e1c586b9a43c
      Matt Harbison authored
      The previous version was 5 years old, and pytype 2022.06.30 started complaining
      about various uses (e.g. seeing `mercurial.thirdparty.attr._make._CountingAttr`
      instead of `bytearray`).  Hopefully this helps.  Additionally, this has official
      python 3.11 support.
      
      The `attrs` package is left out, because it is simply a bunch of *.pyi stubs and
      `from attr.X import *`, and that's not how they've been used up to this point.
      We'd probably need to customize those anyway to
      `from mercurial.thirdparty.attr import *`.
      e1c586b9a43c
    • Matt Harbison's avatar
      tests: update test-util.py for modern attrs package · 7e6f3c69c0fb
      Matt Harbison authored
      When updating to 22.1.0, this test started failing:
      
          Traceback (most recent call last):
            File "/tmp/mercurial-ci/tests/test-util.py", line 53, in <module>
              _start_default = (util.timedcmstats.start.default, 'factory')
          AttributeError: type object 'timedcmstats' has no attribute 'start'
      
      Poking around in `hg debugshell`, the attribute is indeed missing, but looks to
      be attached to `__attrs_attrs__` in both the currently vendored and the modern
      version of attrs.  The old attrs packages will print the same for both accesses,
      so fingers crossed...
      
          >>> print((util.timedcmstats.start.default, 'factory'))
          (Factory(factory=<function timedcmstats.<lambda> at 0x000001EFDF0F21F0>, takes_self=False), 'factory')
      
          >>> print((util.timedcmstats.__attrs_attrs__.start.default, 'factory'))
          (Factory(factory=<function timedcmstats.<lambda> at 0x000001EFDF0F21F0>, takes_self=False), 'factory')
      7e6f3c69c0fb
  8. Nov 15, 2022
  9. Nov 14, 2022
  10. Oct 20, 2022
  11. Nov 19, 2022
    • Raphaël Gomès's avatar
      branching: merge stable into default · 7577b8644558
      Raphaël Gomès authored
      7577b8644558
    • Raphaël Gomès's avatar
      Added signature for changeset c890d8b8bc59 · 3324f39460e5
      Raphaël Gomès authored
      3324f39460e5
    • Raphaël Gomès's avatar
      Added tag 6.3.1 for changeset c890d8b8bc59 · 010d9e362ee4
      Raphaël Gomès authored
      010d9e362ee4
    • Raphaël Gomès's avatar
      relnotes: add 6.3.1 · c890d8b8bc59
      Raphaël Gomès authored
      6.3.1
      c890d8b8bc59
    • Pierre-Yves David's avatar
      tests: fix test-sparse-revlog · da636e7a6d63
      Pierre-Yves David authored
      This one is not covered by the CIbecause I requires an expensive artifact to be
      cached. So it goes out of think on regular basis (we should fix that…)
      
      The test ouput was affected by e706bb41fdb3 as we filtering now happens sooner,
      removing for the output.
      da636e7a6d63
    • Pierre-Yves David's avatar
      memory-usage: fix `hg log --follow --rev R F` space complexity · dcb2581e33be
      Pierre-Yves David authored
      When running `hg log --follow --rev REVS FILES`, the log code will walk the
      history of all FILES starting from the file revisions that exists in each REVS.
      
      Before doing so, it looks if the files actually exists in the target revisions.
      To do so, it opens the manifest of each revision in REVS to look up if we find
      the associated items in FILES.
      
      Before this changeset this was done in a way that created a changectx for
      each target revision, keeping them in memory while we look into each file.
      
      If the set of REVS is large, this means keeping the manifest for each entry in
      REVS in memory. That can be large… if REV is in the form `::X`, this can quickly
      become huge and saturate the memory. We have seen usage allocating 2GB per
      second until memory runs out.
      
      So this changeset invert the two loop so that only one revision is kept in
      memory during the operation. This solve the memory explosion issue.
      dcb2581e33be
  12. Nov 18, 2022
  13. Nov 17, 2022
Loading