1. 05 Apr, 2019 1 commit
    • Gregory Szorc's avatar
      zstandard: vendor python-zstandard 0.11 · 675775c33ab6
      Gregory Szorc authored
      The upstream source distribution from PyPI was extracted. Unwanted
      files were removed.
      
      The clang-format ignore list was updated to reflect the new source
      of files.
      
      The project contains a vendored copy of zstandard 1.3.8. The old
      version was 1.3.6. This should result in some minor performance wins.
      
      test-check-py3-compat.t was updated to reflect now-passing tests on
      Python 3.8.
      
      Some HTTP tests were updated to reflect new zstd compression output.
      
      # no-check-commit because 3rd party code has different style guidelines
      
      Differential Revision: https://phab.mercurial-scm.org/D6199
      675775c33ab6
  2. 04 Apr, 2019 3 commits
  3. 05 Apr, 2019 4 commits
  4. 25 Dec, 2017 1 commit
  5. 04 Apr, 2019 4 commits
  6. 03 Apr, 2019 1 commit
  7. 19 Jul, 2017 1 commit
  8. 03 Apr, 2019 3 commits
    • Martin von Zweigbergk's avatar
      b9e5f3853a97
    • Augie Fackler's avatar
      py2exe: add workaround to allow bundling of hgext3rd.* extensions · 399ed3e86a49
      Augie Fackler authored
      py2exe doesn't know how to handle namespace packages *at all*, so it treats
      them like normal packages. As a result, if we try and bundle hgext3rd.evolve
      in a py2exe build, it won't work if we install evolve into the virtualenv. In
      order to work around this, tortoisehg installs hgext3rd.evolve etc into its
      staged hg directory, since it doesn't use a virtualenv. As a workaround for us,
      we'll just allow any extra packages users want bundled are part of hg during
      the pseudo-install phase that py2exe uses. I'm not happy about this, but it
      *works*.
      
      As a sample of how you'd make an MSI with evolve bundled:
      
          import os
          import shutil
          import subprocess
          import tempfile
      
          def stage_evolve(version):
            """Stage evolve for inclusion in py2exe binary."""
            with tempfile.TemporaryDirectory() as temp:
              evolve = os.path.join(temp, "evolve")
              subprocess.check_call([
                  "hg.exe",
                  "clone",
                  "https://www.mercurial-scm.org/repo/evolve/",
                  "--update",
                  version,
                  evolve,
              ])
              dest = os.path.join('..', 'hgext3rd', 'evolve')
              if os.path.exists(dest):
                  shutil.rmtree(dest)
              shutil.copytree(os.path.join(evolve, "hgext3rd", "evolve"), dest)
      
      
          def main():
            stage_evolve('tip')
            print("\0")
            print("hgext3rd")
            print("hgext3rd.evolve")
            print("hgext3rd.evolve.hack")
            print("hgext3rd.evolve.thirdparty")
      
      
          if __name__ == "__main__":
            main()
      
      is a script you can pass to the wix/build.py as --extra-packages-script,
      and the resulting .msi will have an hg binary with evolve baked in. users
      will still need to enable evolve in their hgrc, so you'd probably also
      want to bundle configs in your msi for an enterprise environment, but that's
      already easy to do with the support for extra features and wxs files in the
      wix build process.
      
      Differential Revision: https://phab.mercurial-scm.org/D6189
      399ed3e86a49
    • Augie Fackler's avatar
      wix: fix the package build when not adding features · 429bed755ce1
      Augie Fackler authored
      Should have used ifdef, not if. Sigh.
      
      Differential Revision: https://phab.mercurial-scm.org/D6187
      429bed755ce1
  9. 02 Apr, 2019 1 commit
  10. 30 Mar, 2019 1 commit
    • Matt Harbison's avatar
      uncommit: abort if an explicitly given file cannot be uncommitted (BC) · f4147ca63d39
      Matt Harbison authored
      I've gotten burned several times by this in the last few days.  The former tests
      look simple enough, but if a good file and a bad file are given, the bad files
      are silently ignored.  Some commands like `forget` will warn about bogus files,
      but that would likely get lost in the noise of an interactive uncommit.  The
      commit command aborts if a bad file is given, so this seems more consistent for
      commands that alter the repository.
      f4147ca63d39
  11. 25 Mar, 2019 1 commit
    • Navaneeth Suresh's avatar
      unshelve: disable unshelve during merge (issue5123) · 03f6480bfdda
      Navaneeth Suresh authored
      As stated in the issue5123, unshelve can destroy the second parent of
      the context when tried to unshelve with an uncommitted merge. This
      patch makes unshelve to abort when called with an uncommitted merge.
      
      See how shelve.mergefiles works. Commit structure looks like this:
      ```
      ... -> pctx -> tmpwctx -> shelvectx
                        /
                       /
              second
              merge parent
      
      pctx = parent before merging working context(first merge parent)
      tmpwctx = commited working directory after merge(with two parents)
      shelvectx = shelved context
      ```
      shelve.mergefiles first updates to pctx then it reverts shelvectx to pctx with:
      
      ```
      cmdutil.revert(ui, repo, shelvectx, repo.dirstate.parents(),
                             *pathtofiles(repo, files),
                             **{'no_backup': True})
      ```
      
      Reverting tmpwctx files that were merged from second parent to pctx makes them
      added because they are not in pctx.
      
      Changing this revert operation is crucial to restore parents after unshelve.
      This is a complicated issue as this is not fixing a regression. Thus, for the
      time being, unshelve during an uncommitted merge can be aborted.
      
      (Details taken from http://mercurial.808500.n3.nabble.com/PATCH-V3-shelve-restore-parents-after-unshelve-issue5123-tt4036858.html#a4037408)
      
      Differential Revision: https://phab.mercurial-scm.org/D6169
      03f6480bfdda
  12. 02 Apr, 2019 1 commit
  13. 01 Apr, 2019 1 commit
  14. 20 Mar, 2019 1 commit
    • Augie Fackler's avatar
      wix: add a hook for a prebuild script to inject extra libraries · 715d3220ac4f
      Augie Fackler authored
      I need this to build packages for Google so we can bundle some
      extensions in the installed image. My assumption is that this is most
      interesting for the .msi images so I only wired it up there. I'm not
      thrilled with the interface this provides, but it was an easy way to
      retain debug messages on Windows while also having enough structure to
      know what lines are actually module names for py2exe.
      
      Still pending on my end: I need to bundle a couple of config files,
      and at least one data file. I'm open to advice on how to do those
      things, and how to do this better.
      
      Differential Revision: https://phab.mercurial-scm.org/D6164
      715d3220ac4f
  15. 27 Mar, 2019 1 commit
    • Pierre-Yves David's avatar
      compression: introduce an official `format.revlog-compression` option · 4ee906aa7b60
      Pierre-Yves David authored
      This option supersedes the `experiment.format.compression` option. The value
      currently supported are zlib (default) and zstd (if Mercurial was compiled with
      zstd support).
      
      The option gained an explicit reference to `revlog` since this is the target
      usage here. Different storage methods might require different compression
      strategies.
      
      In our tests, using zstd give a significant CPU usage improvement (both
      compression and decompressing) while keeping similar repository size.
      
      Zstd as other interresting mode (dictionnary, pre-text, etc…) that are probably
      worth exploring. However, just plain switching from zlib to zstd provide a large
      benefit.
      4ee906aa7b60
  16. 02 Apr, 2019 1 commit
    • Pierre-Yves David's avatar
      compression: display compression level in debugformat · d7e751ec679e
      Pierre-Yves David authored
      Now that we have options to control the compression level, we teach `hg
      debugformat` about them. This is a useful information when comparing
      repositories.
      
      Note that we have no trace of the compression level used to store existing
      deltas. Actually, it would even varies from one delta to another. So we display
      the currently set value.
      d7e751ec679e
  17. 27 Mar, 2019 4 commits
    • Pierre-Yves David's avatar
      compression: introduce a `storage.revlog.zstd.level` configuration · bb271ec2fbfb
      Pierre-Yves David authored
      This option control the zstd compression level used when compressing revlog
      chunk. The usage of zstd for revlog compression has not graduated from
      experimental yet, but we intend to fix that soon.
      
      The option name for the compression level is more straight forward to pick, so
      this changesets comes first.  Having a dedicated option for each compression
      engine is useful because they don't support the same range of values.
      
      I ran the same measurement as for the zlib compression level (in the parent
      changesets). The variation in repository size is stay mostly in the same (small)
      range. The "read/write" performance see smallish variation, but are overall much
      better than zlib. Write performance show the same tend of having better write
      performance for when reaching high-end compression.
      
      Again, we don't intend to change the default zstd compression level (currently:
      3) in this series. However this is worth investigating in the future.
      
      The Performance comparison of zlib vs zstd is quite impressive. The repository
      size stay in the same range, but the performance are much better in all
      situations.
      
      Comparison summary
      ==================
      
      We are looking at:
      - performance range for zlib
      - performance range for zstd
      - comparison of default zstd (level-3) to default zlib (level 6)
      - comparison of the slowest zstd time to the fastest zlib time
      
      Read performance:
      -----------------
                |           zlib          |           zstd          | cmp | f2s
      mercurial |   0.170159 -   0.189219 |   0.144127 -   0.149624 | 80% | 88%
      pypy      |   2.679217 -   2.768691 |   1.532317 -   1.705044 | 60% | 63%
      netbeans  | 122.477027 - 141.620281 |  72.996346 -  89.731560 | 58% | 73%
      mozilla   | 147.867662 - 170.572118 |  91.700995 - 105.853099 | 56% | 71%
      
      Write performance:
      ------------------
                |           zlib          |           zstd          | cmp | f2s
      mercurial |  53.250304 - 56.2936129 |  40.877025 -  45.677286 | 75% | 86%
      pypy      | 460.721984 - 476.589918 | 270.545409 - 301.002219 | 63% | 65%
      netbeans  | 520.560316 - 715.930400 | 370.356311 - 428.329652 | 55% | 82%
      mozilla   | 739.803002 - 987.056093 | 505.152906 - 591.930683 | 57% | 80%
      
      Raw data
      --------
      
      repo      alg lvl  .hg/store size  00manifest.d read       write
      
      mercurial zlib  1      49,402,813     5,963,475   0.170159  53.250304
      mercurial zlib  6      47,197,397     5,875,730   0.182820  56.264320
      mercurial zlib  9      47,121,596     5,849,781   0.189219  56.293612
      
      mercurial zstd  1      49,737,084     5,966,355   0.144127  40.877025
      mercurial zstd  3      48,961,867     5,895,208   0.146376  42.268142
      mercurial zstd  5      48,200,592     5,938,676   0.149624  43.162875
      mercurial zstd 10      47,833,520     5,913,353   0.145185  44.012489
      mercurial zstd 15      47,314,604     5,728,679   0.147686  45.677286
      mercurial zstd 20      47,330,502     5,830,539   0.145789  45.025407
      mercurial zstd 22      47,330,076     5,830,539   0.143996  44.690460
      
      
      pypy      zlib  1     370,830,572    28,462,425   2.679217 460.721984
      pypy      zlib  6     340,112,317    27,648,747   2.768691 467.537158
      pypy      zlib  9     338,360,736    27,639,003   2.763495 476.589918
      
      pypy      zstd  1     362,377,479    27,916,214   1.532317 270.545409
      pypy      zstd  3     354,137,693    27,905,988   1.686718 294.951509
      pypy      zstd  5     342,640,043    27,655,774   1.705044 301.002219
      pypy      zstd 10     334,224,327    27,164,493   1.567287 285.186239
      pypy      zstd 15     329,000,363    26,645,965   1.637729 299.561332
      pypy      zstd 20     324,534,039    26,199,547   1.526813 302.149827
      pypy      zstd 22     324,530,595    26,198,932   1.525718 307.821218
      
      
      netbeans  zlib  1   1,281,847,810   165,495,457 122.477027 520.560316
      netbeans  zlib  6   1,205,284,353   159,161,207 139.876147 715.930400
      netbeans  zlib  9   1,197,135,671   155,034,586 141.620281 678.297064
      
      netbeans  zstd  1   1,259,581,737   160,840,613  72.996346 370.356311
      netbeans  zstd  3   1,232,978,122   157,691,551  81.622317 396.733087
      netbeans  zstd  5   1,208,034,075   160,246,880  83.080549 364.342626
      netbeans  zstd 10   1,188,624,176   156,083,417  79.323935 403.594602
      netbeans  zstd 15   1,176,973,589   153,859,477  89.731560 428.329652
      netbeans  zstd 20   1,162,958,258   151,147,535  82.842667 392.335349
      netbeans  zstd 22   1,162,707,029   151,150,220  82.565695 402.840655
      
      
      mozilla   zlib  1   2,775,497,186   298,527,987 147.867662 751.263721
      mozilla   zlib  6   2,596,856,420   286,597,671 170.572118 987.056093
      mozilla   zlib  9   2,587,542,494   287,018,264 163.622338 739.803002
      
      mozilla   zstd  1   2,723,159,348   286,617,532  91.700995 570.042751
      mozilla   zstd  3   2,665,055,001   286,152,013  95.240155 561.412805
      mozilla   zstd  5   2,607,819,817   288,060,030 101.978048 505.152906
      mozilla   zstd 10   2,558,761,085   283,967,648 104.113481 497.771202
      mozilla   zstd 15   2,526,216,060   275,581,300 105.853099 591.930683
      mozilla   zstd 20   2,485,114,806   266,478,859  95.268795 576.515389
      mozilla   zstd 22   2,484,869,080   266,456,505  94.429282 572.785537
      bb271ec2fbfb
    • Pierre-Yves David's avatar
      compression: introduce a `storage.revlog.zlib.level` configuration · 1fac9b931d46
      Pierre-Yves David authored
      This option control the zlib compression level used when compression revlog
      chunk.
      
      This is also a good excuse to pave the way for a similar configuration option
      for the zstd compression engine. Having a dedicated option for each compression
      algorithm is useful because they don't support the same range of values.
      
      Using a higher zlib compression impact CPU consumption at compression time, but
      does not directly affected decompression time. However dealing with small
      compressed chunk can directly help decompression and indirectly help other
      revlog logic.
      
      I ran some basic test on repositories using different level. I am using the
      mercurial, pypy, netbeans and mozilla-central clone from our benchmark suite.
      All tested repository use sparse-revlog and got all their delta recomputed.
      
      The different compression level has a small effect on the repository size
      (about 10% variation in the total range). My quick analysis is that revlog
      mostly store small delta, that are not affected by the compression level much.
      So the variation probably mostly comes from better compression of the snapshots
      revisions, and snapshot revision only represent a small portion of the
      repository content.
      
      I also made some basic timings measurements. The "read" timings are gathered using
      simple run of `hg perfrevlogrevisions`, the "write" timings using `hg
      perfrevlogwrite` (restricted to the last 5000 revisions for netbeans and
      mozilla central). The timings are gathered on a generic machine, (not one  of
      our performance locked machine), so small variation might not be meaningful.
      However large trend remains relevant.
      
      Keep in mind that these numbers are not pure compression/decompression time.
      They also involve the full revlog logic. In particular the difference in chunk
      size has an impact on the delta chain structure, affecting performance when
      writing or reading them.
      
      On read/write performance, the compression level has a bigger impact.
      Counter-intuitively, the higher compression levels improve "write" performance
      for the large repositories in our tested setting. Maybe because the last 5000
      delta chain end up having a very different shape in this specific spot? Or maybe
      because of a more general trend of better delta chains thanks to the smaller
      chunk and snapshot.
      
      This series does not intend to change the default compression level. However,
      these result call for a deeper analysis of this performance difference in the
      future.
      
      Full data
      =========
      
      repo   level  .hg/store size  00manifest.d read       write
      ----------------------------------------------------------------
      mercurial  1      49,402,813     5,963,475   0.170159  53.250304
      mercurial  6      47,197,397     5,875,730   0.182820  56.264320
      mercurial  9      47,121,596     5,849,781   0.189219  56.293612
      
      pypy       1     370,830,572    28,462,425   2.679217 460.721984
      pypy       6     340,112,317    27,648,747   2.768691 467.537158
      pypy       9     338,360,736    27,639,003   2.763495 476.589918
      
      netbeans   1   1,281,847,810   165,495,457 122.477027 520.560316
      netbeans   6   1,205,284,353   159,161,207 139.876147 715.930400
      netbeans   9   1,197,135,671   155,034,586 141.620281 678.297064
      
      mozilla    1   2,775,497,186   298,527,987 147.867662 751.263721
      mozilla    6   2,596,856,420   286,597,671 170.572118 987.056093
      mozilla    9   2,587,542,494   287,018,264 163.622338 739.803002
      1fac9b931d46
    • Pierre-Yves David's avatar
      compression: accept level management for zlib compression · aaececb4b066
      Pierre-Yves David authored
      We update the zlib related class to be support setting the compression level.
      This changeset focus on updating the internal only. A way to configure this
      level will be introduced in the next changeset.
      aaececb4b066
    • Pierre-Yves David's avatar
      util: extract compression code in `mercurial.utils.compression` · 3e47d1ec9da5
      Pierre-Yves David authored
      The code seems large enough to be worth extracting. This is similar to what was
      done for various module in `mercurial/utils/`.
      
      Since None of the compression logic takes a `ui` objet, issuing deprecation
      warning is tricky. Luckly the logic does not seems to have many external users.
      3e47d1ec9da5
  18. 30 Mar, 2019 1 commit
  19. 31 Mar, 2019 1 commit
  20. 29 Mar, 2019 2 commits
    • Martin von Zweigbergk's avatar
      shelve: let cmdutil.revert() take care of backing up untracked files · e08559769bc6
      Martin von Zweigbergk authored
      cmdutil.revert() backs up untracked files, so I don't see a reason to
      do it shelve.mergefiles(). We have tests for this and they still pass.
      
      Differential Revision: https://phab.mercurial-scm.org/D6174
      e08559769bc6
    • Martin von Zweigbergk's avatar
      shelve: stop passing list of files to revert · 4606585549b1
      Martin von Zweigbergk authored
      It seems to work just fine to not specify any files here. I suspect it
      looked the way it did for historical reasons. It apparently used to
      use merge instead of rebase until 1d7a36ff2615 (shelve: use rebase
      instead of merge (issue4068), 2013-10-23) and it makes sense to want
      to restrict the set of files then.
      
      I noticed this because of the files.extend(shelvectx.p1().files()). If
      the working copy was clean before, then shelvectx.p1() will be the
      working copy parent and that ended up adding all the files in that
      set. In our Google-internal Mercurial setup (including a FUSE) that
      was very noticeably slow when the working copy parent happened to have
      many files in large directories.
      
      This patch doesn't yet remove the call to shelvectx.p1().files(). We
      also use that set for deciding what to back up. I'm pretty sure it's
      safe to back up only the set of files we already back even if we no
      longer restrict the set of files to revert, so this patch should be
      safe on its own. Regardless, the next patch will delegate the
      backing-up to cmdutil.revert().
      
      Incidentally, this also gets rid of a repo.pathto() that I had earlier
      wanted to get rid of.
      
      Differential Revision: https://phab.mercurial-scm.org/D6173
      4606585549b1
  21. 27 Mar, 2019 1 commit
  22. 26 Mar, 2019 5 commits