Skip to content
Snippets Groups Projects
  1. Mar 27, 2019
    • Pierre-Yves David's avatar
      compression: introduce a `storage.revlog.zlib.level` configuration · 1fac9b93
      Pierre-Yves David authored
      This option control the zlib compression level used when compression revlog
      chunk.
      
      This is also a good excuse to pave the way for a similar configuration option
      for the zstd compression engine. Having a dedicated option for each compression
      algorithm is useful because they don't support the same range of values.
      
      Using a higher zlib compression impact CPU consumption at compression time, but
      does not directly affected decompression time. However dealing with small
      compressed chunk can directly help decompression and indirectly help other
      revlog logic.
      
      I ran some basic test on repositories using different level. I am using the
      mercurial, pypy, netbeans and mozilla-central clone from our benchmark suite.
      All tested repository use sparse-revlog and got all their delta recomputed.
      
      The different compression level has a small effect on the repository size
      (about 10% variation in the total range). My quick analysis is that revlog
      mostly store small delta, that are not affected by the compression level much.
      So the variation probably mostly comes from better compression of the snapshots
      revisions, and snapshot revision only represent a small portion of the
      repository content.
      
      I also made some basic timings measurements. The "read" timings are gathered using
      simple run of `hg perfrevlogrevisions`, the "write" timings using `hg
      perfrevlogwrite` (restricted to the last 5000 revisions for netbeans and
      mozilla central). The timings are gathered on a generic machine, (not one  of
      our performance locked machine), so small variation might not be meaningful.
      However large trend remains relevant.
      
      Keep in mind that these numbers are not pure compression/decompression time.
      They also involve the full revlog logic. In particular the difference in chunk
      size has an impact on the delta chain structure, affecting performance when
      writing or reading them.
      
      On read/write performance, the compression level has a bigger impact.
      Counter-intuitively, the higher compression levels improve "write" performance
      for the large repositories in our tested setting. Maybe because the last 5000
      delta chain end up having a very different shape in this specific spot? Or maybe
      because of a more general trend of better delta chains thanks to the smaller
      chunk and snapshot.
      
      This series does not intend to change the default compression level. However,
      these result call for a deeper analysis of this performance difference in the
      future.
      
      Full data
      =========
      
      repo   level  .hg/store size  00manifest.d read       write
      ----------------------------------------------------------------
      mercurial  1      49,402,813     5,963,475   0.170159  53.250304
      mercurial  6      47,197,397     5,875,730   0.182820  56.264320
      mercurial  9      47,121,596     5,849,781   0.189219  56.293612
      
      pypy       1     370,830,572    28,462,425   2.679217 460.721984
      pypy       6     340,112,317    27,648,747   2.768691 467.537158
      pypy       9     338,360,736    27,639,003   2.763495 476.589918
      
      netbeans   1   1,281,847,810   165,495,457 122.477027 520.560316
      netbeans   6   1,205,284,353   159,161,207 139.876147 715.930400
      netbeans   9   1,197,135,671   155,034,586 141.620281 678.297064
      
      mozilla    1   2,775,497,186   298,527,987 147.867662 751.263721
      mozilla    6   2,596,856,420   286,597,671 170.572118 987.056093
      mozilla    9   2,587,542,494   287,018,264 163.622338 739.803002
      1fac9b93
    • Pierre-Yves David's avatar
      compression: accept level management for zlib compression · aaececb4
      Pierre-Yves David authored
      We update the zlib related class to be support setting the compression level.
      This changeset focus on updating the internal only. A way to configure this
      level will be introduced in the next changeset.
      aaececb4
    • Pierre-Yves David's avatar
      util: extract compression code in `mercurial.utils.compression` · 3e47d1ec
      Pierre-Yves David authored
      The code seems large enough to be worth extracting. This is similar to what was
      done for various module in `mercurial/utils/`.
      
      Since None of the compression logic takes a `ui` objet, issuing deprecation
      warning is tricky. Luckly the logic does not seems to have many external users.
      3e47d1ec
  2. Mar 30, 2019
  3. Mar 31, 2019
  4. Mar 29, 2019
    • Martin von Zweigbergk's avatar
      shelve: let cmdutil.revert() take care of backing up untracked files · e0855976
      Martin von Zweigbergk authored
      cmdutil.revert() backs up untracked files, so I don't see a reason to
      do it shelve.mergefiles(). We have tests for this and they still pass.
      
      Differential Revision: https://phab.mercurial-scm.org/D6174
      e0855976
    • Martin von Zweigbergk's avatar
      shelve: stop passing list of files to revert · 46065855
      Martin von Zweigbergk authored
      It seems to work just fine to not specify any files here. I suspect it
      looked the way it did for historical reasons. It apparently used to
      use merge instead of rebase until 1d7a36ff2615 (shelve: use rebase
      instead of merge (issue4068), 2013-10-23) and it makes sense to want
      to restrict the set of files then.
      
      I noticed this because of the files.extend(shelvectx.p1().files()). If
      the working copy was clean before, then shelvectx.p1() will be the
      working copy parent and that ended up adding all the files in that
      set. In our Google-internal Mercurial setup (including a FUSE) that
      was very noticeably slow when the working copy parent happened to have
      many files in large directories.
      
      This patch doesn't yet remove the call to shelvectx.p1().files(). We
      also use that set for deciding what to back up. I'm pretty sure it's
      safe to back up only the set of files we already back even if we no
      longer restrict the set of files to revert, so this patch should be
      safe on its own. Regardless, the next patch will delegate the
      backing-up to cmdutil.revert().
      
      Incidentally, this also gets rid of a repo.pathto() that I had earlier
      wanted to get rid of.
      
      Differential Revision: https://phab.mercurial-scm.org/D6173
      46065855
  5. Mar 27, 2019
  6. Mar 26, 2019
  7. Mar 17, 2019
  8. Mar 21, 2019
    • Alexander Kobjolke's avatar
      crecord: draw on the whole screen · 66fc05ff
      Alexander Kobjolke authored
      When starting crecord, one can see that it has a small gap on the rightmost
      column which doesn't get used. This is in contrast to other interactive curses
      frontends such as chistedit.
      
      Disabling the displaying of the cursor allows drawing on the whole availabe
      area and thus some hacky code in align() could be removed.
      
      Differential Revision: https://phab.mercurial-scm.org/D6171
      66fc05ff
  9. Mar 15, 2019
    • Gregory Szorc's avatar
      automation: perform tasks on remote machines · b05a3e28
      Gregory Szorc authored
      Sometimes you don't have access to a machine in order to
      do something. For example, you may not have access to a Windows
      machine required to build Windows binaries or run tests on that
      platform.
      
      This commit introduces a pile of code intended to help
      "automate" common tasks, like building release artifacts.
      
      In its current form, the automation code provides functionality
      for performing tasks on Windows EC2 instances.
      
      The hgautomation.aws module provides functionality for integrating
      with AWS. It manages EC2 resources such as IAM roles, EC2
      security groups, AMIs, and instances.
      
      The hgautomation.windows module provides a higher-level
      interface for performing tasks on remote Windows machines.
      
      The hgautomation.cli module provides a command-line interface to
      these higher-level primitives.
      
      I attempted to structure Windows remote machine interaction
      around Windows Remoting / PowerShell. This is kinda/sorta like
      SSH + shell, but for Windows. In theory, most of the functionality
      is cloud provider agnostic, as we should be able to use any
      established WinRM connection to interact with a remote. In
      reality, we're tightly coupled to AWS at the moment because
      I didn't want to prematurely add abstractions for a 2nd cloud
      provider. (1 was hard enough to implement.)
      
      In the aws module is code for creating an image with a fully
      functional Mercurial development environment. It contains VC9,
      VC2017, msys, and other dependencies. The image is fully capable
      of building all the existing Mercurial release artifacts and
      running tests.
      
      There are a few things that don't work. For example, running
      Windows tests with Python 3. But building the Windows release
      artifacts does work. And that was an impetus for this work.
      (Although we don't yet support code signing.)
      
      Getting this functionality to work was extremely time consuming.
      It took hours debugging permissions failures and other wonky
      behavior due to PowerShell Remoting. (The permissions model for
      PowerShell is crazy and you brush up against all kinds of
      issues because of the user/privileges of the user running
      the PowerShell and the permissions of the PowerShell session
      itself.)
      
      The functionality around AWS resource management could use some
      improving. In theory we support shared tenancy via resource
      name prefixing. In reality, we don't offer a way to configure
      this.
      
      Speaking of AWS resource management, I thought about using a tool
      like Terraform to manage resources. But at our scale, writing a
      few dozen lines of code to manage resources seemed acceptable.
      Maybe we should reconsider this if things grow out of control.
      Time will tell.
      
      Currently, emphasis is placed on Windows. But I only started
      there because it was likely to be the most difficult to implement.
      It should be relatively trivial to automate tasks on remote Linux
      machines. In fact, I have a ~1 year old script to run tests on a
      remote EC2 instance. I will likely be porting that to this new
      "framework" in the near future.
      
      # no-check-commit because foo_bar functions
      
      Differential Revision: https://phab.mercurial-scm.org/D6142
      b05a3e28
  10. Mar 10, 2019
  11. Mar 26, 2019
  12. Mar 25, 2019
  13. Mar 17, 2019
  14. Mar 16, 2019
  15. Mar 25, 2019
  16. Mar 22, 2019
  17. Mar 19, 2019
    • Pulkit Goyal's avatar
      branchcache: have a hasnode function to validate nodes · b5511845
      Pulkit Goyal authored
      Upcoming patches will delay node validation until it's required. So we need to a
      way in branchcache class to check whether a node exists or node.
      
      Other options were making repo or changelog an attribute of branchcache object.
      But the branchcache depends on filters so I decided to pass a function object.
      
      Differential Revision: https://phab.mercurial-scm.org/D6157
      b5511845
    • Pulkit Goyal's avatar
      branchcache: add attributes to track which nodes are verified · 111de135
      Pulkit Goyal authored
      Half of the cost of loading branchcache comes from verifiying all the nodes it
      has. We don't need to verify all the nodes in all the cases. Sometimes we need
      to verify only a set of nodes for a set of branches. We can ignore nodes of
      other branches as we are not going to read them.
      
      This patch introduces two attributes to branchcache class. _verifiedbranches is
      a set which will tell the branches for which it's head nodes are verified.
      _closedverified is a boolean which will tell whether all closednodes are
      verified or not.
      
      Another idea which I had was to keep a set of nodes which are verified, but it
      will be more ugly and difficult to track things on node level.
      
      Differential Revision: https://phab.mercurial-scm.org/D6156
      111de135
  18. Mar 18, 2019
Loading