Skip to content
Snippets Groups Projects
  1. Jan 19, 2017
  2. Jan 18, 2017
  3. Jan 17, 2017
  4. Jan 18, 2017
  5. Jan 17, 2017
  6. Jan 16, 2017
  7. Jan 13, 2017
    • Pulkit Goyal's avatar
      shelve: allow multiple shelves with --patch and --stat · 806a830e6612
      Pulkit Goyal authored
      Before this patch, there was a single way to see multiple shelves using
      `--patch --list` which show all the shelves. Doing `--patch s1 s2` returns an
      error. This patch allows to show multiple shelves using `--patch` and `--stat`.
      806a830e6612
  8. Jan 15, 2017
  9. Jan 14, 2017
    • Pulkit Goyal's avatar
      util: add length argument to util.buffer() · 7005c03f7387
      Pulkit Goyal authored
      util.buffer() either returns inbuilt buffer function or defines a new one which
      slices. The inbuilt buffer() also has a length argument which is missing from
      the ones we defined. This patch adds that length argument.
      7005c03f7387
  10. Jan 15, 2017
  11. Jan 14, 2017
    • Gregory Szorc's avatar
      localrepo: experimental support for non-zlib revlog compression · 4c0a5a256ae8
      Gregory Szorc authored
      The final part of integrating the compression manager APIs into
      revlog storage is the plumbing for repositories to advertise they
      are using non-zlib storage and for revlogs to instantiate a non-zlib
      compression engine.
      
      The main intent of the compression manager work was to zstd all
      of the things. Adding zstd to revlogs has proved to be more involved
      than other places because revlogs are... special. Very small inputs
      and the use of delta chains (which are themselves a form of
      compression) are a completely different use case from streaming
      compression, which bundles and the wire protocol employ. I've
      conducted numerous experiments with zstd in revlogs and have yet
      to formalize compression settings and a storage architecture that
      I'm confident I won't regret later. In other words, I'm not yet
      ready to commit to a new mechanism for using zstd - or any other
      compression format - in revlogs.
      
      That being said, having some support for zstd (and other compression
      formats) in revlogs in core is beneficial. It can allow others to
      conduct experiments.
      
      This patch introduces *highly experimental* support for non-zlib
      compression formats in revlogs. Introduced is a config option to
      control which compression engine to use. Also introduced is a namespace
      of "exp-compression-*" requirements to denote support for non-zlib
      compression in revlogs. I've prefixed the namespace with "exp-"
      (short for "experimental") because I'm not confident of the
      requirements "schema" and in no way want to give the illusion of
      supporting these requirements in the future. I fully intend to drop
      support for these requirements once we figure out what we're doing
      with zstd in revlogs.
      
      A good portion of the patch is teaching the requirements system
      about registered compression engines and passing the requested
      compression engine as an opener option so revlogs can instantiate
      the proper compression engine for new operations.
      
      That's a verbose way of saying "we can now use zstd in revlogs!"
      
      On an `hg pull` conversion of the mozilla-unified repo with no extra
      redelta settings (like aggressivemergedeltas), we can see the impact
      of zstd vs zlib in revlogs:
      
      $ hg perfrevlogchunks -c
      ! chunk
      ! wall 2.032052 comb 2.040000 user 1.990000 sys 0.050000 (best of 5)
      ! wall 1.866360 comb 1.860000 user 1.820000 sys 0.040000 (best of 6)
      
      ! chunk batch
      ! wall 1.877261 comb 1.870000 user 1.860000 sys 0.010000 (best of 6)
      ! wall 1.705410 comb 1.710000 user 1.690000 sys 0.020000 (best of 6)
      
      $ hg perfrevlogchunks -m
      ! chunk
      ! wall 2.721427 comb 2.720000 user 2.640000 sys 0.080000 (best of 4)
      ! wall 2.035076 comb 2.030000 user 1.950000 sys 0.080000 (best of 5)
      
      ! chunk batch
      ! wall 2.614561 comb 2.620000 user 2.580000 sys 0.040000 (best of 4)
      ! wall 1.910252 comb 1.910000 user 1.880000 sys 0.030000 (best of 6)
      
      $ hg perfrevlog -c -d 1
      ! wall 4.812885 comb 4.820000 user 4.800000 sys 0.020000 (best of 3)
      ! wall 4.699621 comb 4.710000 user 4.700000 sys 0.010000 (best of 3)
      
      $ hg perfrevlog -m -d 1000
      ! wall 34.252800 comb 34.250000 user 33.730000 sys 0.520000 (best of 3)
      ! wall 24.094999 comb 24.090000 user 23.320000 sys 0.770000 (best of 3)
      
      Only modest wins for the changelog. But manifest reading is
      significantly faster. What's going on?
      
      One reason might be data volume. zstd decompresses faster. So given
      more bytes, it will put more distance between it and zlib.
      
      Another reason is size. In the current design, zstd revlogs are
      *larger*:
      
      debugcreatestreamclonebundle (size in bytes)
      zlib: 1,638,852,492
      zstd: 1,680,601,332
      
      I haven't investigated this fully, but I reckon a significant cause of
      larger revlogs is that the zstd frame/header has more bytes than
      zlib's. For very small inputs or data that doesn't compress well, we'll
      tend to store more uncompressed chunks than with zlib (because the
      compressed size isn't smaller than original). This will make revlog
      reading faster because it is doing less decompression.
      
      Moving on to bundle performance:
      
      $ hg bundle -a -t none-v2 (total CPU time)
      zlib: 102.79s
      zstd:  97.75s
      
      So, marginal CPU decrease for reading all chunks in all revlogs
      (this is somewhat disappointing).
      
      $ hg bundle -a -t <engine>-v2 (total CPU time)
      zlib: 191.59s
      zstd: 115.36s
      
      This last test effectively measures the difference between zlib->zlib
      and zstd->zstd for revlogs to bundle. This is a rough approximation of
      what a server does during `hg clone`.
      
      There are some promising results for zstd. But not enough for me to
      feel comfortable advertising it to users. We'll get there...
      4c0a5a256ae8
    • Gregory Szorc's avatar
      revlog: use compression engine APIs for decompression · 2b279126b8f5
      Gregory Szorc authored
      Now that compression engines declare their header in revlog chunks
      and can decompress revlog chunks, we refactor revlog.decompress()
      to use them.
      
      Making full use of the property that revlog compressor objects are
      reusable, revlog instances now maintain a dict mapping an engine's
      revlog header to a compressor object. This is not only a performance
      optimization for engines where compressor object reuse can result in
      better performance, but it also serves as a cache of header values
      so we don't need to perform redundant lookups against the compression
      engine manager. (Yes, I measured and the overhead of a function call
      versus a dict lookup was observed.)
      
      Replacing the previous inline lookup table with a dict lookup was
      measured to make chunk reading ~2.5% slower on changelogs and ~4.5%
      slower on manifests. So, the inline lookup table has been mostly
      preserved so we don't lose performance. This is unfortunate. But
      many decompression operations complete in microseconds, so Python
      attribute lookup, dict lookup, and function calls do matter.
      
      The impact of this change on mozilla-unified is as follows:
      
      $ hg perfrevlogchunks -c
      ! chunk
      ! wall 1.953663 comb 1.950000 user 1.920000 sys 0.030000 (best of 6)
      ! wall 1.946000 comb 1.940000 user 1.910000 sys 0.030000 (best of 6)
      ! chunk batch
      ! wall 1.791075 comb 1.800000 user 1.760000 sys 0.040000 (best of 6)
      ! wall 1.785690 comb 1.770000 user 1.750000 sys 0.020000 (best of 6)
      
      $ hg perfrevlogchunks -m
      ! chunk
      ! wall 2.587262 comb 2.580000 user 2.550000 sys 0.030000 (best of 4)
      ! wall 2.616330 comb 2.610000 user 2.560000 sys 0.050000 (best of 4)
      ! chunk batch
      ! wall 2.427092 comb 2.420000 user 2.400000 sys 0.020000 (best of 5)
      ! wall 2.462061 comb 2.460000 user 2.400000 sys 0.060000 (best of 4)
      
      Changelog chunk reading is slightly faster but manifest reading is
      slower. What gives?
      
      On this repo, 99.85% of changelog entries are zlib compressed (the 'x'
      header). On the manifest, 67.5% are zlib and 32.4% are '\0'. This patch
      swapped the test order of 'x' and '\0' so now 'x' is tested first. This
      makes changelogs faster since they almost always hit the first branch.
      This makes a significant percentage of manifest '\0' chunks slower
      because that code path now performs an extra test. Yes, I too can't
      believe we're able to measure the impact of an if..elif with simple
      string compares. I reckon this code would benefit from being written
      in C...
      2b279126b8f5
  12. Jan 13, 2017
    • Denis Laxalde's avatar
      hgweb: build the "entries" list directly in filelog command · 96f811bceb85
      Denis Laxalde authored
      There's no apparent reason to have this "entries" generator function that
      builds a list and then yields its elements in reverse order and which is only
      called to build the "entries" list. So just build the list directly, in
      reverse order.
      
      Adjust "parity" generator's offset to keep rendering the same.
      96f811bceb85
  13. Jan 14, 2017
    • Gregory Szorc's avatar
      convert: remove "replacecommitter" action · c5bf2e8ec18c
      Gregory Szorc authored
      As pointed out by Yuya, this action doesn't add much (any?) value.
      c5bf2e8ec18c
    • Yuya Nishihara's avatar
      ui: check EOF of getpass() response read from command-server channel · b96c57c1f860
      Yuya Nishihara authored
      readline() returns '' only when EOF is encountered, in which case, Python's
      getpass() raises EOFError. We should do the same to abort the session as
      "response expected."
      
      This bug was reported to
      https://bitbucket.org/tortoisehg/thg/issues/4659/
      b96c57c1f860
    • Gregory Szorc's avatar
      convert: config option to control Git committer actions · 2cbbd4622ab0
      Gregory Szorc authored
      When converting a Git repository to Mercurial at Mozilla, I encountered
      a scenario where I didn't want `hg convert` to automatically add the
      "committer: <committer>" line to commit messages. While I can hack around
      this by rewriting the Git commit before it is fed into `hg convert`,
      I figured it would be a useful knob to control.
      
      This patch introduces a config option that allows lots of control
      over the committer value. I initially implemented this as a single
      boolean flag to control whether to save the committer message. But
      then there was feedback that it would be useful to save the committer
      in extra data. While this patch doesn't implement support for saving
      in extra data, it does add a mechanism for extending which actions
      to take on the committer field. We should be able to easily add
      actions to save in extra data.
      
      Some of the implemented features weren't asked for. But I figured they
      could be useful. If nothing else they demonstrate the extensibility
      of this mechanism.
      2cbbd4622ab0
    • Gregory Szorc's avatar
      help: make "mergetool" an alias for "merge-tools" · 98bfce9bd5e5
      Gregory Szorc authored
      I've probably typed `hg help mergetool` dozens of times. I'm tired
      of it not working.
      98bfce9bd5e5
  14. Jan 12, 2017
    • Matthieu Laneuville's avatar
      templatekw: force noprefix=False to insure diffstat consistency (issue4755) · cf1e15f91c90
      Matthieu Laneuville authored
      The result of diffstatdata should not depend on having noprefix set or not, as
      was reported in issue 4755. Forcing noprefix to false on call makes sure the
      parser receives the diff in the correct format and returns the proper result.
      
      Another way to fix this would have been to change the regular expressions in
      path.diffstatdata(), but that would have introduced many unecessary special
      cases.
      cf1e15f91c90
  15. Jan 13, 2017
  16. Jan 09, 2017
Loading