Skip to content
Snippets Groups Projects
  1. Jan 12, 2017
    • Matthieu Laneuville's avatar
      templatekw: force noprefix=False to insure diffstat consistency (issue4755) · cf1e15f91c90
      Matthieu Laneuville authored
      The result of diffstatdata should not depend on having noprefix set or not, as
      was reported in issue 4755. Forcing noprefix to false on call makes sure the
      parser receives the diff in the correct format and returns the proper result.
      
      Another way to fix this would have been to change the regular expressions in
      path.diffstatdata(), but that would have introduced many unecessary special
      cases.
      cf1e15f91c90
  2. Jan 13, 2017
  3. Jan 09, 2017
  4. Jan 08, 2017
  5. Jan 09, 2017
  6. Jan 13, 2017
  7. Jan 02, 2017
    • Gregory Szorc's avatar
      util: compression APIs to support revlog decompression · f50c0db50025
      Gregory Szorc authored
      Previously, compression engines had APIs for performing revlog
      compression but no mechanism to perform revlog decompression. This
      patch changes that.
      
      Revlog decompression is slightly more complicated than compression
      because in the compression case there is (currently) only a single
      engine that can be used at a time. However for decompression, a
      revlog could contain chunks from multiple compression engines. This
      means decompression needs to map to multiple engines and
      decompressors. This functionality is outside the scope of this patch.
      But it drives the decision for engines to declare a byte header
      sequence that identifies revlog data as belonging to an engine and
      an API for obtaining an engine from a revlog header.
      f50c0db50025
  8. Jan 08, 2017
    • Anton Shestakov's avatar
      crecord: add an experimental option for space key to move cursor down · 0bde7372e4c0
      Anton Shestakov authored
      I really want to have an option of toggling a selection on a line and also
      moving cursor down as a single keystroke. It also kinda makes sense for space
      key to do this, because some other curses UIs in the wild do this (e.g. various
      file managers, htop). So I got an idea to make a config option that defaults to
      False for compatibility, but allows making crecord UI a lot more useful for
      people with big hunks.
      
      We add this an experimental option to experiment with this behavior.
      0bde7372e4c0
  9. Jan 02, 2017
    • Gregory Szorc's avatar
      perf: support multiple compression engines in perfrevlogchunks · 168ef0a4eb3b
      Gregory Szorc authored
      Now that the revlog has a reference to a compressor, it is
      possible to swap in other compression engines. So, teach
      `hg perfrevlogchunks` to do that.
      
      The default behavior of `hg perfrevlogchunks` is now to measure the
      compression performance of all compression engines implementing the
      revlog compressor API. This effectively adds the no-op "none"
      compressor and zstd (when available) into the default set.
      
      While we can't yet plug alternate compressors into revlogs, this
      command gives us a preview of the performance. On the mozilla-unified
      repository:
      
      $ hg perfrevlogchunks -c
      ! compress w/ none
      ! wall 0.115159 comb 0.110000 user 0.110000 sys 0.000000 (best of 86)
      ! compress w/ zlib
      ! wall 5.681406 comb 5.680000 user 5.680000 sys 0.000000 (best of 3)
      ! compress w/ zstd
      ! wall 2.624781 comb 2.620000 user 2.620000 sys 0.000000 (best of 4)
      
      $ hg perfrevlogchunks -m
      ! compress w/ none
      ! wall 0.124486 comb 0.120000 user 0.120000 sys 0.000000 (best of 79)
      ! compress w/ zlib
      ! wall 10.144701 comb 10.150000 user 10.150000 sys 0.000000 (best of 3)
      ! compress w/ zstd
      ! wall 4.383118 comb 4.390000 user 4.390000 sys 0.000000 (best of 3)
      
      Those numbers for zstd look promising. But they aren't the full story.
      For that, we'll need to look at decompression times and storage sizes.
      Stay tuned...
      168ef0a4eb3b
    • Gregory Szorc's avatar
      revlog: use compression engine API for compression · 78ac56aebab6
      Gregory Szorc authored
      This commit swaps in the just-added revlog compressor API into
      the revlog class.
      
      Instead of implementing zlib compression inline in compress(), we
      now store a cached-on-first-use revlog compressor on each revlog
      instance and invoke its "compress()" method.
      
      As part of this, revlog.compress() has been refactored a bit to use
      a cleaner code flow and modern formatting (e.g. avoiding
      parenthesis around returned tuples).
      
      On a mozilla-unified repo, here are the "compress" times for a few
      commands:
      
      $ hg perfrevlogchunks -c
      ! wall 5.772450 comb 5.780000 user 5.780000 sys 0.000000 (best of 3)
      ! wall 5.795158 comb 5.790000 user 5.790000 sys 0.000000 (best of 3)
      
      $ hg perfrevlogchunks -m
      ! wall 9.975789 comb 9.970000 user 9.970000 sys 0.000000 (best of 3)
      ! wall 10.019505 comb 10.010000 user 10.010000 sys 0.000000 (best of 3)
      
      Compression times did seem to slow down just a little. There are
      360,210 changelog revisions and 359,342 manifest revisions. For the
      changelog, mean time to compress a revision increased from ~16.025us to
      ~16.088us. That's basically a function call or an attribute lookup. I
      suppose this is the price you pay for abstraction. It's so low that
      I'm not concerned.
      78ac56aebab6
    • Gregory Szorc's avatar
      util: compression APIs to support revlog compression · 31e1f0d4ab44
      Gregory Szorc authored
      As part of "zstd all of the things," we need to teach revlogs to
      use non-zlib compression formats. Because we're routing all compression
      via the "compression manager" and "compression engine" APIs, we need to
      introduction functionality there for performing revlog operations.
      
      Ideally, revlog compression and decompression operations would be
      implemented in terms of simple "compress" and "decompress" primitives.
      However, there are a few considerations that make us want to have a
      specialized primitive for handling revlogs:
      
      1) Performance. Revlogs tend to do compression and especially
         decompression operations in batches. Any overhead for e.g.
         instantiating a "context" for performing an operation can be
         noticed. For this reason, our "revlog compressor" primitive is
         reusable. For zstd, we reuse the same compression "context" for
         multiple operations. I've measured this to have a performance
         impact versus constructing new contexts for each operation.
      
      2) Specialization. By having a primitive dedicated to revlog use,
         we can make revlog-specific choices and leave the door open for
         more functionality in the future. For example, the zstd revlog
         compressor may one day make use of dictionary compression.
      
      A future patch will introduce a decompress() on the compressor
      object.
      
      The code for the zlib compressor is basically copied from
      revlog.compress(). Although it doesn't handle the empty input
      case, the null first byte case, and the 'u' prefix case. These
      cases will continue to be handled in revlog.py once that code is
      ported to use this API.
      31e1f0d4ab44
    • Gregory Szorc's avatar
      revlog: move decompress() from module to revlog class (API) · b6f455a6e4d6
      Gregory Szorc authored
      Upcoming patches will convert revlogs to use the compression engine
      APIs to perform all things compression. The yet-to-be-introduced
      APIs support a persistent "compressor" object so the same object
      can be reused for multiple compression operations, leading to
      better performance. In addition, compression engines like zstd
      may wish to tweak compression engine state based on the revlog
      (e.g. per-revlog compression dictionaries).
      
      A global and shared decompress() function will shortly no longer
      make much sense. So, we move decompress() to be a method of the
      revlog class. It joins compress() there.
      
      On the mozilla-unified repo, we can measure the impact of this change
      on reading performance:
      
      $ hg perfrevlogchunks -c
      ! chunk
      ! wall 1.932573 comb 1.930000 user 1.900000 sys 0.030000 (best of 6)
      ! wall 1.955183 comb 1.960000 user 1.930000 sys 0.030000 (best of 6)
      ! chunk batch
      ! wall 1.787879 comb 1.780000 user 1.770000 sys 0.010000 (best of 6
      ! wall 1.774444 comb 1.770000 user 1.750000 sys 0.020000 (best of 6)
      
      "chunk" appeared to become slower but "chunk batch" got faster. Upon
      further examination by running both sets multiple times, the numbers
      appear to converge across all runs. This tells me that there is no
      perceived performance impact to this refactor.
      b6f455a6e4d6
    • Gregory Szorc's avatar
      revlog: make compressed size comparisons consistent · 4215dc1b708b
      Gregory Szorc authored
      revlog.compress() compares the compressed size to the input size
      and throws away the compressed data if it is larger than the input.
      This is the correct thing to do, as storing compressed data that
      is larger than the input takes up more storage space and makes reading
      slower.
      
      However, the comparison was implemented inconsistently. For the
      streaming compression mode, we threw away the result if it was
      greater than or equal to the input size. But for the one-shot
      compression, we threw away the compression only if it was greater
      than the input size!
      
      This patch changes the comparison for the simple case so it is
      consistent with the streaming case.
      
      As a few tests demonstrate, this adds 1 byte to some revlog entries.
      This is because of an added 'u' header on the chunk. It seems
      somewhat wrong to increase the revlog size here. However, IMO the cost
      of 1 byte in storage is insignificant compared to the performance gains
      of avoiding decompression. This patch should invite questions around
      the heuristic for throwing away compressed data. For example, I'd argue
      we should be more liberal about rejecting compressed data, additionally
      doing so where the number of bytes saved fails to reach a threshold.
      But we can have this discussion another time.
      4215dc1b708b
  10. Jan 08, 2017
  11. Jan 09, 2017
  12. Dec 31, 2016
    • Sean Farley's avatar
      patch: add index line for diff output · b8ad243f5ded
      Sean Farley authored
      This helps highlighting in third-party diff coloring (which assumes git
      output) and maintains pedantic correctness with diff --git.
      
      Tests will be added at the end of the series.
      b8ad243f5ded
  13. Jan 09, 2017
    • Sean Farley's avatar
      patch: add config knob for displaying the index header · d1901c4c8ec0
      Sean Farley authored
      This config knob can take an integer between 0 and 40 or a
      keyword ('none', 'short', 'full') to control the length of hash to
      output. It will display diffs with the git index header as such,
      
        diff --git a/mercurial/mdiff.py b/mercurial/mdiff.py
        index 112edf7..d6b52c5 100644
      
      We'll put this in the experimental section for now.
      d1901c4c8ec0
  14. Jan 12, 2017
  15. Jan 08, 2017
    • Matt Harbison's avatar
      help: eliminate duplicate text for revset string patterns · 5dd67f0993ce
      Matt Harbison authored
      There's no reason to duplicate this so many times, and it's likely an instance
      will be missed if support for a new pattern is added and documented.  The
      stringmatcher is mostly used by revsets, though it is also used for the 'tag'
      related templates, and namespace filtering in the journal extension.  So maybe
      there's a better place to document it.  `hg help patterns` seems inappropriate,
      because that is all file pattern matching.
      
      While here, indicate how to perform case insensitive regex searches.
      5dd67f0993ce
    • Matt Harbison's avatar
      revset: add regular expression support to 'desc' · 931a60880df4
      Matt Harbison authored
      This is a case insensitive predicate like 'author', so it conforms to the
      existing behavior of performing a case insensitive regex.
      931a60880df4
  16. Jan 12, 2017
  17. Nov 25, 2016
    • Gregory Szorc's avatar
      repair: clean up stale lock file from store backup · f2c069bf78ee
      Gregory Szorc authored
      Since we did a directory rename on the stores, the source
      repository's lock path now references the dest repository's
      lock path and the dest repository's lock path now references
      a non-existent filename.
      
      So releasing the lock on the source will unlock the dest and
      releasing the lock on the dest will no-op because it fails due
      to file not found. So we clean up the dest's lock manually.
      f2c069bf78ee
    • Gregory Szorc's avatar
      repair: copy non-revlog store files during upgrade · 2603d04889e1
      Gregory Szorc authored
      The store contains more than just revlogs. This patch teaches the
      upgrade code to copy regular files as well.
      
      As the test changes demonstrate, the phaseroots file is now copied.
      2603d04889e1
  18. Dec 19, 2016
    • Gregory Szorc's avatar
      repair: migrate revlogs during upgrade · 38aa1ca97b6a
      Gregory Szorc authored
      Our next step for in-place upgrade is to migrate store data. Revlogs
      are the biggest source of data within the store and a store is useless
      without them, so we implement their migration first.
      
      Our strategy for migrating revlogs is to walk the store and call
      `revlog.clone()` on each revlog. There are some minor complications.
      
      Because revlogs have different storage options (e.g. changelog has
      generaldelta and delta chains disabled), we need to obtain the
      correct class of revlog so inserted data is encoded properly for its
      type.
      
      Various attempts at implementing progress indicators that didn't lead
      to frustration from false "it's almost done" indicators were made.
      
      I initially used a single progress bar based on number of revlogs.
      However, this quickly churned through all filelogs, got to 99% then
      effectively froze at 99.99% when it got to the manifest.
      
      So I converted the progress bar to total revision count. This was a
      little bit better. But the manifest was still significantly slower
      than filelogs and it took forever to process the last few percent.
      
      I then tried both revision/chunk bytes and raw bytes as the
      denominator. This had the opposite effect: because so much data is in
      manifests, it would churn through filelogs without showing much
      progress. When it got to manifests, it would fill in 90+% of the
      progress bar.
      
      I finally gave up having a unified progress bar and instead implemented
      3 progress bars: 1 for filelog revisions, 1 for manifest revisions, and
      1 for changelog revisions. I added extra messages indicating the total
      number of revisions of each so users know there are more progress bars
      coming.
      
      I also added extra messages before and after each stage to give extra
      details about what is happening. Strictly speaking, this isn't
      necessary. But the numbers are impressive. For example, when converting
      a non-generaldelta mozilla-central repository, the messages you see are:
      
         migrating 2475593 total revisions (1833043 in filelogs, 321156 in manifests, 321394 in changelog)
         migrating 1.67 GB in store; 2508 GB tracked data
         migrating 267868 filelogs containing 1833043 revisions (1.09 GB in store; 57.3 GB tracked data)
         finished migrating 1833043 filelog revisions across 267868 filelogs; change in size: -415776 bytes
         migrating 1 manifests containing 321156 revisions (518 MB in store; 2451 GB tracked data)
      
      That "2508 GB" figure really blew me away. I had no clue that the raw
      tracked data in mozilla-central was that large. Granted, 2451 GB is in
      the manifest and "only" 57.3 GB is in filelogs. But still.
      
      It's worth noting that gratuitous loading of source revlogs in order
      to display numbers and progress bars does serve a purpose: it ensures
      we can open all source revlogs. We don't want to spend several minutes
      copying revlogs only to encounter a permissions error or similar later.
      
      As part of this commit, we also add swapping of the store directory
      to the upgrade function. After revlogs are converted, we move the
      old store into the backup directory then move the temporary repo's
      store into the old store's location. On well-behaved systems, this
      should be 2 atomic operations and the window of inconsistency show be
      very narrow.
      
      There are still a few improvements to be made to store copying and
      upgrading. But this commit gets the bulk of the work out of the way.
      38aa1ca97b6a
    • Gregory Szorc's avatar
      revlog: add clone method · 1c7368d1a25f
      Gregory Szorc authored
      Upcoming patches will introduce functionality for in-place
      repository/store "upgrades." Copying the contents of a revlog
      feels sufficiently low-level to warrant being in the revlog
      class. So this commit implements that functionality.
      
      Because full delta recomputation can be *very* expensive (we're
      talking several hours on the Firefox repository), we support
      multiple modes of execution with regards to delta (re)use. This
      will allow repository upgrades to choose the "level" of
      processing/optimization they wish to perform when converting
      revlogs.
      
      It's not obvious from this commit, but "addrevisioncb" will be
      used for progress reporting.
      1c7368d1a25f
    • Gregory Szorc's avatar
      repair: begin implementation of in-place upgrading · 7de7afd8bdd9
      Gregory Szorc authored
      Now that all the upgrade planning work is in place, we can start
      doing the real work: actually upgrading a repository.
      
      The main goal of this commit is to get the "framework" for running
      in-place upgrade actions in place.
      
      Rather than get too clever and low-level with regards to in-place
      upgrades, our strategy is to create a new, temporary repository,
      copy data to it, then replace the old data with the new. This allows
      us to reuse a lot of code in localrepo.py around store interaction,
      which will eventually consume the bulk of the upgrade code.
      
      But we have to start small. This patch implements adding new
      repository requirements. But it still sets up a temporary
      repository and locks it and the source repo before performing the
      requirements file swap. This means all the plumbing is in place
      to implement store copying in subsequent commits.
      7de7afd8bdd9
    • Gregory Szorc's avatar
      repair: determine what upgrade will do · 3997edc4a86d
      Gregory Szorc authored
      This commit introduces code for determining what actions/improvements
      an upgrade should perform.
      
      The "upgradefindimprovements" function introduces a mechanism to
      return a list of improvements that can be made to a repository.
      Each improvement is effectively an action that an upgrade will
      perform. Associated with each of these improvements is metadata
      that will be used to inform users what's wrong and what an
      upgrade will do.
      
      Each "improvement" is categorized as a "deficiency" or an
      "optimization." TBH, I'm not thrilled about the terminology and
      am receptive to constructive bikeshedding. The main difference
      between a "deficiency" and an "optimization" is a deficiency
      is always corrected (if it deviates from the current config) and
      an "optimization" is an optional action that goes above and beyond
      to improve the state of the repository (usually by requiring more
      CPU during upgrade).
      
      Our initial set of improvements identifies missing repository
      requirements, a single, easily correctable problem with
      changelog storage, and a set of "optimizations" related to delta
      recalculation.
      
      The main "upgraderepo" function has been expanded to handle
      improvements. It queries for the list of improvements and determines
      which of them will run based on the current repository state and user
      
      I went through numerous iterations of the output format before
      settling on a ReST-inspired definition list format. (I used
      bulleted lists in the first submission of this commit and could
      not get it to format just right.) Even with the various iterations,
      I'm still not super thrilled with the format. But, this is a debug*
      command, so that should mean we can refine the output without BC
      concerns.
      3997edc4a86d
    • Gregory Szorc's avatar
      repair: implement requirements checking for upgrades · 513d68a90398
      Gregory Szorc authored
      This commit introduces functionality for upgrading a repository in
      place. The first part that's implemented is testing for upgrade
      "compatibility." This is done by examining repository requirements.
      
      There are 5 functions returning sets of requirements that control
      upgrading. Why so many functions? Mainly to support extensions.
      Functions are easier to monkeypatch than module variables.
      
      Astute readers will see that we don't support "manifestv2" and
      "treemanifest" requirements in the upgrade mechanism. I don't have
      a great answer for why other than this is a complex set of patches
      and I don't want to deal with the complexity of these experimental
      features just yet. We can teach the upgrade mechanism about them
      later, once the basic upgrade mechanism is in place.
      
      This commit also introduces the "upgraderepo" function. This will be
      our main routine for performing an in-place upgrade. Currently, it
      just implements requirements checking. The structure of some code in
      this function may look a bit weird (e.g. the inline function that is
      only called once). But this will make sense after future commits.
      513d68a90398
  19. Nov 25, 2016
    • Gregory Szorc's avatar
      debugcommands: stub for debugupgraderepo command · eaa5607132a2
      Gregory Szorc authored
      Currently, if Mercurial introduces a new repository/store feature or
      changes behavior of an existing feature, users must perform an
      `hg clone` to create a new repository with hopefully the
      correct/optimal settings. Unfortunately, even `hg clone` may not
      give the correct results. For example, if you do a local `hg clone`,
      you may get hardlinks to revlog files that inherit the old state.
      If you `hg clone` from a remote or `hg clone --pull`, changegroup
      application may bypass some optimization, such as converting to
      generaldelta.
      
      Optimizing a repository is harder than it seems and requires more
      than a simple `hg` command invocation.
      
      This commit starts the process of changing that. We introduce
      `hg debugupgraderepo`, a command that performs an in-place upgrade
      of a repository to use new, optimal features. The command is just
      a stub right now. Features will be added in subsequent commits.
      
      This commit does foreshadow some of the behavior of the new command,
      notably that it doesn't do anything by default and that it takes
      arguments that influence what actions it performs. These will be
      explained more in subsequent commits.
      eaa5607132a2
  20. Jan 12, 2017
    • Matt Harbison's avatar
      util: teach stringmatcher to handle forced case insensitive matches · c390b40fe1d7
      Matt Harbison authored
      The 'author' and 'desc' revsets are documented to be case insensitive.
      Unfortunately, this was implemented in 'author' by forcing the input to
      lowercase, including for regex like '\B'.  (This actually inverts the meaning of
      the sequence.)  For backward compatibility, we will keep that a case insensitive
      regex, but by using matcher options instead of brute force.
      
      This doesn't preclude future hypothetical 'icase-literal:' style prefixes that
      can be provided by the user.  Such user specified cases can probably be handled
      up front by stripping 'icase-', setting the variable, and letting it drop
      through the existing code.
      c390b40fe1d7
    • Matt Harbison's avatar
      revset: point to 'grep' in the 'keyword' help for regex searches · b1012cb1bec3
      Matt Harbison authored
      The help for 'grep' already points to 'keyword'.
      b1012cb1bec3
Loading