Skip to content
Snippets Groups Projects
  1. Feb 22, 2025
  2. Mar 11, 2024
  3. Jan 07, 2024
    • Pierre-Yves David's avatar
      delta-find: explicitly deal with usage of the cached revision · 7083b33a2699
      Pierre-Yves David authored
      We can remove this from the general logic path and directly deal with this
      corner case early.
      
      This result in a small change in test-generaldelta.t as it turns out that:
      
      - at commit time we (sometimes) precompute a delta against p1 and pass it as the
        cached delta.
      - since cached delta where going through the same filtering as everything, we
        could "optimize" the base if it applied to an empty delta, resulting in not
        using the pre-computed delta.
      
      The simpler logic fix the second item, making the cached delta base always actually
      tested when requested.
      
      Note that the computation of a fast delta against p1 only is questionable, but
      looking into that is out of scope for this series.
      7083b33a2699
  4. Sep 19, 2023
  5. Feb 28, 2023
    • Matt Harbison's avatar
      configitems: enable changegroup3 by default (unless using infinitepush) · 2cf264e9aa75
      Matt Harbison authored
      The LFS extension requires this, and if it isn't enabled on the client (or the
      LFS extension isn't loaded), a web client gets a 500 instead of a sensible error
      message.  Now it gets a different (client) error, but maybe it can be handled
      more gracefully.  c0f11347b107 indicates that treemanifest repos have this issue
      too.  29cfc474c5fd mentions gating this behind `experimental` so that the format
      could change, but that was 7 years ago and we now have an experimental
      `changegroup4` as well.
      
      We can keep this as a config for the next cycle in case someone runs into an
      unexpected problem, and then jettison it if the infinitepush bundle name changes
      are either acceptable as-is or can be created differently.
      2cf264e9aa75
  6. May 18, 2022
  7. Feb 20, 2022
  8. Aug 25, 2021
  9. Apr 06, 2021
  10. Feb 27, 2019
    • Pierre-Yves David's avatar
      storage: introduce a `revlog.reuse-external-delta-parent` config · f6eff9e4de80
      Pierre-Yves David authored
      As pointed in c59987ab96b4, we had no simple way to get a client to not blindly
      reuse the delta parent from a bundle. Instead one had to rely on a side effect
      of the deprecated `format.generaldelta` configuration.
      
      We introduce an explicit `revlog.reuse-external-delta-parent` configuration
      option (default to True) to control this behavior. If the option is not set,
      `format.generaldelta` still control this behavior.
      
      To test the new option, we convert a couple of place where `generaldelta` have
      been used for its side effects.
      f6eff9e4de80
  11. Nov 12, 2018
  12. Oct 31, 2018
    • Boris Feld's avatar
      changegroup: restore default node ordering (issue6001) · 634b45317459
      Boris Feld authored
      Changeset db5501d93bcf changed the default node ordering from "storage" to
      "linearize".
      
      While the new API is more explicit and cleaner, the "linearize" order is
      problematic on certain repositories like netbeans where it makes bundling
      slower the more nodes we bundle.
      
      Pushing and pulling 100 changesets was ~20% slower and pushing and pulling
      1000 changesets was ~600% slower.
      
      A very quick analysis of profile traces showed that the pull operation was
      taking more time creating the delta.
      
      Putting back the old default order seems to be the safe option. With more time
      during the next cycle, we can understand better the impact of sorting with the
      DAG order by default, the source of the regression and how to mitigate it.
      
      /!\ We are still waiting for the full performance impact but with this patch,
      bundling and pulling locally (not on the performance workstation) 1000
      changesets on the netbeans repository is as fast as before the regression.
      
      Differential Revision: https://phab.mercurial-scm.org/D5196
      634b45317459
  13. Sep 24, 2018
    • Gregory Szorc's avatar
      changegroup: remove reordering control (BC) · db5501d93bcf
      Gregory Szorc authored
      This logic - including the experimental bundle.reorder option -
      was originally added in a8e3931e3fb5 in 2011 and then later ported
      to changegroup.py.
      
      The intent of this option and associated logic is to control
      the ordering of revisions in deltagroups in changegroups. At the
      time it was implemented, only changegroup version 1 existed
      and generaldelta revlogs were just coming into the world. Changegroup
      version 1 requires that deltas be made against the last revision
      sent over the wire. Used with generaldelta, this created an
      impedance mismatch of sorts and resulted in changegroup producers
      spending a lot of time recomputing deltas.
      
      Revision reordering was introduced so outgoing revisions would be
      sent in "generaldelta order" and producers would be able to
      reuse internal deltas from storage.
      
      Later on, we introduced changegroup version 2. It supported denoting
      which revision a delta was against. So we no longer needed to
      sort outgoing revisions to ensure optimal delta generation from the
      producer. So, subsequent changegroup versions disabled reordering.
      
      We also later made the changelog not store deltas by default. And
      we also made the changelog send out deltas in storage order. Why we
      do this for changelog, I'm not sure. Maybe we want to preserve revision
      order across clones? It doesn't really matter for this commit.
      
      Fast forward to 2018. We want to abstract storage backends. And having
      changegroup code require knowledge about how deltas are stored
      internally interferes with that goal.
      
      This commit removes reordering control from changegroup generation.
      After this commit, the reordering behavior is:
      
      * The changelog is always sent out in storage order (no behavior
        change).
      * Non-changelog generaldelta revlogs are reordered to always be in DAG
        topological order (previously, generaldelta revlogs would be emitted
        in storage order for version 2 and 3 changegroups).
      * Non-changelog non-generaldelta revlogs are sent in storage order (no
        behavior change).
      * There exists no config option to override behavior.
      
      The big difference here is that generaldelta revlogs now *always* have
      their revisions sorted in DAG order before going out over the wire. This
      behavior was previously only done for changegroup version 1. Version 2
      and version 3 changegroups disabled reordering because the interchange
      format supported encoding arbitrary delta parents, so reordering wasn't
      strictly necessary.
      
      I can think of a few significant implications for this change.
      
      Because changegroup receivers will now see non-changelog revisions
      in DAG order instead of storage order, the internal storage order of
      manifests and files may differ substantially between producer and
      consumer. I don't think this matters that much, since the storage
      order of manifests and files is largely hidden from users. Only
      the storage order of changelog matters (because `hg log` shows the
      changelog in storage order). I don't think there should be any
      controversy here.
      
      The reordering of revisions has implications for changegroup producers.
      Previously, generaldelta revlogs would be emitted in storage order.
      And in the common case, the internally-stored delta could effectively
      be copied from disk into the deltagroup delta. This meant that emitting
      delta groups for generaldelta revlogs would be mostly linear read I/O.
      This is desirable for performance. With us now reordering generaldelta
      revlog revisions in DAG order, the read operations may use more random
      I/O instead of sequential I/O. This could result in performance
      loss. But with the prevalence of SSDs and fast random I/O, I'm not
      too worried. (Note: the optimal emission order for revlogs is actually
      delta encoding order. But the changegroup code wasn't doing that before
      or after this change. We could potentially implement that in a later
      commit.)
      
      Changegroups in DAG order will have implications for receivers.
      Previously, receiving storage order might mean seeing a number of
      interleaved branches. This would mean long delta chains, sparse
      I/O, and possibly more fulltext revisions instead of deltas, blowing
      up storage storage. (This is the same set of problems that sparse
      revlogs aims to address.) With the producer now sending revisions in DAG
      order, the receiver also stores revisions in DAG order. That means
      revisions for the same DAG branch are all grouped together. And this
      should yield better storage outcomes. In other words, sending the
      reordered changegroup allows the receiver to have better storage
      order and for the producer to not propagate its (possibly sub-optimal)
      internal storage order.
      
      On the mozilla-unified repository, this change influences bundle
      generation:
      
      $ hg bundle -t none-v2 -a
      before: time: real 355.680 secs (user 256.790+0.000 sys 16.820+0.000)
      after:  time: real 382.950 secs (user 281.700+0.000 sys 17.690+0.000)
      
      before: 7,150,228,967 bytes (uncompressed)
      after:  7,041,556,273 bytes (uncompressed)
      
      before: 1,669,063,234 bytes (zstd l=3)
      after:  1,628,598,830 bytes (zstd l=3)
      
      $ hg unbundle
      before: time: real 511.910 secs (user 466.750+0.000 sys 32.680+0.000)
      after:  time: real 487.790 secs (user 443.940+0.000 sys 30.840+0.000)
      
      00manifest.d size:
      source: 274,924,292 bytes
      before: 304,741,626 bytes
      after:  245,252,087 bytes
      
      .hg/store total file size:
      source: 2,649,133,490
      before: 2,680,888,130
      after:  2,627,875,673
      
      We see the bundle size drop. That's probably because if a revlog
      internally isn't storing a delta, it will choose to delta against
      the last emitted revision. And on repos with interleaved branches
      (like mozilla-unified), the previous revision could be an unrelated
      branch and therefore be a large delta. But with this patch, the
      previous revision is likely p1 or p2 and a delta should be small.
      
      We also see the manifest size drop by ~50 MB. It's worth noting that
      the manifest actually *increased* in size by ~25 MB in the old
      strategy and decreased ~25 MB from its source in the new strategy.
      Again, my explanation for this is that the DAG ordering in the
      changegroup is resulting in better grouping of revisions in the
      receiver, which results in more compact delta chains and higher
      storage efficiency.
      
      Unbundle time also dropped. I suspect this is due to the revlog having
      to work less to compute deltas since the incoming deltas are more
      optimal. i.e. the receiver spends less time resolving fulltext
      revisions as incoming deltas bounce around between DAG branches and
      delta chains.
      
      We also see bundle generation time increase. This is not desirable.
      However, the regression is only significant on the original repository:
      if we generate a bundle from the repository created from the new,
      always reordered bundles, we're close to baseline (if not at it with
      expected noise):
      
      $ hg bundle -t none-v2 -a
      before (original): time: real 355.680 secs (user 256.790+0.000 sys 16.820+0.000)
      after (original):  time: real 382.950 secs (user 281.700+0.000 sys 17.690+0.000)
      after (new repo):  time: real 362.280 secs (user 260.300+0.000 sys 17.700+0.000)
      
      This regression is a bit worrying because it will impact serving
      canonical repositories (that don't have optimal internal storage
      unless they are reordered - possibly as part of running
      `hg debugupgraderepo`). However, this regression will only be
      noticed by very large changegroups. And I'm guessing/hoping that
      any repository that large is using clonebundles to mitigate server
      load.
      
      Again, sending DAG order isn't the optimal send order for servers:
      sending in storage-delta order is. But in order to enable
      storage-optimal send order, we'll need a storage API that handles
      sorting. Future commits will introduce such an API.
      
      Differential Revision: https://phab.mercurial-scm.org/D4721
      db5501d93bcf
  14. Jul 27, 2018
  15. Jul 20, 2018
    • Boris Feld's avatar
      config: rename `revlog` section into `storage` · ae17555ef93f
      Boris Feld authored
      The idea was suggested by Gregory Szorc on IRC. It is more generic and seems
      better. It is probably best to rename the section before it ever makes into an
      official (non-rc) release.
      
      The only config option currently in this section have been prefixed with
      `revlog` to clarify it applies to `revlog` related storage.
      ae17555ef93f
  16. Jul 19, 2018
    • Boris Feld's avatar
      aggressivemergedelta: document rename and move to `revlog` section · 913ca175c4ae
      Boris Feld authored
      The config does not follow our naming guideline and "Aggressive" is probably a
      word to keep away from users.
      
      The option does not truly fit in the `format` section. It can be turned on and
      off for existing repository without much consequence regarding compatibility.
      
      A new `revlog` option is created to control behavior related to revlog writing
      and reading. We can see multiple other config options that could be migrated
      there.
      
      * format.maxchainlen
      * experimental.mmapindexthreshold
      * experimental.sparse-read.density-threshold (in an updated form)
      * experimental.sparse-read.min-gap-size (in an updated form)
      
      In addition, we can foresee at least a couple of sparse-revlog related option
      coming too (to reduce delta chain length and increase snapshot reuse)
      
      These two extra options might fit there too. Unless we want to create a
      section dedicated to caches and performance.
      
      * format.chunkcachesize
      * format.manifestcachesize
      
      For now, we only migrate `optimize-delta-parent-choice` since it is getting
      out of experimental. It is too close to the release to move the other one. In
      addition, we still lack proper the prioritization of alias that would help
      renaming them without bad consequence for users.
      
      (Not fully happy about the `revlog` name but could not find better).
      913ca175c4ae
  17. Jun 21, 2018
    • Paul Morelle's avatar
      debugdeltachain: avoid division by zero when a chain is empty · 93313f66b69b
      Paul Morelle authored
      The two ratios chainratio and extraratio are computed using dividers
      that may be zero when the file is empty.
      As the denominators are integers, the limit of the ratio "just before zero" is
      the numerator value itself.
      If the numerator itself is zero, the ratio value is still meaningful: in both
      cases, a "good" value is a low ratio, and a size of zero is the optimal case.
      93313f66b69b
    • Paul Morelle's avatar
      aggressivemergedeltas: enabled the option by default · 44f5acfb9ad2
      Paul Morelle authored
      The option has been around for a while (August 2015) but was never turned on by
      default. In-depth testing shows large wins for having that on with no
      significant drawbacks.
      
      When enabled, revlog consider delta against both p1 and p2 at the same time when
      storing a revision. Selecting a delta against "p2" can produce better deltas and
      chain. This raise large benefit for all repositories, especially if they have a
      lot of merges.
      
      Comparison of `.hg/store/` size:
      
          mercurial (6.74% merges):
              before:    54,225,348 bytes
              after:     47,279,959 bytes -13%
          pypy (8.30% merges):
              before:   459,041,759 bytes
              after:    346,090,067 bytes -25%
          netbeans (34.21% merges):
              before: 2,468,041,333 bytes
              after:  1,364,077,645 bytes -45%
          mozilla-central (4.84% merges):
              before: 2,731,799,546 bytes
              after:  2,157,718,019 bytes -21%
      
      Comparison of `00manifest.d` size:
      
          mercurial (6.74% merges):
              before:    11,682,516 bytes
              after:      6,143,044 bytes -47%
          pypy (8.30% merges):
              before:   156,447,163 bytes
              after:     52,941,780 bytes -66%
          netbeans (34.21% merges):
              before: 1,250,363,851 bytes
              after:    130,088,982 bytes -90%
          mozilla-central (4.84% merges):
              before:   468,202,733 bytes
              after:    215,096,339 bytes -54%
      
      In addition, the better deltas help with the performance of multiple core
      operations. However, better chains mean longer chains, which can affect
      performance negatively (mostly manifest revision retrieval time). Chains length
      is a deeper problem that also affects linear repository too. Overall we think
      the benefits of using p2 as a diff target are bigger than the downsizes. In
      addition, we are also working on ways to improve the performance impact of chain
      length, so theses downsizes get fixed in the future.
      
      Below are interesting items from the full benchmark run:
      
          bundling 100 revisions from pypy:
              before: 670ms
              after:  480ms -28%
          bundle 10000 revisions from pypy:
              before: 1.38s
              after:  1.10s -54%
          bundle 10000 revisions from pypy:
              before: 16.1s
              after:  7.81s -52%
          bundle 10000 revisions from netbeans:
              before: 19.3s
              after:  15.5s -19%
      
          unbundle 1000 revisions to pypy:
              before: 641ms
              after:  315ms - 51%
      
          clone mercurial (http):
              before: 26.0s
              after:  22.6s -23%
      
          pulling 1000 revisions from pypy (shh):
              before: 2.07s
              after:  1.36s -44%
      
          pushing 1000 revision through http (pypy repository)
              before: 2.18s
              after:  1.35s -48%
      
          diff time in mozilla-central:
              before: 1.420s
              after:  0.983s -31%
      
          status time in mozilla-central:
              before: 1.260s
              after:  0.828s -34%
      
      Impact in other cases seems minimal (within a couple of percent in worse cases)
      and can be seen in both direction:
      
      Timing for a simple `hg commit`:
      
          mozilla-central:
              before: 3.37s
              after:  3.22s -4%
          pypy:
              before:  194ms
              after:   197ms +2%
      
      Timing for status (from tip to parent of tip):
          mercurial:
              before: 52.4ms
              after:  52.4ms (same)
          pypy:
              before: 55.2
              after:  56.9 +3%
      
      Timing for `hg update`
      
          mozilla-central, across 10 revisions:
              before: 4.82s
              after:  4.59s -5%
          mozilla-central, across 10000 revisions:
              before: 49.1s
              after:  49.9s +2%
          pypy, across 10 revisions:
              before: 213ms
              after:  216ms +1%
          pypy, across 10000 revisions:
              before: 5.31ms
              after:  5.24ms -1%
      
      The negative consequences are related to manifest fetch time:
      (timing for the tip revision tested by the benchmark)
      
          pypy-2018:
              before: 2.60ms
              after:  3.88ms +50%
          mozilla-central-2018:
              before: 565ms
              after:  652ms +15% (~+100ms)
          netbeans-2018:
              before: 101ms
              after:  250ms +48% (~+150ms)
      
      This shows up as a fixed overhead on some command we benchmarked:
      
          no-op push of mozilla-central:
              before:  945ms
              after:  1040ms +10% (~+100ms)
          pushing 10 changeset in netbeabs over ssh:
              before:  557ms
              after:   712ms +28% (+155ms)
          pushing 100 changeset in netbeabs over ssh:
              before: 592ms
              after:  771ms +30% (+179ms)
      44f5acfb9ad2
  18. May 08, 2018
  19. Apr 03, 2018
  20. Apr 02, 2018
  21. Feb 21, 2018
    • Boris Feld's avatar
      bundle: include advisory rev branch cache part in bundle2 bundle · b89a7ef29013
      Boris Feld authored
      `hg bundle` command producing bundle2 will now include an optional part
      containing the revision-branch cache data.
      
      The data sent are mostly nodes so it is quite compact.  The goal of the
      rev-branch-cache is to speed up branch map computation, especially when the
      branchmap gets invalidated so we send data for all exchanged changesets. In
      addition, computing the relevant heads to send in case of partial pulling would
      be challenging.
      
      As a reminder, the rev branch cache data significantly speed up branch
      computation. Having it around provides a small speedup to pull/clone and much
      higher tolerance to branch map cache invalidation that might happens from later
      commands.
      
      On the Mercurial repository, computing the visible branchmap from scratch move
      from 2.00 seconds to 0.34s (a -83% speedup).
      
      Using this new part, Unbundling the full Mercurial repository moves from 25.736
      seconds to 24.030 seconds (around -7% speedup). The bundle size increase is
      around 3% (from 22.43 MB to 23.13MB)
      
      
      On an half a million revision repository with twenty thousand
      branches, computing the branchmap moves from 75 seconds to 45 second (-40%) if
      the caches is used.
      
      A bundle containing 50 000 changesets in such repository get a 0.5% size
      increase from such part for a -3% unbundling time speedup.
      b89a7ef29013
  22. Dec 11, 2017
    • Matt Harbison's avatar
      tests: remove (glob) annotations that were only for '\' matches · 4441705b7111
      Matt Harbison authored
      # skip-blame because this was mechanically rewritten the following script.  I
      ran it on both *.t and *.py, but none of the *.py changes were proper.  All *.t
      ones appear to be, and they run without addition failures on both Windows and
      Linux.
      
        import argparse
        import os
        import re
      
        ap = argparse.ArgumentParser()
        ap.add_argument('path', nargs='+')
        opts = ap.parse_args()
      
        globre = re.compile(r'^(.*) \(glob\)(.*)$')
      
        for p in opts.path:
            tmp = p + '.tmp'
            with open(p, 'rb') as src, open(tmp, 'wb') as dst:
                for line in src:
                    m = globre.match(line)
                    if not m or '$LOCALIP' in line or '*' in line:
                        dst.write(line)
                        continue
                    if '?' in line[:-3] or ('?' in line[:-3] and line[-3:] != '(?)'):
                        dst.write(line)
                        continue
                    dst.write(m.group(1) + m.group(2) + '\n')
            os.unlink(p)
            os.rename(tmp, p)
      4441705b7111
  23. Oct 12, 2017
    • Denis Laxalde's avatar
      transaction-summary: show the range of new revisions upon pull/unbundle (BC) · eb586ed5d8ce
      Denis Laxalde authored
      Upon pull or unbundle, we display a message with the range of new revisions
      fetched. This revision range could readily be used after a pull to look out
      what's new with 'hg log'. The algorithm takes care of filtering "obsolete"
      revisions that might be present in transaction's "changes" but should not be
      displayed to the end user.
      eb586ed5d8ce
  24. Aug 23, 2017
  25. Jul 05, 2017
  26. Jun 23, 2017
    • Pierre-Yves David's avatar
      revlog: add an experimental option to mitigated delta issues (issue5480) · 895ecec31c70
      Pierre-Yves David authored
      The general delta heuristic to select a delta do not scale with the number of
      branch. The delta base is frequently too far away to be able to reuse a chain
      according to the "distance" criteria. This leads to insertion of larger delta (or
      even full text) that themselves push the bases for the next delta further away
      leading to more large deltas and full texts. This full text and frequent
      recomputation throw Mercurial performance in disarray.
      
      For example of a slightly large repository
      
        280 000 files (2 150 000 versions)
        430 000 changesets (10 000 topological heads)
      
      Number below compares repository with and without the distance criteria:
      
      manifest size:
          with:    21.4 GB
          without:  0.3 GB
      
      store size:
          with:    28.7 GB
          without   7.4 GB
      
      bundle last 15 00 revisions:
          with:    800 seconds
                   971 MB
          without:  50 seconds
                    73 MB
      
      unbundle time (of the last 15K revisions):
          with:    1150 seconds (~19 minutes)
          without:   35 seconds
      
      Similar issues has been observed in other repositories.
      
      
      Adding a new option or "feature" on stable is uncommon. However, given that this
      issues is making Mercurial practically unusable, I'm exceptionally targeting
      this patch for stable.
      
      What is actually needed is a full rework of the delta building and reading
      logic. However, that will be a longer process and churn not suitable for stable.
      
      In the meantime, we introduces a quick and dirty mitigation of this in the
      'experimental' config space. The new option introduces a way to set the maximum
      amount of memory usable to store a diff in memory. This extend the ability for
      Mercurial to create chains without removing all safe guard regarding memory
      access. The option should be phased out when core has a more proper solution
      available.
      
      Setting the limit to '0' remove all limits, setting it to '-1' use the default
      limit (textsize x 4).
      895ecec31c70
  27. Jul 04, 2017
  28. Jun 23, 2017
    • Pierre-Yves David's avatar
      revlog: add an experimental option to mitigated delta issues (issue5480) · e9d325cfe071
      Pierre-Yves David authored
      The general delta heuristic to select a delta do not scale with the number of
      branch. The delta base is frequently too far away to be able to reuse a chain
      according to the "distance" criteria. This leads to insertion of larger delta (or
      even full text) that themselves push the bases for the next delta further away
      leading to more large deltas and full texts. This full text and frequent
      recomputation throw Mercurial performance in disarray.
      
      For example of a slightly large repository
      
        280 000 files (2 150 000 versions)
        430 000 changesets (10 000 topological heads)
      
      Number below compares repository with and without the distance criteria:
      
      manifest size:
          with:    21.4 GB
          without:  0.3 GB
      
      store size:
          with:    28.7 GB
          without   7.4 GB
      
      bundle last 15 00 revisions:
          with:    800 seconds
                   971 MB
          without:  50 seconds
                    73 MB
      
      unbundle time (of the last 15K revisions):
          with:    1150 seconds (~19 minutes)
          without:   35 seconds
      
      Similar issues has been observed in other repositories.
      
      
      Adding a new option or "feature" on stable is uncommon. However, given that this
      issues is making Mercurial practically unusable, I'm exceptionally targeting
      this patch for stable.
      
      What is actually needed is a full rework of the delta building and reading
      logic. However, that will be a longer process and churn not suitable for stable.
      
      In the meantime, we introduces a quick and dirty mitigation of this in the
      'experimental' config space. The new option introduces a way to set the maximum
      amount of memory usable to store a diff in memory. This extend the ability for
      Mercurial to create chains without removing all safe guard regarding memory
      access. The option should be phased out when core has a more proper solution
      available.
      
      Setting the limit to '0' remove all limits, setting it to '-1' use the default
      limit (textsize x 4).
      e9d325cfe071
  29. Jun 15, 2017
    • Martin von Zweigbergk's avatar
      strip: include phases in bundle (BC) · 8e3021fd1a44
      Martin von Zweigbergk authored
      Before this patch, unbundling a stripped changeset would make it a
      draft (unless the parent was secret). This meant that one would lose
      phase information when stripping and unbundling secret changesets. The
      same thing was true for public changesets. While stripping public
      changesets is generally rare, it's done frequently by e.g. the
      narrowhg extension.
      
      We also include the phases in the temporary bundle, just in case
      stripping were to fail after that point, so the user can still restore
      the repo including phase information. Before this patch, the phases
      were left untouched during the bundling and unbundling of the
      temporary bundle. Only at the end of the transaction would
      phasecache.filterunknown() be called to remove phase roots that were
      no longer valid. We now need to call that also after the first
      stripping, i.e. before applying the temporary bundle. Otherwise
      unbundling the temporary bundle will cause a read of the phase cache
      which has stripped changesets in the cache and that fails.
      
      Like with obsmarkers, we unconditionally include the phases in the
      bundle when stripping (when using bundle2, such as when generaldelta
      is enabled). The reason for doing that for strip but not for bundle is
      that strip bundles are not meant to be shared outside the repo, so we
      don't care as much about compatibility.
      8e3021fd1a44
  30. Oct 17, 2016
  31. Jul 17, 2016
    • Gregory Szorc's avatar
      bundle2: store changeset count when creating file bundles · 953839de96ab
      Gregory Szorc authored
      The bundle2 changegroup part has an advisory param saying how many
      changesets are in the part. Before this patch, we were setting
      this part when generating bundle2 parts via the wire protocol but
      not when generating local bundle2 files.
      
      A side effect of not setting the changeset count part is that progress
      bars don't work when applying changesets. As the tests show, this
      impacted clone bundles, shelve, backup bundles, `hg unbundle`, and
      anything touching bundle2 files.
      
      This patch adds a backdoor to allow us to pass state from
      changegroup generation into the unbundler. We store the number
      of changesets in the changegroup in this state and use it to
      populate the aforementioned advisory part parameter when generating
      the bundle2 bundle.
      
      I concede that I'm not thrilled by how state is being passed in
      changegroup.py (it feels a bit hacky). I would love to overhaul the
      rather confusing set of functions in changegroup.py with something that
      passes rich objects around instead of e.g. low-level generators.
      However, given the code freeze for 3.9 is imminent, I'd rather not
      undertake this endeavor right now. This feels like the easiest way
      to get the parameter added to the changegroup part.
      953839de96ab
    • Gregory Szorc's avatar
      util: implement a deterministic __repr__ on sortdict · 37cccad55410
      Gregory Szorc authored
      `hg debugbundle` is calling repr() on bundle2 part params, which are
      now util.sortdict instances. Unfortunately, repr() doesn't appear
      to be deterministic for util.sortdict. So, we implement one.
      
      We include the type name because that's the common convention for
      __repr__ implementations. Having the type name in `hg debugbundle`
      is a bit ugly. But it's a debug command and I don't care enough to
      fix it.
      37cccad55410
  32. Jul 03, 2016
  33. Dec 05, 2015
    • Martin von Zweigbergk's avatar
      revlog: don't consider nullrev when choosing delta base · d9bfe6289acf
      Martin von Zweigbergk authored
      In the most complex case, we try using the incoming delta base, then
      we try both parents, and then we try the previous revlog entry. If
      none of these result in a good delta, we natually use the null
      revision as base. However, we sometimes consider the nullrev before we
      have exhausted our other options. Specifically, when both parents are
      null, we use the nullrev as delta base if it produces a good delta
      (according to _isgooddelta()), and we fail to try the previous revlog
      entry as delta base. After 20a9226bdc8a (addrevision: use general
      delta when the incoming base delta is bad, 2015-12-01), it can also
      happen for non-merge commits when the incoming delta is not good.
      
      The Firefox repo (from many months back) shrinks a tiny bit with this
      patch: from 1.855GB to 1.830GB (1.4%). The hg repo itself shrinks even
      less: by less than 0.1%. There may be repos that get larger instead.
      
      This undoes the unexplained test change in 20a9226bdc8a.
      d9bfe6289acf
  34. Dec 02, 2015
    • Pierre-Yves David's avatar
      addrevision: use general delta when the incoming base delta is bad · 20a9226bdc8a
      Pierre-Yves David authored
      We unify the delta selection process to be a simple three options process:
      
      - try to use the incoming delta (if lazydeltabase is on)
      - try to find a suitable parents to delta against (if gd is on)
      - try to delta against the tipmost revision
      
      The first of this option that yield a valid delta will be used.
      
      The test change in 'test-generaldelta.t' show this behavior as we use a delta
      against the parent instead of a full delta when the incoming delta is not
      suitable.
      
      This as some impact on 'test-bundle.t' because a delta somewhere changes. It
      does not seems to change the test semantic and have been ignored.
      20a9226bdc8a
    • Pierre-Yves David's avatar
      test: use a bigger manifest in general delta test · 762fbd28e7df
      Pierre-Yves David authored
      The currently used manifest is too small and cannot sustain a chain length
      above "1".  This make testing the 'lazybasedelta' behavior hard. So we add an
      extra file in the manifest to help testing in the next changeset.
      
      The semantic of existing tests have been checked and is not changed.
      762fbd28e7df
Loading