Skip to content
Snippets Groups Projects
  1. Jun 20, 2017
  2. May 02, 2017
    • Pierre-Yves David's avatar
      caches: stop warming the cache after changegroup application · 24f55686a63d
      Pierre-Yves David authored
      Now that we garantee that branchmap cache is updated at the end of the
      transaction we can drop this update. This removes a problematic case with
      nested transaction where the new cache could be written on disk before the
      transaction is finished (and even roll-backed)
      
      Such premature cache write was visible in the following test:
      
      * tests/test-acl.t
      * tests/test-rebase-conflicts.t
      
      In addition, running the cache update later means having more date about the
      state of the repository (in particular: phases). So we can generate caches with
      more information. This creates harmless changes to the following tests:
      
      * tests/test-hardlinks-whitelisted.t
      * tests/test-hardlinks.t
      * tests/test-phases.t
      * tests/test-tags.t
      * tests/test-inherit-mode.t
      24f55686a63d
    • Pierre-Yves David's avatar
      caches: move the 'updating the branch cache' message in 'updatecaches' · c2380b448265
      Pierre-Yves David authored
      We are about to remove the branchmap cache update in changegroup application.
      There is a debug message alongside this update that we do not want to loose. We
      move the message beforehand to simplify the test update in the next changeset.
      The message move is quite noisy and isolating that noise is useful.
      
      Most tests update are just line reordering since the message is issued at a
      later point during the transaction.
      
      After this changes, the message is displayed in more case since local commit
      creation also issue it.
      c2380b448265
  3. Oct 13, 2016
    • Pierre-Yves David's avatar
      changegroup: skip delta when the underlying revlog do not use them · 6b0741d6d234
      Pierre-Yves David authored
      Revlog can now be configured to store full snapshot only. This is used on the
      changelog. However, the changegroup packing was still recomputing deltas to be
      sent over the wire.
      
      We now just reuse the full snapshot directly in this case, skipping delta
      computation. This provides use with a large speed up(-30%):
      
      # perfchangegroupchangelog on mercurial
      ! wall 2.010326 comb 2.020000 user 2.000000 sys 0.020000 (best of 5)
      ! wall 1.382039 comb 1.380000 user 1.370000 sys 0.010000 (best of 8)
      
      # perfchangegroupchangelog on pypy
      ! wall 5.792589 comb 5.780000 user 5.780000 sys 0.000000 (best of 3)
      ! wall 3.911158 comb 3.920000 user 3.900000 sys 0.020000 (best of 3)
      
      # perfchangegroupchangelog on mozilla central
      ! wall 20.683727 comb 20.680000 user 20.630000 sys 0.050000 (best of 3)
      ! wall 14.190204 comb 14.190000 user 14.150000 sys 0.040000 (best of 3)
      
      Many tests have to be updated because of the change in bundle content. All
      theses update have been verified.  Because diffing changelog was not very
      valuable, the resulting bundle have similar size (often a bit smaller):
      
      # full bundle of mozilla central
      with delta:    1142740533B
      without delta: 1142173300B
      
      So this is a win all over the board.
      6b0741d6d234
    • Gregory Szorc's avatar
      changelog: disable delta chains · b7a966ce89ed
      Gregory Szorc authored
      This patch disables delta chains on changelogs. After this patch, new
      entries on changelogs - including existing changelogs - will be stored
      as the fulltext of that data (likely compressed). No delta computation
      will be performed.
      
      An overview of delta chains and data justifying this change follows.
      
      Revlogs try to store entries as a delta against a previous entry (either
      a parent revision in the case of generaldelta or the previous physical
      revision when not using generaldelta). Most of the time this is the
      correct thing to do: it frequently results in less CPU usage and smaller
      storage.
      
      Delta chains are most effective when the base revision being deltad
      against is similar to the current data. This tends to occur naturally
      for manifests and file data, since only small parts of each tend to
      change with each revision. Changelogs, however, are a different story.
      
      Changelog entries represent changesets/commits. And unless commits in a
      repository are homogonous (same author, changing same files, similar
      commit messages, etc), a delta from one entry to the next tends to be
      relatively large compared to the size of the entry. This means that
      delta chains tend to be short. How short? Here is the full vs delta
      revision breakdown on some real world repos:
      
      Repo             % Full    % Delta   Max Length
      hg                45.8       54.2        6
      mozilla-central   42.4       57.6        8
      mozilla-unified   42.5       57.5       17
      pypy              46.1       53.9        6
      python-zstandard  46.1       53.9        3
      
      (I threw in python-zstandard as an example of a repo that is homogonous.
      It contains a small Python project with changes all from the same
      author.)
      
      Contrast this with the manifest revlog for these repos, where 99+% of
      revisions are deltas and delta chains run into the thousands.
      
      So delta chains aren't as useful on changelogs. But even a short delta
      chain may provide benefits. Let's measure that.
      
      Delta chains may require less CPU to read revisions if the CPU time
      spent reading smaller deltas is less than the CPU time used to
      decompress larger individual entries. We can measure this via
      `hg perfrevlog -c -d 1` to iterate a revlog to resolve each revision's
      fulltext. Here are the results of that command on a repo using delta
      chains in its changelog and on a repo without delta chains:
      
      hg (forward)
      ! wall 0.407008 comb 0.410000 user 0.410000 sys 0.000000 (best of 25)
      ! wall 0.390061 comb 0.390000 user 0.390000 sys 0.000000 (best of 26)
      
      hg (reverse)
      ! wall 0.515221 comb 0.520000 user 0.520000 sys 0.000000 (best of 19)
      ! wall 0.400018 comb 0.400000 user 0.390000 sys 0.010000 (best of 25)
      
      mozilla-central (forward)
      ! wall 4.508296 comb 4.490000 user 4.490000 sys 0.000000 (best of 3)
      ! wall 4.370222 comb 4.370000 user 4.350000 sys 0.020000 (best of 3)
      
      mozilla-central (reverse)
      ! wall 5.758995 comb 5.760000 user 5.720000 sys 0.040000 (best of 3)
      ! wall 4.346503 comb 4.340000 user 4.320000 sys 0.020000 (best of 3)
      
      mozilla-unified (forward)
      ! wall 4.957088 comb 4.950000 user 4.940000 sys 0.010000 (best of 3)
      ! wall 4.660528 comb 4.650000 user 4.630000 sys 0.020000 (best of 3)
      
      mozilla-unified (reverse)
      ! wall 6.119827 comb 6.110000 user 6.090000 sys 0.020000 (best of 3)
      ! wall 4.675136 comb 4.670000 user 4.670000 sys 0.000000 (best of 3)
      
      pypy (forward)
      ! wall 1.231122 comb 1.240000 user 1.230000 sys 0.010000 (best of 8)
      ! wall 1.164896 comb 1.160000 user 1.160000 sys 0.000000 (best of 9)
      
      pypy (reverse)
      ! wall 1.467049 comb 1.460000 user 1.460000 sys 0.000000 (best of 7)
      ! wall 1.160200 comb 1.170000 user 1.160000 sys 0.010000 (best of 9)
      
      The data clearly shows that it takes less wall and CPU time to resolve
      revisions when there are no delta chains in the changelogs, regardless
      of the direction of traversal. Furthermore, not using a delta chain
      means that fulltext resolution in reverse is as fast as iterating
      forward. So not using delta chains on the changelog is a clear CPU win
      for reading operations.
      
      An example of a user-visible operation showing this speed-up is revset
      evaluation. Here are results for
      `hg perfrevset 'author(gps) or author(mpm)'`:
      
      hg
      ! wall 1.655506 comb 1.660000 user 1.650000 sys 0.010000 (best of 6)
      ! wall 1.612723 comb 1.610000 user 1.600000 sys 0.010000 (best of 7)
      
      mozilla-central
      ! wall 17.629826 comb 17.640000 user 17.600000 sys 0.040000 (best of 3)
      ! wall 17.311033 comb 17.300000 user 17.260000 sys 0.040000 (best of 3)
      
      What about 00changelog.i size?
      
      Repo                Delta Chains     No Delta Chains
      hg                    7,033,250         6,976,771
      mozilla-central      82,978,748        81,574,623
      mozilla-unified      88,112,349        86,702,162
      pypy                 20,740,699        20,659,741
      
      The data shows that removing delta chains from the changelog makes the
      changelog smaller.
      
      Delta chains are also used during changegroup generation. This
      operation essentially converts a series of revisions to one large
      delta chain. And changegroup generation is smart: if the delta in
      the revlog matches what the changegroup is emitting, it will reuse
      the delta instead of recalculating it. We can measure the impact
      removing changelog delta chains has on changegroup generation via
      `hg perfchangegroupchangelog`:
      
      hg
      ! wall 1.589245 comb 1.590000 user 1.590000 sys 0.000000 (best of 7)
      ! wall 1.788060 comb 1.790000 user 1.790000 sys 0.000000 (best of 6)
      
      mozilla-central
      ! wall 17.382585 comb 17.380000 user 17.340000 sys 0.040000 (best of 3)
      ! wall 20.161357 comb 20.160000 user 20.120000 sys 0.040000 (best of 3)
      
      mozilla-unified
      ! wall 18.722839 comb 18.720000 user 18.680000 sys 0.040000 (best of 3)
      ! wall 21.168075 comb 21.170000 user 21.130000 sys 0.040000 (best of 3)
      
      pypy
      ! wall 4.828317 comb 4.830000 user 4.820000 sys 0.010000 (best of 3)
      ! wall 5.415455 comb 5.420000 user 5.410000 sys 0.010000 (best of 3)
      
      The data shows eliminating delta chains makes the changelog part of
      changegroup generation slower. This is expected since we now have to
      compute deltas for revisions where we could recycle the delta before.
      
      It is worth putting this regression into context of overall changegroup
      times. Here is the rough total CPU time spent in changegroup generation
      for various repos while using delta chains on the changelog:
      
      Repo              CPU Time (s)    CPU Time w/ compression
      hg                  4.50              7.05
      mozilla-central   111.1             222.0
      pypy               28.68             75.5
      
      Before compression, removing delta chains from the changegroup adds
      ~4.4% overhead to hg changegroup generation, 1.3% to mozilla-central,
      and 2.0% to pypy. When you factor in zlib compression, these percentages
      are roughly divided by 2.
      
      While the increased CPU usage for changegroup generation is unfortunate,
      I think it is acceptable because the percentage is small, server
      operators (those likely impacted most by this) have other mechanisms
      to mitigate CPU consumption (namely reducing zlib compression level and
      pre-generated clone bundles), and because there is room to optimize this
      in the future. For example, we could use the nullid as the base revision,
      effectively encoding the full revision for each entry in the changegroup.
      When doing this, `hg perfchangegroupchangelog` nearly halves:
      
      mozilla-unified
      ! wall 21.168075 comb 21.170000 user 21.130000 sys 0.040000 (best of 3)
      ! wall 11.196461 comb 11.200000 user 11.190000 sys 0.010000 (best of 3)
      
      This looks very promising as a future optimization opportunity.
      
      It's worth that the changes in test-acl.t to the changegroup part size.
      This is because revision 6 in the changegroup had a delta chain of
      length 2 before and after this patch the base revision is nullrev.
      When the base revision is nullrev, cg2packer.deltaparent() hardcodes
      the *previous* revision from the changegroup as the delta parent.
      This caused the delta in the changegroup to switch base revisions,
      the delta to change, and the size to change accordingly. While the
      size increased in this case, I think sizes will remain the same
      on average, as the delta base for changelog revisions doesn't matter
      too much (as this patch shows). So, I don't consider this a regression.
      b7a966ce89ed
  4. Aug 09, 2016
  5. Aug 02, 2016
  6. Jan 13, 2016
  7. Dec 14, 2015
  8. Dec 11, 2015
    • Augie Fackler's avatar
      changegroup: introduce cg3, which has support for exchanging treemanifests · 77d25b913f80
      Augie Fackler authored
      I'm not entirely happy with using a trailing / on a "file" entry for
      transferring a treemanifest. We've discussed putting some flags on
      each file header[0], but I'm unconvinced that's actually any better:
      if we were going to add another feature to the cg format we'd still be
      doing a version bump anyway to cg4, so I'm inclined to not spend time
      coming up with a more sophisticated format until we actually know what
      the next feature we want to stuff in a changegroup will be.
      
      Test changes outside test-treemanifest.t are only due to the new CG3
      bundlecap showing up in the wire protocol.
      
      Many thanks to adgar@google.com and martinvonz@google.com for helping
      me with various odd corners of the changegroup and treemanifest API.
      
      0: It's not hard refactoring, nor is it a lot of work. I'm just
      disinclined to do speculative work when it's not clear what the
      customer would actually be.
      77d25b913f80
  9. Jun 11, 2015
  10. May 28, 2015
  11. Jun 10, 2015
  12. Jun 06, 2015
  13. May 27, 2015
    • Pierre-Yves David's avatar
      push: make pushkey part advisory · d410336fdb3c
      Pierre-Yves David authored
      The current behavior (with bundle1) is to let the rest of the push succeed if
      the pushkey call (phases, bookmarks) failed (this comes from the fact that each
      item is sent in its own command).
      
      We kept this behavior with bundle2, which is highly debatable, but let us keep
      thing as they are now as a start. We are about to enforce 'mandatory' pushkey
      part as 'mandatory' successful, so we need to marks parts as advisory to
      preserve the current (debatable) behavior.
      d410336fdb3c
  14. Jun 02, 2015
    • Gregory Szorc's avatar
      bundle2: part handler for processing .hgtags fnodes mappings · d29201352af7
      Gregory Szorc authored
      .hgtags fnodes cache entries can be expensive to compute, especially
      if there are hundreds of even thousands of them. This patch implements
      support for receiving a bundle2 part that contains a mapping of
      changeset to .hgtags fnodes.
      
      An upcoming patch will teach the server to send this part, allowing
      clients to bypass having to redundantly compute these values.
      
      A number of tests changed due to the client advertising the "hgtagsfnodes"
      capability.
      d29201352af7
  15. May 27, 2015
    • Pierre-Yves David's avatar
      test: use bundle2 in test-acl · 9793e52279a1
      Pierre-Yves David authored
      This test makes extensive use of --debug so moving to bundle2 based exchange
      has a massive impact. We do it early to reduce the noise create by a future
      usage of bundle2 as the default protocol.
      9793e52279a1
  16. May 10, 2015
    • Pierre-Yves David's avatar
      progress: get the extremely verbose output out of default debug · bd625cd4e5e7
      Pierre-Yves David authored
      When the progress extension is not enabled, each call to 'ui.progress' used to
      issue a debug message. This results is a very verbose output and often redundant
      in tests. Dropping it makes tests less volatile to factor they do not meant to
      test.
      
      We had to alter the sed trick in 'test-rename-merge2.t'. Sed is used to drop all
      output from a certain point and hidding the progress output remove its anchor.
      So we anchor on something else.
      bd625cd4e5e7
  17. Sep 09, 2014
    • durin42's avatar
      test-acl: alter sed construct to avoid changes in .hg/hgrc formatting · 38a393d59e77
      durin42 authored
      A future patch is going to add some extra commented-out boilerplate to
      the top of .hg/hgrc during clone. In order to make this test not
      require regular updates, switch to searching for [hooks] or [acl] and
      print file from the first match to that pattern.
      38a393d59e77
  18. Aug 16, 2014
  19. Jul 31, 2014
  20. Apr 15, 2014
    • Pierre-Yves David's avatar
      bundle2: feed a binary stream to `peer.unbundle` · 408877d491fb
      Pierre-Yves David authored
      This input will have to travel over the wire anyway, so we feed the peer method
      with a simple binary stream and rely on the server side to use `readbundle`
      to create the python object.
      
      The test output changes because the bundle is created marginally sooner and the
      debug output interleaves in a different way.
      408877d491fb
  21. Apr 05, 2014
    • Pierre-Yves David's avatar
      localrepo: add unbundle support · 7a679918ee2b
      Pierre-Yves David authored
      Localrepo now supports the unbundle method of pushing changegroups. We
      plan to use the unbundle call for bundle2 so it is important that all
      peers supports it. The `peer.unbundle` and `peer.addchangegroup` code
      path have small difference so cause some test output changes. None of those
      changes seems problematic.
      7a679918ee2b
  22. Jan 15, 2013
    • Pierre-Yves David's avatar
      destroyed: drop complex branchcache rebuilt logic · 904b7109938e
      Pierre-Yves David authored
      The strip code used a trick to lower the cost of branchcache update after a
      strip. However is less necessary since we have branchcache collaboration.
      Invalid branchcache are likely to be cheaply rebuilt again a near subset of the
      repo.
      
      Moreover, this trick would need update to be relevant in the now filtered
      repository world. It currently update the unfiltered branchcache that few people
      cares about. Make it smarter on that aspect would need complexes update of the
      calling logic
      
      
      So this mechanism is:
      - Arguably needed,
      - Currently irrelevant,
      - Hard to update
      and I'm dropping it.
      
      We now update the branchcache in all case by courtesy of the read only reader.
      
      This changeset have a few expected impact on the testsuite are different cache
      are updated.
      904b7109938e
  23. Jan 13, 2013
    • Kevin Bullock's avatar
      filtering: rename filters to their antonyms · f3b21beb9802
      Kevin Bullock authored
      Now that changelog filtering is in place, it's become evident that
      naming the filters according to the set of revs _not_ included in the
      filtered changelog is confusing. This is especially evident in the
      collaborative branch cache scheme.
      
      This changes the names of the filters to reflect the revs that _are_
      included:
      
        hidden -> visible
        unserved -> served
        mutable -> immutable
        impactable -> base
      
      repoview.filteredrevs is renamed to filterrevs, so that callers read a
      bit more sensibly, e.g.:
      
        filterrevs('visible') # filter revs according to what's visible
      f3b21beb9802
  24. Jan 02, 2013
    • Pierre-Yves David's avatar
      clfilter: add mutable filtering · aff706b3a21c
      Pierre-Yves David authored
      It filters all mutable changesets, leaving only public changeset unfiltered.
      This filtering set is expected to be much more stable that the previous one as
      public changeset are unlikely to disapear.
      
      The only official use of this filter is for branchcache.
      aff706b3a21c
  25. Jan 07, 2013
    • Pierre-Yves David's avatar
      branchmap: allow to use cache of subset · a55b06885cda
      Pierre-Yves David authored
      Filtered repository are *subset* of unfiltered repository. This means that a
      filtered branchmap could be use to compute the unfiltered version.
      
      And filtered version happen to be subset of each other:
      - "all() - unserved()" is a subset of "all() - hidden()"
      - "all() - hidden()" is a subset of "all()"
      
      This means that branchmap with "unfiltered" filter can be used as a base for
      "hidden" branchmap that itself could be used as a base for unfiltered
      branchmap.
      
         unserved < hidden < None
      
      This changeset implements this mechanism. If the on disk branchcache is not valid
      we use the branchcache of the nearest subset as base instead of computing it from
      scratch. Such fallback can be cascaded multiple time is necessary.
      
      Note that both "hidden" and "unserved" set are a bit volatile. We will add more
      stable filtering in next changesets.
      
      This changeset enables collaboration between no filtering and "unserved"
      filtering. Fixing performance regression introduced by 47f00b0de337
      a55b06885cda
  26. Jan 02, 2013
    • Pierre-Yves David's avatar
      branchmap: disable fallback to unfiltered branchcache · 47f00b0de337
      Pierre-Yves David authored
      Disables this simple optimisation to allow coming more powerfull approach: cache
      collaboration.
      
      Our goal is to have branchcache collaborate. This means that unfiltered
      branchcache will fallback to some filtered branchcache if invalid. We can't have
      the filtered branchcache to use the unfiltered one. That would loop.
      47f00b0de337
  27. Dec 27, 2012
  28. Dec 04, 2012
  29. Jul 26, 2012
  30. Jul 28, 2012
  31. May 28, 2012
    • Elifarley Callado Coelho Cruz's avatar
      acl: use of "!" prefix in user or group names · c49cf339b5bb
      Elifarley Callado Coelho Cruz authored
      The "!" prefix allows you to prevent anyone except a given user or group to push
      changesets in a given branch or path.
      
      This patch enables a use case suggested by a user (Julien Bonnet):
      There's a branch that only a given user (or group) should be able to push to,
      and you don't want to restrict access to any other branch that may be created.
      
      With the "!" prefix, you simply deny access to "!givenuser" (or "!@givengroup").
      c49cf339b5bb
  32. Jun 01, 2012
  33. Dec 08, 2011
  34. Nov 16, 2011
  35. Oct 06, 2011
  36. Sep 18, 2011
    • Greg Ward's avatar
      rollback: only restore dirstate and branch when appropriate. · 7c26ce9edbd2
      Greg Ward authored
      If the working dir parent was destroyed by rollback, then the old
      behaviour is perfectly reasonable: restore dirstate, branch, and
      bookmarks. That way the working dir moves back to an existing
      changeset rather than becoming an orphan.
      
      But if the working dir parent was unaffected -- say, you updated to an
      older changeset and then did rollback -- then it's silly to restore
      dirstate and branch. So don't do that. Leave the status of the working
      dir alone. (But always restore bookmarks, because that file refers to
      changeset IDs that may have been destroyed.)
      7c26ce9edbd2
  37. Jun 03, 2011
Loading