Skip to content
Snippets Groups Projects
  1. Jun 12, 2019
  2. Jun 14, 2019
  3. Apr 25, 2019
    • Pierre-Yves David's avatar
      deltas: set estimated compression upper bound to "3x" instead of "10x" · 4a3abb33380a
      Pierre-Yves David authored
      In pratice, we very rarely observer compression better than "3x" on manifest
      deltas. Having a more aggressive estimate significantly helps our pathological
      use case on a private repository. Here are a comparison of timings using
      different upper bound.
      
      Estimated compression |    ø   |  ×10 |  ×5  |  ×3  |
      timing                |  14.11 | 2.61 | 1.96 | 1.53 |
      
      
      We also tested the impact of this series on an array of public repositories.
      This shown no impact in either size nor timing.
      
      Full data set below for those interested.
      
      Size
      ----
      
      Regarding size, not significant impact have been noticed on neither public nor
      private repositories. Here are the number we gathered on public repositories:
      
      zlib/upperbound | no            | 10x           | 5x            | 3x
      mercurial       |     5 875 730 |     5 875 730 |     5 875 730 |     5 875 730
      pypy            |    27 782 913 |    27 782 913 |    27 782 913 |    27 782 913
      netbeans        |   159 161 207 |   159 161 207 |   159 161 207 |   159 959 879 (+0.5%)
      mozilla-central |   323 841 642 |   323 841 642 |   323 841 642 |   319 867 519 (-2.5%)
      mozilla-try     |   746 649 123 |   746 649 123 |   746 649 123 |   741 155 568 (-0.7%)
      private-repo    | 1 485 287 294 | 1 485 287 294 | 1 485 287 294 | 1 409 248 382 (-5.1%)
      
      zstd/upperbound | no            | 10x           | 5x            | 3x
      mercurial       |     5 895 206 |     5 895 206 |     5 895 206 |     5 895 206
      pypy            |    28 689 230 |    28 689 230 |    28 689 230 |    28 689 230
      netbeans        |   157 636 387 |   157 636 387 |   157 636 387 |   159 692 678 (+1.3%)
      mozilla-central |   317 650 281 |   317 650 281 |   317 650 281 |   319 613 603 (+0.6%)
      mozilla-try     |   737 555 275 |   737 555 275 |   737 555 275 |   738 079 473 (+0.1%)
      private-repo    | 1 352 362 982 | 1 352 362 982 | 1 346 961 880 | 1 361 327 384 (+0.7%)
      
      
      Speed
      ------
      
      Timing gathered using `hg perfrevlogwrite -m`. Value are in seconds.
      
      mercurial
      
      zlib   | no        | 10x       | 5x        | 3x        |
      total  | 65.551783 | 65.388887 | 65.260658 | 65.321199 |
      max    |  0.034544 |  0.034571 |  0.034659 |  0.034521 |
      99.99% |  0.034544 |  0.034571 |  0.034659 |  0.034521 |
      
      zstd   | no        | 10x       | 5x        | 3x        |
      total  | 49.118449 | 49.054062 | 48.753588 | 48.740230 |
      max    |  0.009338 |  0.009239 |  0.009202 |  0.009178 |
      99.99% |  0.007618 |  0.007639 |  0.007626 |  0.007621 |
      
      pypy
      
      zlib   | no         | 10x        | 5x         | 3x         |
      total  | 560.865984 | 558.983817 | 559.083815 | 559.349152 |
      max    |   0.219614 |   0.215922 |   0.218112 |   0.218107 |
      99.99% |   0.219614 |   0.215922 |   0.218112 |   0.218107 |
      
      zstd   | no         | 10x        | 5x         | 3x         |
      total  | 349.393280 | 347.395819 | 347.185407 | 345.643985 |
      max    |   0.084143 |   0.083536 |   0.081834 |   0.082178 |
      99.99% |   0.039445 |   0.039639 |   0.039612 |   0.039175 |
      
      netbeans
      zlib   | no           | 10x          | 5x           | 3x           |
      total  | 33103.327727 | 33314.932260 | 33211.745233 | 33345.891778 |
      max    |     2.666852 |     2.672059 |     2.662453 |     2.662936 |
      99.99% |     2.058772 |     2.070429 |     2.069569 |     2.064653 |
      
      zstd   | no           | 10x         | 5x            | 3x           |
      total  | 20112.102708 | 20095.879719 | 20083.390300 | 20123.221859 |
      max    |     2.063482 |     2.062851 |     2.065229 |     2.060147 |
      99.99% |     1.146647 |     1.143794 |     1.142933 |     1.146529 |
      
      mozilla
      zlib   | no           | 10x          | 5x           | 3x           |
      total  | 41374.102138 | 41418.816773 | 41381.956370 | 41334.280732 |
      max    |     3.383474 |     3.387400 |     3.405711 |     3.387316 |
      99.99% |     1.006755 |     1.005954 |     1.007700 |     1.007373 |
      
      zstd   | no           | 10x          | 5x           | 3x           |
      total  | 24689.691520 | 24643.939662 | 24664.630027 | 24664.512714 |
      max    |     1.460822 |     1.449640 |     1.439747 |     1.465304 |
      99.99% |     0.527111 |     0.527377 |     0.527807 |     0.527226 |
      4a3abb33380a
  4. Jan 21, 2019
    • Valentin Gatien-Baron's avatar
      deltas: skip if projected compressed size is bigger than previous snapshot · 9b5fbe5ead89
      Valentin Gatien-Baron authored
      If we have a delta, we check constraints against a lower bound estimate of the
      resulting compressed delta. We then checks this projected size against the
      `size(snapshotⁿ) > size(snapshotⁿ⁺¹)` constraint. This allows to exclude
      potential base candidates before doing any expensive computation.
      
      This only apply to the intermediate-snapshot case since this constraint only
      apply to them.
      
      For some pathological cases of a private repository this step provide a
      further performance boost (timing from `hg perfrevlogwrite`):
      
      before:  3.010646 seconds
      after:   2.609307 seconds
      9b5fbe5ead89
    • Valentin Gatien-Baron's avatar
      deltas: skip if projected compressed size does not match text size constraint · c1c1872d25d1
      Valentin Gatien-Baron authored
      If we have a delta, we check constraints against a lower bound estimate of the
      resulting compressed delta. We then checks this projected size against the ½ⁿ
      size constraints. This allows to exclude potential base candidates before doing
      any expensive computation.
      
      This only apply to the intermediate-snapshot case since this constraint only apply to
      them.
      
      For some pathological cases of a private repository this step provide a
      further performance boost (timing from `hg perfrevlogwrite`):
      
      before:  3.145906 seconds
      after:   3.010646 seconds
      c1c1872d25d1
    • Valentin Gatien-Baron's avatar
      deltas: accept and skip None return for delta info · 465f2d0df9ae
      Valentin Gatien-Baron authored
      They are some extra computation that will shortcut the delta compression if the
      delta seems hopeless, returning None.
      465f2d0df9ae
    • Valentin Gatien-Baron's avatar
      delta: move some delta chain related computation earlier in deltainfo · 6e9ba867a946
      Valentin Gatien-Baron authored
      They are some more optimization change that will make use of this in the
      function. So we retrieve the data earlier.
      6e9ba867a946
  5. Apr 25, 2019
    • Valentin Gatien-Baron's avatar
      deltas: skip if projected delta size is bigger than previous snapshot · 66c27df1be84
      Valentin Gatien-Baron authored
      Before computing any delta, we get a basic estimation of the delta size we can
      expect and the resulted compressed value. We then checks this projected size
      against the `size(snapshotⁿ) > size(snapshotⁿ⁺¹)` constraint. This allows to
      exclude potential base candidates before doing any expensive computation.
      
      This only apply to the intermediate-snapshot case since this constraint only
      apply to them.
      
      For some pathological cases of a private repository this step provide a
      significant performance boost (timing from `hg perfrevlogwrite`):
      
      before: 14.115908 seconds
      after:   3.145906 seconds
      66c27df1be84
    • Valentin Gatien-Baron's avatar
      deltas: skip if projected delta size does not match text size constraint · a0b26fc8fbba
      Valentin Gatien-Baron authored
      Before computing any delta, we get a basic estimation of the delta size we can
      expect and the resulted compressed value. We then checks this projected size
      against the ½ⁿ size constraints. This allows to exclude potential base
      candidates before doing any expensive computation.
      
      This only apply to the intermediate-snapshot case since this constraint only
      apply to them.
      
      In practice we only perform this new checks for the manifestlog. Manifest log
      combine two property: it is likely to have delta chain issue and its
      diffing/compression is fairly predictable.
      
      The initial author of this changeset is Valentin Gatien-Baron providing the
      initial idea and initial testing, Pierre-Yves David later consolidated the code
      in the right location and run more extensive testing.
      a0b26fc8fbba
    • Pierre-Yves David's avatar
      revlog: add the option to track the expected compression upper bound · bc4373babd04
      Pierre-Yves David authored
      There are various optimization we can do if we can estimate the size of delta
      before actually spending CPU compressing them. So we add a attributed dedicated
      to tracking that.
      
      We only use it on Manifest because (1) it structure is quite stable across all
      Mercurial repository so its compression ratio is fairly universal. This is the
      revlog with most extreme delta (cf the sparse-revlog optimization).
      
      This will be put to use in later changesets.
      
      Right now the compression upper bound is set to 10. This is a fairly
      conservative value (observed value is more around 3), but I prefer to be safe
      while introducing the optimization principles. We can tune the optimization
      threshold later.
      bc4373babd04
  6. Jun 12, 2019
  7. May 16, 2019
  8. Jun 14, 2019
    • Georges Racinet's avatar
      rust-cpython: fix build for MacOSX · 8ee0fdf3b087
      Georges Racinet authored
      MacOSX needs special link flags. Quoting the README of rust-cpython:
      
        create a `.cargo/config` with the following content:
      
        ```
        [target.x86_64-apple-darwin]
        rustflags = [
          "-C", "link-arg=-undefined",
          "-C", "link-arg=dynamic_lookup",
        ]
        ```
      
      This is tested with Python 2.7 (Anaconda install) and Python 3
      (Homebrew install)
      8ee0fdf3b087
    • Georges Racinet's avatar
      rust-cpython: management of shared libray suffix · f4a65077e949
      Georges Racinet authored
      Before this changeset, the shared library objects suffixes
      were both (rustc output and Python input) hardcoded to '.so',
      which is wrong for Python3 and non Linux targets.
      f4a65077e949
  9. May 27, 2019
    • sliquister's avatar
      merge: fix race that could cause wrong size in dirstate · 87a34c767384
      sliquister authored
      The problem is that hg merge/update/etc work the following way:
      1. figure out what files to update
      2. apply the update to disk
      3. apply the update to in-memory dirstate
      4. write dirstate
      
      where step3 looks at the filesystem and assumes it sees the result of
      step2. If a file is changed between step2 and step3, step3 will record
      incorrect information in the dirstate.
      
      I avoid this by passing the size step3 needs directly from step2, for
      the common path (not implemented for change/delete conflicts for
      instance).
      
      I didn't fix the same race for the exec bit for now, because it's less
      likely to be problematic and I had trouble due to the fact that the
      dirstate stores the permissions differently from the manifest (st_mode
      vs '' 'l' 'x'), in combination with tests that pretend that symlinks
      are not supported.
      
      However, I moved the lstat from step3 to step2, which should tighten
      the race window markedly, both for the exec bit and for the mtime.
      
      Differential Revision: https://phab.mercurial-scm.org/D6475
      87a34c767384
  10. Jun 12, 2019
  11. May 19, 2019
  12. May 23, 2019
    • Georges Racinet's avatar
      rust: new rust options in setup.py · 94167e701e12
      Georges Racinet authored
      The --rust global option turns on usage (and by default compilation)
      of the rust-cpython based mercurial.rustext.
      
      Similarly to what's previously done for zstd, there is a --no-rust
      option for the build_ext subcommand in order not to build
      mercurial.rustext, allowing for an OS distribution to prebuild it.
      
      The HGWITHRUSTEXT environment variable is still honored, and has
      the same effect as before, but now it works mostly by making
      the --rust global option defaulting to True, with some special
      cases for the direct-ffi case (see more about that below)
      
      Coincidentally, the --rust flag can also be passed from the make
      commands, like actually all global options, in the PURE variable
      
         make local PURE=--rust
      
      This feels inappropriate, though, and we should follow up with
      a proper make variable for that case.
      
      Although the direct-ffi bindings aren't directly useful any more, we
      keep them at this stage because
      
      - they provide a short prototyping path for experiments in which a C extension
        module has to call into a Rust extension. The proper way of doing that would
        be to use capsules, and it's best to wait for our pull request onto
        rust-cpython for that: https://github.com/dgrunwald/rust-cpython/pull/169
      - Build support for capsules defined in Rust will probably need to reuse
        some of what's currently in use for direct-ffi.
      94167e701e12
  13. May 30, 2019
    • Georges Racinet's avatar
      rust: using policy.importrust from Python callers · a3a8887e4426
      Georges Racinet authored
      This commit converts all current Python callers of
      mercurial.rustext to the new policy.importrust system.
      
      After this point, going through policy.importrust
      or policy.importmod (in some more distant future)
      is mandatory for callers of Rust code outside of
      Python tests.
      
      We felt it to be appropriate to keep Rust-specific tests
      run inconditionally if the Rust extensions are present.
      a3a8887e4426
  14. May 29, 2019
    • Georges Racinet's avatar
      rust: module policy with importrust · 810f66b468cd
      Georges Racinet authored
      We introduce two rust+c module policies and a new
      `policy.importrust()` that makes use of them.
      
      This simple approach provides runtime switching of
      implementations, which is crucial for the performance
      measurements such as those Octobus does with ASV.
      
      It can also be useful for bug analysis.
      
      It also has the advantage of making conditionals in
      Rust callers more uniform, in particular
      abstracting over specifics like `demandimport`
      
      At this point, the build stays unchanged, with the rust-cpython based
      `rustext` module being built if HGWITHRUSTEXT=cpython.
      
      More transparency for the callers, i.e., just using
      `policy.importmod` would be a much longer term and riskier
      effort for the following reasons:
      
      1. It would require to define common module boundaries
         for the three or four cases (pure, c, rust+ext, cffi) and that
         is premature with the Rust extension currently under heavy
         development in areas that are outside the scope of the C extensions.
      2. It would imply internal API changes that are not currently wished,
         as the case of ancestors demonstrates.
      3. The lack of data or property-like attributes (tp_member
         and tp_getset) in current `rust-cpython` makes it impossible to
         achieve direct transparent replacement of pure Python classes by
         Rust extension code, meaning that the caller sometimes has to be able
         to make adjustments or provide additional wrapping.
      810f66b468cd
  15. Jun 13, 2019
  16. Jun 12, 2019
    • Ian Moody's avatar
      phabricator: use parents.set to always set dependencies · c19d259fd6ad
      Ian Moody authored
      Now that Mercurial's Phabricator instance has been updated to a version that
      supports the parents.set transaction on revision.edit we can use that to set
      dependency relationships in patch stacks instead of abusing the summary.
      This has the advantage that we can use it on every `phabsend` so commit
      reordering is picked up without spamming changes like abusing the summary would,
      and using parents.set will clear previous parents unlike parents.add.
      
      Differential Revision: https://phab.mercurial-scm.org/D6514
      c19d259fd6ad
  17. May 31, 2019
  18. Jun 10, 2019
    • Kyle Lippincott's avatar
      rebase: tweak description of inmemory working even w/ dirty working dir · cda591d2bfe1
      Kyle Lippincott authored
      One of our users was confused because they read this, and then attempted to run
      `hg rebase` with a dirty working directory, and it still complained. The reason
      was that they were attempting to rebase the commit they currently had checked
      out, which (at least with evolve workflows enabled) involves updating the
      working directory to be based on the newly rebased commit.
      
      Differential Revision: https://phab.mercurial-scm.org/D6507
      cda591d2bfe1
    • Valentin Gatien-Baron's avatar
      revlog: speed up isancestor · 055c3e2c44f0
      Valentin Gatien-Baron authored
      Currently, it is implemented on top of commonancestorsheads.
      Implement it on top of reachableroots instead, as reachableroots could
      stop walking the graph much sooner than commonancestorsheads.
      
      Measuring repo.changelog.isancestorrev on two revisions in a private repository:
      before: ! wall 0.005175 comb 0.010000 user 0.010000 sys 0.000000 (best of 550)
      after : ! wall 0.000072 comb 0.000000 user 0.000000 sys 0.000000 (best of 36199)
      
      When hg does this kind of operations 1500 times in pull ->
      bookmarks.comparebookmarks -> bookmarks.validdest, that's 11s that
      drop from the --profile output.
      
      Differential Revision: https://phab.mercurial-scm.org/D6506
      055c3e2c44f0
    • Valentin Gatien-Baron's avatar
      dagop: fix documentation of reachableroots · 3e42fc243741
      Valentin Gatien-Baron authored
      The previous revset couldn't be correct as it is symmetric in <roots>
      and <heads>, but reachableroots has no such symmetry. It makes a
      difference with for instance reachableroots(2, 3) where 2 and 3 are
      both children of 1.
      
      Differential Revision: https://phab.mercurial-scm.org/D6505
      3e42fc243741
  19. Jun 11, 2019
    • Ian Moody's avatar
      phabricator: add --blocker argument to phabsend to specify blocking reviewers · f33d3ee110da
      Ian Moody authored
      The way to signal to Conduit that a reviewer is considered blocking is just to
      wrap their PHID in "blocking()" when including it in the list of PHIDs passed
      to `reviewers.add`.
      
      arc doesn't have a --blocker, instead one is supposed to append a '!' to the
      end of reviewer names (I think reviewers are usually added in an editor rather
      than the command line, where '!'s can be more hazardous).
      
      moz-phab (Mozilla's arcanist wrapper) does have a --blocker argument, and being
      explicit like this is also more discoverable. Even `arc diff`'s help doesn't
      seem to mention the reviewer! syntax.
      
      Differential Revision: https://phab.mercurial-scm.org/D6512
      f33d3ee110da
    • Ian Moody's avatar
      phabricator: auto-sanitise API tokens and HTTP cookies from VCR recordings · d3c81439e2ee
      Ian Moody authored
      Currently when making VCR recordings one needs to manually sanitise sensitive
      credentials before committing and submitting them as part of tests. It is easy
      to imagine this being accidentally missed one time by a fallible human and said
      credentials being leaked. It is also possible that it wouldn't be noticed to
      alert the user to the leak since the recording files are so large and
      practically unreviewable. Thus do so automatically, so the only place that needs
      checking is in the test-phabricator.t file.
      
      Differential Revision: https://phab.mercurial-scm.org/D6513
      d3c81439e2ee
    • Pulkit Goyal's avatar
      py3: use .startswith() instead of bytes[0] · c1bf63ac30c5
      Pulkit Goyal authored
      Doing bytes[0] will return the ascii value of that position which breaks
      comparison with a bytechar.
      
      This makes test-absorb.t work again on py3.
      
      Differential Revision: https://phab.mercurial-scm.org/D6508
      c1bf63ac30c5
  20. Jun 09, 2019
  21. Jun 01, 2019
  22. Jun 06, 2019
Loading