Skip to content
Snippets Groups Projects
  1. Jun 17, 2019
  2. May 03, 2019
  3. Apr 19, 2019
    • Martin von Zweigbergk's avatar
      copies: avoid unnecessary copying of copy dict · 5ceb91136ebe
      Martin von Zweigbergk authored
      When storing copy information in the changesets, this patch speeds up
      
        hg debugpathcopies FENNEC_58_0_2_BUILD1 FIREFOX_59_0b8_BUILD2
      
      from 11s to 5.9s. That command takes 6.2s when storing copy
      information in filelogs.
      
      Differential Revision: https://phab.mercurial-scm.org/D6421
      5ceb91136ebe
    • Martin von Zweigbergk's avatar
      copies: don't filter out copy targets created on other side of merge commit · 35d674a3d5db
      Martin von Zweigbergk authored
      If file X is copied to Y on one side of merge and the other side
      creates Y (no copy), we would not mark that as copy. In the
      changeset-centric pathcopies() version, that was done by checking if
      the copy target existed on the other branch. Even though merge commits
      are pretty uncommon, it still turned out to be too expensive to load
      the manifest of the parents of merge commits. In a repo of
      mozilla-unified converted to storing copies in changesets, about 2m30s
      of `hg debugpathcopies FIREFOX_BETA_59_END FIREFOX_BETA_60_BASE` is
      spent on this check of merge commits.
      
      I tried to think of a way of storing more information in the
      changesets in order to cheaply detect these cases, but I couldn't
      think of a solution. So this patch simply removes those checks.
      
      For reference, these extra copies are reported from the aforementioned
      command after this patch:
      
        browser/base/content/sanitize.js -> browser/modules/Sanitizer.jsm
        testing/mozbase/mozprocess/tests/process_normal_finish_python.ini -> testing/mozbase/mozprocess/tests/process_normal_finish.ini
        testing/mozbase/mozprocess/tests/process_waittimeout_python.ini -> testing/mozbase/mozprocess/tests/process_waittimeout.ini
        testing/mozbase/mozprocess/tests/process_waittimeout_10s_python.ini -> testing/mozbase/mozprocess/tests/process_waittimeout_10s.ini
      
      Since these copies were created on one side of some merge, it still
      seems reasonable to include them, so I'm not even sure it's worse than
      filelog pathcopies(), just different.
      
      Differential Revision: https://phab.mercurial-scm.org/D6420
      35d674a3d5db
  4. Apr 18, 2019
    • Martin von Zweigbergk's avatar
      copies: do full filtering at end of _changesetforwardcopies() · 4c39c99d9492
      Martin von Zweigbergk authored
      As mentioned earlier, pathcopies() is very slow when copies are stored
      in the changeset. Most of the cost comes from calling _chain() for
      every changeset, which is slow because it needs to read manifests. It
      needs to read manifests to be able to filter out copies that are were
      created in one commit and then deleted. (It also filters out copies
      that were created from a file that didn't exist in the starting
      revision, but that's a fixed revision across calls to _chain(), so
      it's much cheaper.)
      
      This patch changes from _chainandfilter() to just _chain() in the main
      loop in _changesetforwardcopies(). It instead removes copies that have
      subsequently been removed by using ctx.filesremoved(). We thus rely on
      that to be fast.
      
      It timed this command in mozilla-unified:
      
        hg debugpathcopies FIREFOX_59_0b3_BUILD2 FIREFOX_BETA_59_END
      
      It took 18s before and 1.1s after. It's still faster when copy
      information is stored in filelogs: 0.70s. It also still gets slow when
      there are merge commits involved, because we read manifests there
      too. We'll deal with that later.
      
      Differential Revision: https://phab.mercurial-scm.org/D6419
      4c39c99d9492
  5. Jun 15, 2019
  6. Jun 17, 2019
  7. Jun 16, 2019
  8. Jun 14, 2019
  9. Oct 04, 2018
    • Matt Harbison's avatar
      cat: don't prefetch files unless the output requires it · 561cd02c58ff
      Matt Harbison authored
      It's a waste to cache lfs blobs when cat'ing the raw data at best, but a hassle
      debugging when the blob is missing.  I'm not sure if there are other commands
      that have '{data}' for output, and if there's a general way to prefetch on that
      keyword.
      
      It's interesting that the verbose output seems to leak into the JSON output, but
      that seems like an existing bug.
      561cd02c58ff
  10. Jun 12, 2019
  11. Jun 14, 2019
  12. Apr 25, 2019
    • Pierre-Yves David's avatar
      deltas: set estimated compression upper bound to "3x" instead of "10x" · 4a3abb33380a
      Pierre-Yves David authored
      In pratice, we very rarely observer compression better than "3x" on manifest
      deltas. Having a more aggressive estimate significantly helps our pathological
      use case on a private repository. Here are a comparison of timings using
      different upper bound.
      
      Estimated compression |    ø   |  ×10 |  ×5  |  ×3  |
      timing                |  14.11 | 2.61 | 1.96 | 1.53 |
      
      
      We also tested the impact of this series on an array of public repositories.
      This shown no impact in either size nor timing.
      
      Full data set below for those interested.
      
      Size
      ----
      
      Regarding size, not significant impact have been noticed on neither public nor
      private repositories. Here are the number we gathered on public repositories:
      
      zlib/upperbound | no            | 10x           | 5x            | 3x
      mercurial       |     5 875 730 |     5 875 730 |     5 875 730 |     5 875 730
      pypy            |    27 782 913 |    27 782 913 |    27 782 913 |    27 782 913
      netbeans        |   159 161 207 |   159 161 207 |   159 161 207 |   159 959 879 (+0.5%)
      mozilla-central |   323 841 642 |   323 841 642 |   323 841 642 |   319 867 519 (-2.5%)
      mozilla-try     |   746 649 123 |   746 649 123 |   746 649 123 |   741 155 568 (-0.7%)
      private-repo    | 1 485 287 294 | 1 485 287 294 | 1 485 287 294 | 1 409 248 382 (-5.1%)
      
      zstd/upperbound | no            | 10x           | 5x            | 3x
      mercurial       |     5 895 206 |     5 895 206 |     5 895 206 |     5 895 206
      pypy            |    28 689 230 |    28 689 230 |    28 689 230 |    28 689 230
      netbeans        |   157 636 387 |   157 636 387 |   157 636 387 |   159 692 678 (+1.3%)
      mozilla-central |   317 650 281 |   317 650 281 |   317 650 281 |   319 613 603 (+0.6%)
      mozilla-try     |   737 555 275 |   737 555 275 |   737 555 275 |   738 079 473 (+0.1%)
      private-repo    | 1 352 362 982 | 1 352 362 982 | 1 346 961 880 | 1 361 327 384 (+0.7%)
      
      
      Speed
      ------
      
      Timing gathered using `hg perfrevlogwrite -m`. Value are in seconds.
      
      mercurial
      
      zlib   | no        | 10x       | 5x        | 3x        |
      total  | 65.551783 | 65.388887 | 65.260658 | 65.321199 |
      max    |  0.034544 |  0.034571 |  0.034659 |  0.034521 |
      99.99% |  0.034544 |  0.034571 |  0.034659 |  0.034521 |
      
      zstd   | no        | 10x       | 5x        | 3x        |
      total  | 49.118449 | 49.054062 | 48.753588 | 48.740230 |
      max    |  0.009338 |  0.009239 |  0.009202 |  0.009178 |
      99.99% |  0.007618 |  0.007639 |  0.007626 |  0.007621 |
      
      pypy
      
      zlib   | no         | 10x        | 5x         | 3x         |
      total  | 560.865984 | 558.983817 | 559.083815 | 559.349152 |
      max    |   0.219614 |   0.215922 |   0.218112 |   0.218107 |
      99.99% |   0.219614 |   0.215922 |   0.218112 |   0.218107 |
      
      zstd   | no         | 10x        | 5x         | 3x         |
      total  | 349.393280 | 347.395819 | 347.185407 | 345.643985 |
      max    |   0.084143 |   0.083536 |   0.081834 |   0.082178 |
      99.99% |   0.039445 |   0.039639 |   0.039612 |   0.039175 |
      
      netbeans
      zlib   | no           | 10x          | 5x           | 3x           |
      total  | 33103.327727 | 33314.932260 | 33211.745233 | 33345.891778 |
      max    |     2.666852 |     2.672059 |     2.662453 |     2.662936 |
      99.99% |     2.058772 |     2.070429 |     2.069569 |     2.064653 |
      
      zstd   | no           | 10x         | 5x            | 3x           |
      total  | 20112.102708 | 20095.879719 | 20083.390300 | 20123.221859 |
      max    |     2.063482 |     2.062851 |     2.065229 |     2.060147 |
      99.99% |     1.146647 |     1.143794 |     1.142933 |     1.146529 |
      
      mozilla
      zlib   | no           | 10x          | 5x           | 3x           |
      total  | 41374.102138 | 41418.816773 | 41381.956370 | 41334.280732 |
      max    |     3.383474 |     3.387400 |     3.405711 |     3.387316 |
      99.99% |     1.006755 |     1.005954 |     1.007700 |     1.007373 |
      
      zstd   | no           | 10x          | 5x           | 3x           |
      total  | 24689.691520 | 24643.939662 | 24664.630027 | 24664.512714 |
      max    |     1.460822 |     1.449640 |     1.439747 |     1.465304 |
      99.99% |     0.527111 |     0.527377 |     0.527807 |     0.527226 |
      4a3abb33380a
  13. Jan 21, 2019
    • Valentin Gatien-Baron's avatar
      deltas: skip if projected compressed size is bigger than previous snapshot · 9b5fbe5ead89
      Valentin Gatien-Baron authored
      If we have a delta, we check constraints against a lower bound estimate of the
      resulting compressed delta. We then checks this projected size against the
      `size(snapshotⁿ) > size(snapshotⁿ⁺¹)` constraint. This allows to exclude
      potential base candidates before doing any expensive computation.
      
      This only apply to the intermediate-snapshot case since this constraint only
      apply to them.
      
      For some pathological cases of a private repository this step provide a
      further performance boost (timing from `hg perfrevlogwrite`):
      
      before:  3.010646 seconds
      after:   2.609307 seconds
      9b5fbe5ead89
    • Valentin Gatien-Baron's avatar
      deltas: skip if projected compressed size does not match text size constraint · c1c1872d25d1
      Valentin Gatien-Baron authored
      If we have a delta, we check constraints against a lower bound estimate of the
      resulting compressed delta. We then checks this projected size against the ½ⁿ
      size constraints. This allows to exclude potential base candidates before doing
      any expensive computation.
      
      This only apply to the intermediate-snapshot case since this constraint only apply to
      them.
      
      For some pathological cases of a private repository this step provide a
      further performance boost (timing from `hg perfrevlogwrite`):
      
      before:  3.145906 seconds
      after:   3.010646 seconds
      c1c1872d25d1
    • Valentin Gatien-Baron's avatar
      deltas: accept and skip None return for delta info · 465f2d0df9ae
      Valentin Gatien-Baron authored
      They are some extra computation that will shortcut the delta compression if the
      delta seems hopeless, returning None.
      465f2d0df9ae
    • Valentin Gatien-Baron's avatar
      delta: move some delta chain related computation earlier in deltainfo · 6e9ba867a946
      Valentin Gatien-Baron authored
      They are some more optimization change that will make use of this in the
      function. So we retrieve the data earlier.
      6e9ba867a946
  14. Apr 25, 2019
    • Valentin Gatien-Baron's avatar
      deltas: skip if projected delta size is bigger than previous snapshot · 66c27df1be84
      Valentin Gatien-Baron authored
      Before computing any delta, we get a basic estimation of the delta size we can
      expect and the resulted compressed value. We then checks this projected size
      against the `size(snapshotⁿ) > size(snapshotⁿ⁺¹)` constraint. This allows to
      exclude potential base candidates before doing any expensive computation.
      
      This only apply to the intermediate-snapshot case since this constraint only
      apply to them.
      
      For some pathological cases of a private repository this step provide a
      significant performance boost (timing from `hg perfrevlogwrite`):
      
      before: 14.115908 seconds
      after:   3.145906 seconds
      66c27df1be84
    • Valentin Gatien-Baron's avatar
      deltas: skip if projected delta size does not match text size constraint · a0b26fc8fbba
      Valentin Gatien-Baron authored
      Before computing any delta, we get a basic estimation of the delta size we can
      expect and the resulted compressed value. We then checks this projected size
      against the ½ⁿ size constraints. This allows to exclude potential base
      candidates before doing any expensive computation.
      
      This only apply to the intermediate-snapshot case since this constraint only
      apply to them.
      
      In practice we only perform this new checks for the manifestlog. Manifest log
      combine two property: it is likely to have delta chain issue and its
      diffing/compression is fairly predictable.
      
      The initial author of this changeset is Valentin Gatien-Baron providing the
      initial idea and initial testing, Pierre-Yves David later consolidated the code
      in the right location and run more extensive testing.
      a0b26fc8fbba
    • Pierre-Yves David's avatar
      revlog: add the option to track the expected compression upper bound · bc4373babd04
      Pierre-Yves David authored
      There are various optimization we can do if we can estimate the size of delta
      before actually spending CPU compressing them. So we add a attributed dedicated
      to tracking that.
      
      We only use it on Manifest because (1) it structure is quite stable across all
      Mercurial repository so its compression ratio is fairly universal. This is the
      revlog with most extreme delta (cf the sparse-revlog optimization).
      
      This will be put to use in later changesets.
      
      Right now the compression upper bound is set to 10. This is a fairly
      conservative value (observed value is more around 3), but I prefer to be safe
      while introducing the optimization principles. We can tune the optimization
      threshold later.
      bc4373babd04
  15. Jun 12, 2019
  16. May 16, 2019
  17. Jun 14, 2019
    • Georges Racinet's avatar
      rust-cpython: fix build for MacOSX · 8ee0fdf3b087
      Georges Racinet authored
      MacOSX needs special link flags. Quoting the README of rust-cpython:
      
        create a `.cargo/config` with the following content:
      
        ```
        [target.x86_64-apple-darwin]
        rustflags = [
          "-C", "link-arg=-undefined",
          "-C", "link-arg=dynamic_lookup",
        ]
        ```
      
      This is tested with Python 2.7 (Anaconda install) and Python 3
      (Homebrew install)
      8ee0fdf3b087
    • Georges Racinet's avatar
      rust-cpython: management of shared libray suffix · f4a65077e949
      Georges Racinet authored
      Before this changeset, the shared library objects suffixes
      were both (rustc output and Python input) hardcoded to '.so',
      which is wrong for Python3 and non Linux targets.
      f4a65077e949
  18. May 27, 2019
    • sliquister's avatar
      merge: fix race that could cause wrong size in dirstate · 87a34c767384
      sliquister authored
      The problem is that hg merge/update/etc work the following way:
      1. figure out what files to update
      2. apply the update to disk
      3. apply the update to in-memory dirstate
      4. write dirstate
      
      where step3 looks at the filesystem and assumes it sees the result of
      step2. If a file is changed between step2 and step3, step3 will record
      incorrect information in the dirstate.
      
      I avoid this by passing the size step3 needs directly from step2, for
      the common path (not implemented for change/delete conflicts for
      instance).
      
      I didn't fix the same race for the exec bit for now, because it's less
      likely to be problematic and I had trouble due to the fact that the
      dirstate stores the permissions differently from the manifest (st_mode
      vs '' 'l' 'x'), in combination with tests that pretend that symlinks
      are not supported.
      
      However, I moved the lstat from step3 to step2, which should tighten
      the race window markedly, both for the exec bit and for the mtime.
      
      Differential Revision: https://phab.mercurial-scm.org/D6475
      87a34c767384
  19. Jun 12, 2019
  20. May 19, 2019
  21. May 23, 2019
    • Georges Racinet's avatar
      rust: new rust options in setup.py · 94167e701e12
      Georges Racinet authored
      The --rust global option turns on usage (and by default compilation)
      of the rust-cpython based mercurial.rustext.
      
      Similarly to what's previously done for zstd, there is a --no-rust
      option for the build_ext subcommand in order not to build
      mercurial.rustext, allowing for an OS distribution to prebuild it.
      
      The HGWITHRUSTEXT environment variable is still honored, and has
      the same effect as before, but now it works mostly by making
      the --rust global option defaulting to True, with some special
      cases for the direct-ffi case (see more about that below)
      
      Coincidentally, the --rust flag can also be passed from the make
      commands, like actually all global options, in the PURE variable
      
         make local PURE=--rust
      
      This feels inappropriate, though, and we should follow up with
      a proper make variable for that case.
      
      Although the direct-ffi bindings aren't directly useful any more, we
      keep them at this stage because
      
      - they provide a short prototyping path for experiments in which a C extension
        module has to call into a Rust extension. The proper way of doing that would
        be to use capsules, and it's best to wait for our pull request onto
        rust-cpython for that: https://github.com/dgrunwald/rust-cpython/pull/169
      - Build support for capsules defined in Rust will probably need to reuse
        some of what's currently in use for direct-ffi.
      94167e701e12
  22. May 30, 2019
    • Georges Racinet's avatar
      rust: using policy.importrust from Python callers · a3a8887e4426
      Georges Racinet authored
      This commit converts all current Python callers of
      mercurial.rustext to the new policy.importrust system.
      
      After this point, going through policy.importrust
      or policy.importmod (in some more distant future)
      is mandatory for callers of Rust code outside of
      Python tests.
      
      We felt it to be appropriate to keep Rust-specific tests
      run inconditionally if the Rust extensions are present.
      a3a8887e4426
Loading