Skip to content
Snippets Groups Projects
  1. Sep 12, 2018
    • Gregory Szorc's avatar
      wireprotov2: define and implement "changesetdata" command · 9c2c77c73f23
      Gregory Szorc authored
      This commit introduces the "changesetdata" wire protocol command.
      The role of the command is to expose data associated with changelog
      revisions, including the raw revision data itself.
      
      This command is the first piece of a new clone/pull strategy that
      is built on top of domain-specific commands for data retrieval.
      Instead of a monolithic "getbundle" command that transfers all of the
      things, we'll be introducing commands for fetching specific pieces
      of data.
      
      Since the changeset is the fundamental unit from which we derive
      pointers to other data (manifests, file nodes, etc), it makes sense
      to start reimplementing pull with this data.
      
      The command accepts as arguments a set of root and head revisions
      defining the changesets that should be fetched as well as an explicit
      list of nodes. By default, the command returns only the node values:
      the client must explicitly request additional fields be added to the
      response. Current supported fields are the list of parent nodes and
      the revision fulltext.
      
      My plan is to eventually add support for transferring other data
      associated with changesets, including phases, bookmarks, obsolescence
      markers, etc. Since the response format is CBOR, we'll be able to add
      this data into the response object relatively easily (it should be
      as simple as adding a key in a map).
      
      The documentation captures a number of TODO items. Some of these may
      require BC breaking changes. That's fine: wire protocol v2 is still
      highly experimental.
      
      Differential Revision: https://phab.mercurial-scm.org/D4481
      9c2c77c73f23
    • Gregory Szorc's avatar
      exchangev2: start to implement pull with wire protocol v2 · a86d21e70b2b
      Gregory Szorc authored
      Wire protocol version 2 will take a substantially different
      approach to exchange than version 1 (at least as far as pulling
      is concerned).
      
      This commit establishes a new exchangev2 module for holding
      code related to exchange using wire protocol v2. I could have
      added things to the existing exchange module. But it is already
      quite big. And doing things inline isn't in question because
      the existing code is already littered with conditional code
      for various states of support for the existing wire protocol
      as it evolved over 10+ years. A new module gives us a chance
      to make a clean break.
      
      This approach does mean we'll end up writing some duplicate
      code. And there's a significant chance we'll miss functionality
      as code is ported. The plan is to eventually add #testcase's
      to existing tests so the new wire protocol is tested side-by-side
      with the existing one. This will hopefully tease out any
      features that weren't ported properly. But before we get there,
      we need to build up support for the new exchange methods.
      
      Our journey towards implementing a new exchange begins with pulling.
      And pulling begins with discovery.
      
      The discovery code added to exchangev2 is heavily drawn from
      the following functions:
      
      * exchange._pulldiscoverychangegroup
      * discovery.findcommonincoming
      
      For now, we build on top of existing discovery mechanisms. The
      new wire protocol should be capable of doing things more efficiently.
      But I'd rather defer on this problem.
      
      To foster the transition, we invent a fake capability on the HTTPv2
      peer and have the main pull code in exchange.py call into exchangev2
      when the new wire protocol is being used.
      
      Differential Revision: https://phab.mercurial-scm.org/D4480
      a86d21e70b2b
  2. Aug 21, 2018
  3. Sep 14, 2018
    • Kyle Lippincott's avatar
      narrow: intersect provided matcher with narrowmatcher in `hg diff` · ee7ee0c516ca
      Kyle Lippincott authored
      This provides significant speedups when running diff, and no change in behavior
      that I'm aware of (or that the tests found). I tested with a repo that I started
      using narrow in after it was created and attempted to run `hg diff -c .` and
      similar commands in it on a commit that had files not in the narrowspec.
      
      Timing numbers below, using a similar setup as my previous commits.
      before=9db856446298, m-u is mozilla-unified at eb39298e432d (flatmanifest) and
      0553b7f29eaf (treemanifest). l-d-r is a repo simulating a situation I've
      encountered where there's one directory with 30k+ subdirectories. N means
      narrow, T means treemanifest. The narrowspec is pretty small when in use, and
      importantly the narrowspec is applied *after* doing the initial checkout
      (without narrowing), so all of these files exist in the filesystem, which is not
      normally the case if someone has been using narrow for the entire life of the
      clone.
      
      Anything less than a 5% difference in performance is most likely noise.
      
          diff --git:
          repo  | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
          ------+---+---+------------------------+-----------------------+------------
          m-u   |   |   | 1.292 s +-  0.009 s    | 1.295 s +-  0.010 s   | 100.2%
          m-u   |   | x | 1.296 s +-  0.042 s    | 1.299 s +-  0.026 s   | 100.2%
          m-u   | x |   | 1.292 s +-  0.010 s    | 1.297 s +-  0.021 s   | 100.4%
          m-u   | x | x | 84.2 ms +-   1.2 ms    | 83.6 ms +-   0.2 ms   |  99.3%
          l-d-r |   |   | 188.7 ms +-   2.7 ms   | 188.8 ms +-   2.0 ms  | 100.1%
          l-d-r |   | x | 189.9 ms +-   1.5 ms   | 189.4 ms +-   1.2 ms  |  99.7%
          l-d-r | x |   | 97.1 ms +-   1.0 ms    | 87.1 ms +-   1.0 ms   |  89.7% <--
          l-d-r | x | x | 96.9 ms +-   0.8 ms    | 87.2 ms +-   0.7 ms   |  90.0% <--
      
          diff -c . --git:
          repo  | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
          ------+---+---+------------------------+-----------------------+------------
          m-u   |   |   | 231.6 ms +-   3.1 ms   | 228.9 ms +-   1.6 ms  |  98.8%
          m-u   |   | x | 150.5 ms +-   1.7 ms   | 150.7 ms +-   1.4 ms  | 100.1%
          m-u   | x |   | 233.7 ms +-   2.4 ms   | 232.2 ms +-   1.9 ms  |  99.4%
          m-u   | x | x | 126.1 ms +-   1.2 ms   | 126.8 ms +-   1.2 ms  | 100.6%
          l-d-r |   |   | 82.1 ms +-   2.0 ms    | 81.8 ms +-   1.4 ms   |  99.6%
          l-d-r |   | x | 3.732 s +-  0.020 s    | 3.746 s +-  0.027 s   | 100.4%
          l-d-r | x |   | 83.1 ms +-   0.8 ms    | 107.6 ms +-   2.4 ms  | 129.5% <--
          l-d-r | x | x | 758.2 ms +-  38.8 ms   | 188.5 ms +-   1.8 ms  |  24.9% <--
      
          rebase -r . --keep -d .^^:
          repo  | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
          ------+---+---+------------------------+-----------------------+------------
          m-u   |   |   | 5.532 s +-  0.087 s    | 5.496 s +-  0.016 s   |  99.3%
          m-u   |   | x | 5.554 s +-  0.061 s    | 5.532 s +-  0.013 s   |  99.6%
          m-u   | x |   | 5.602 s +-  0.134 s    | 5.508 s +-  0.035 s   |  98.3%
          m-u   | x | x | 582.2 ms +-  15.2 ms   | 572.9 ms +-  12.0 ms  |  98.4%
          l-d-r |   |   | 629.5 ms +-  12.3 ms   | 622.5 ms +-   7.3 ms  |  98.9%
          l-d-r |   | x | 6.173 s +-  0.062 s    | 6.185 s +-  0.076 s   | 100.2%
          l-d-r | x |   | 274.5 ms +-  10.0 ms   | 272.1 ms +-   6.2 ms  |  99.1%
          l-d-r | x | x | 4.835 s +-  0.056 s    | 4.826 s +-  0.034 s   |  99.8%
      
          status --change . --copies:
          repo  | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
          ------+---+---+------------------------+-----------------------+------------
          m-u   |   |   | 214.4 ms +-   1.4 ms   | 212.2 ms +-   1.7 ms  |  99.0%
          m-u   |   | x | 130.9 ms +-   1.2 ms   | 131.7 ms +-   1.1 ms  | 100.6%
          m-u   | x |   | 215.0 ms +-   2.1 ms   | 214.9 ms +-   2.7 ms  | 100.0%
          m-u   | x | x | 109.5 ms +-   2.3 ms   | 107.8 ms +-   0.9 ms  |  98.4%
          l-d-r |   |   | 79.6 ms +-   0.9 ms    | 79.8 ms +-   1.6 ms   | 100.3%
          l-d-r |   | x | 3.799 s +-  0.037 s    | 3.928 s +-  0.021 s   | 103.4% <--?
          l-d-r | x |   | 82.7 ms +-   0.7 ms    | 83.2 ms +-   1.0 ms   | 100.6%
          l-d-r | x | x | 746.8 ms +-   6.1 ms   | 739.0 ms +-   4.2 ms  |  99.0%
      
          status --copies:
          repo  | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
          ------+---+---+------------------------+-----------------------+------------
          m-u   |   |   | 1.884 s +-  0.012 s    | 1.885 s +-  0.013 s   | 100.1%
          m-u   |   | x | 1.897 s +-  0.027 s    | 1.909 s +-  0.077 s   | 100.6%
          m-u   | x |   | 1.886 s +-  0.021 s    | 1.891 s +-  0.030 s   | 100.3%
          m-u   | x | x | 92.0 ms +-   0.7 ms    | 92.4 ms +-   0.4 ms   | 100.4%
          l-d-r |   |   | 570.3 ms +-  18.7 ms   | 552.2 ms +-   4.5 ms  |  96.8%
          l-d-r |   | x | 568.9 ms +-  16.1 ms   | 567.2 ms +-  11.9 ms  |  99.7%
          l-d-r | x |   | 171.1 ms +-   2.5 ms   | 170.4 ms +-   1.2 ms  |  99.6%
          l-d-r | x | x | 171.6 ms +-   3.4 ms   | 171.5 ms +-   1.7 ms  |  99.9%
      
          update $rev^; ~/src/hg/hg{hg}/hg update $rev:
          repo  | N | T | before (mean +- stdev) | after (mean +- stdev) | % of before
          ------+---+---+------------------------+-----------------------+------------
          m-u   |   |   | 3.107 s +-  0.017 s    | 3.116 s +-  0.012 s   | 100.3%
          m-u   |   | x | 2.943 s +-  0.010 s    | 2.945 s +-  0.019 s   | 100.1%
          m-u   | x |   | 3.116 s +-  0.033 s    | 3.118 s +-  0.027 s   | 100.1%
          m-u   | x | x | 318.5 ms +-   2.7 ms   | 320.8 ms +-   4.8 ms  | 100.7%
          l-d-r |   |   | 428.9 ms +-   4.4 ms   | 429.5 ms +-   4.0 ms  | 100.1%
          l-d-r |   | x | 9.593 s +-  0.081 s    | 9.869 s +-  0.043 s   | 102.9%
          l-d-r | x |   | 253.2 ms +-   3.6 ms   | 254.0 ms +-   2.8 ms  | 100.3%
          l-d-r | x | x | 1.613 s +-  0.009 s    | 1.630 s +-  0.017 s   | 101.1%
      
      Differential Revision: https://phab.mercurial-scm.org/D4587
      ee7ee0c516ca
  4. Sep 01, 2018
  5. Jun 07, 2018
    • Yuya Nishihara's avatar
      formatter: populate ctx from repo and node value · 34ecc0a09c76
      Yuya Nishihara authored
      This will basically replace the fm.contexthint() API. I originally thought
      this would be too complicated, and I wrote 8399438bc7ef "formatter: provide
      hint of context keys required by template" because of that. However, I had
      to add a similar mechanism for fctx templates, and the overall machinery
      became way simpler than my original patch.
      
      The test output slightly changed as {author} is no longer available in
      the {manifest} context, which isn't the point this test targeted on.
      34ecc0a09c76
  6. Sep 14, 2018
  7. Sep 13, 2018
  8. Sep 14, 2018
  9. Sep 13, 2018
    • Augie Fackler's avatar
      py3: whitelist another passing test · 053aa673947f
      Augie Fackler authored
      Differential Revision: https://phab.mercurial-scm.org/D4562
      053aa673947f
    • Matt Harbison's avatar
      py3: prevent the win32 ctype _fields_ from being transformed to bytes · f3900f4c63d4
      Matt Harbison authored
      Otherwise, any hg invocation dies with
      
          TypeError: '_fields_' must be a sequence of (name, C type) pairs
      
      # skip-blame just a r prefix
      f3900f4c63d4
    • Matt Harbison's avatar
      cext: fix warnings when building for py3 on Windows · 11ed2eadf937
      Matt Harbison authored
      MSVC++ 14 now has standard int types that don't need to be redefined (I didn't
      go back to see when they came along since the build system wants either 2008 or
      2015), but doesn't have ssize_t.  The FILE pointer in posixfile is only used on
      python2.
      11ed2eadf937
    • Matt Harbison's avatar
      cext: stop preprocessing a partial function call · c06c585f166b
      Matt Harbison authored
      MSVC++ 14 yelled:
      
          mercurial/cext/revlog.c(1913): fatal error C1057: unexpected end of file in
                                         macro expansion
      
      At this point, the C extensions build (with warnings), and it dies in win32.py
      because the `_fields_` strings in the ctypes classes are being converted to
      bytes by the source translator.
      c06c585f166b
    • Matt Harbison's avatar
      py3: add b'' to some setup.py strings for Windows · ec68135a8935
      Matt Harbison authored
      These were things found trying to do `make PYTHON="py -3" local`.  The following
      is dumped out, before dying while compiling the C extensions:
      
      C:\Program Files\Python37\lib\site-packages\setuptools\dist.py:406: UserWarning:
        The version specified (b'4.7.1') is an invalid version, this may not work as
        expected with newer versions of setuptools, pip, and PyPI. Please see PEP 440
        for more details.
        "details." % self.metadata.version
      running build_py
      byte-compiling .\mercurial\thirdparty\concurrent\futures\_base.py to _base.cpython-37.pyc
        File "mercurial\thirdparty\concurrent\futures\_base.py", line 416
          raise exception_type, self._exception, self._traceback
                              ^
      SyntaxError: invalid syntax
      
      # skip-blame since these are just converting to bytes literals
      ec68135a8935
    • Augie Fackler's avatar
      5362c96f2feb
    • Gregory Szorc's avatar
      hg: don't reuse repo instance after unshare() · c5e6c1ba1c79
      Gregory Szorc authored
      Unsharing a repository is a pretty invasive procedure and fundamentally
      changes the behavior of the repository.
      
      Currently, hg.unshare() calls into localrepository.__init__ to
      re-initialize the repository instance. This is a bit hacky. And
      future commits that refactor how localrepository instances are
      constructed will make this difficult to support.
      
      This commit changes unshare() so it constructs a new repo instance
      once the unshare I/O has completed. It then poisons the old repo
      instance so any further use will result in error.
      
      Surprisingly, nothing in core appears to access a repo instance
      after it has been unshared!
      
      .. api::
      
         ``hg.unshare()`` now poisons the repo instance so it can't be used.
      
         It also returns a new repo instance suitable for interacting with
         the unshared repository.
      
      Differential Revision: https://phab.mercurial-scm.org/D4557
      c5e6c1ba1c79
  10. Sep 12, 2018
    • Gregory Szorc's avatar
      unionrepo: dynamically create repository type from base repository · 23f2299e9e53
      Gregory Szorc authored
      This is basically the same thing we just did for bundlerepo except
      for union repositories.
      
      .. api::
      
         ``unionrepo.unionrepository()`` is no longer usable on its own.
      
         To instantiate an instance, call ``unionrepo.instance()`` or
         ``unionrepo.makeunionrepository()``.
      
      Differential Revision: https://phab.mercurial-scm.org/D4556
      23f2299e9e53
    • Gregory Szorc's avatar
      bundlerepo: dynamically create repository type from base repository · 335ae4d0a552
      Gregory Szorc authored
      Previously, bundlerepository inherited from localrepo.localrepository.
      You simply instantiated a bundlerepository and its __init__ called
      localrepo.localrepository.__init__. Things were simple.
      
      Unfortunately, this strategy is limiting because it assumes that
      the base repository is a localrepository instance. And it assumes
      various properties of localrepository, such as the arguments its
      __init__ takes. And it prevents us from changing behavior of
      localrepository.__init__ without also having to change derived classes.
      
      Previous and ongoing work to abstract storage revealed these
      limitations.
      
      This commit changes the initialization strategy of bundle repositories
      to dynamically create a type to represent the repository. Instead of
      a static type, we instantiate a new local repo instance via
      localrepo.instance(). We then combine its __class__ with
      bundlerepository to produce a new type. This ensures that no matter
      how localrepo.instance() decides to create a repository object, we
      can derive a bundle repo object from it. i.e. localrepo.instance()
      could return a type that isn't a localrepository and it would "just
      work."
      
      Well, it would "just work" if bundlerepository's custom implementations
      only accessed attributes in the documented repository interface. I'm
      pretty sure it violates the interface contract in a handful of
      places. But we can worry about that another day. This change gets us
      closer to doing more clever things around instantiating repository
      instances without having to worry about teaching bundlerepository about
      them.
      
      .. api::
      
         ``bundlerepo.bundlerepository`` is no longer usable on its own.
      
         The class is combined with the class of the base repository it is
         associated with at run-time.
      
         New bundlerepository instances can be obtained by calling
         ``bundlerepo.instance()`` or ``bundlerepo.makebundlerepository()``.
      
      Differential Revision: https://phab.mercurial-scm.org/D4555
      335ae4d0a552
    • Gregory Szorc's avatar
      bundlerepo: factor out code for instantiating a bundle repository · a8d2faeca49e
      Gregory Szorc authored
      This code will soon become a bit more complicated. So extract to its
      own function.
      
      And change both instantiators of bundlerepository to use it.
      
      Differential Revision: https://phab.mercurial-scm.org/D4554
      a8d2faeca49e
    • Gregory Szorc's avatar
      bundlerepo: pass create=True · 2d2bbf3f3e09
      Gregory Szorc authored
      I don't want to know how this came to be. Maybe a holdover from the
      days before Python had a bool type?
      
      Differential Revision: https://phab.mercurial-scm.org/D4553
      2d2bbf3f3e09
    • Gregory Szorc's avatar
      shelve: use bundlerepo.instance() to construct a repo object · 84d6e9a2b104
      Gregory Szorc authored
      The instance() functions are preferred over cls.__init__ for
      creating repo instances. It doesn't really matter now. But future
      commits will refactor the bundlerepository class in ways that will
      cause the old way to break.
      
      Differential Revision: https://phab.mercurial-scm.org/D4552
      84d6e9a2b104
  11. Jul 29, 2018
  12. Sep 13, 2018
  13. Sep 10, 2018
    • Boris Feld's avatar
      revlog: reuse cached delta for identical base revision (issue5975) · a911932d5003
      Boris Feld authored
      Since 8f83a953dddf, we skip over empty deltas when choosing a delta base. Such
      delta happens when two distinct revisions have the same content.
      
      The remote might be sending a delta against such revision within the bundle.
      In that case, the delta base is no longer considered, but the cached one could
      still, be used with the equivalent revision.
      
      Not reusing the delta from the bundle can have a significant performance
      impact, so we now make sure with doing so when possible.
      a911932d5003
    • Boris Feld's avatar
      snapshot: fix line order when skipping over empty deltas · bdb41eaa8b59
      Boris Feld authored
      The code movement in 37957e07138c introduced an error.
      
      Since 8f83a953dddf, we discarded some revisions because they are identical to
      their delta base (and use that delta base instead). That logic is good,
      however, in 37957e07138c we mixed up the order of two line, adding the "new"
      revision to the set of already tested one, instead of the discarded one. So in
      practice, we were never investigating any revisions in a chain starting with
      an empty delta. Creating significantly worst delta chain (eg: Mercurial's
      manifest move goes from about 60MB up to about 80MB).
      bdb41eaa8b59
  14. Sep 13, 2018
  15. Aug 21, 2018
  16. Sep 12, 2018
Loading