Skip to content
Snippets Groups Projects
  1. Apr 06, 2018
  2. Apr 05, 2018
    • Gregory Szorc's avatar
      revlog: move censor logic into main revlog class · 65250a66
      Gregory Szorc authored
      Previously, the revlog class implemented dummy methods for
      various censor-related functionality. Revision censoring was
      (and will continue to be) only possible on filelog instances.
      So filelog implemented these methods to perform something
      reasonable.
      
      A problem with implementing censoring on filelog is that
      it assumes filelog is a revlog. Upcoming work to formalize
      the filelog interface will make this not true.
      
      Furthermore, the censoring logic is security-sensitive. I
      think action-at-a-distance with custom implementation of core
      revlog APIs in derived classes is a bit dangerous. I think at
      a minimum the censor logic should live in revlog.py.
      
      I was tempted to created a "censored revlog" class that
      basically pulled these methods out of filelog. But, I wasn't
      a huge fan of overriding core methods in child classes. A
      reason to do that would be performance. However, the censoring
      code only comes into play when:
      
      * hash verification fails
      * delta generation
      * applying deltas from changegroups
      
      The new code is conditional on an instance attribute. So the
      overhead for running the censored code when the revlog isn't
      censorable is an attribute lookup. All of these operations are
      at least a magnitude slower than a Python attribute lookup. So
      there shouldn't be a performance concern.
      
      Differential Revision: https://phab.mercurial-scm.org/D3151
      65250a66
  3. Apr 06, 2018
    • Gregory Szorc's avatar
      revlog: move parsemeta() and packmeta() from filelog (API) · 0596d274
      Gregory Szorc authored
      filelog.parsemeta() and filelog.packmeta() are used to decode
      and encode metadata for file copies and censor.
      
      An upcoming commit will move the core logic for censoring revlogs
      into revlog.py. This would create a cycle between revlog.py and
      filelog.py. So we move these metadata functions to revlog.py.
      
      .. api::
      
         filelog.parsemeta() and filelog.packmeta() have been moved to
         the revlog module.
      
      Differential Revision: https://phab.mercurial-scm.org/D3150
      0596d274
  4. Apr 05, 2018
    • Gregory Szorc's avatar
      filelog: declare that filelog implements a storage interface · a3202fa8
      Gregory Szorc authored
      Now that we have a declared interface, let's declare that filelog
      implements it.
      
      Tests have been added that confirm the object conforms to the
      interface.
      
      The existing interface checks verify there are no extra public
      attributes outside the declared interface. filelog has several
      extra attributes. So we added a mechanism to suppress this check.
      The goal is to modify the filelog class so we can drop this check.
      
      Differential Revision: https://phab.mercurial-scm.org/D3149
      a3202fa8
    • Gregory Szorc's avatar
      repository: define existing interface for file storage · 4335a75f
      Gregory Szorc authored
      Now that we have mostly successfully implemented an alternate
      storage backend for files data, let's start to define the
      interface for it!
      
      This commit takes the mostly-working interface as defined by the
      simple store repo and codifies it as the file storage interface.
      
      The interface has been split into its logical components:
      
      * index metadata
      * fulltext data
      * mutation
      * everything else
      
      I don't consider the existing interface to be great. But it will
      help to have it more formally defined so we can start chipping away
      at refactoring it.
      
      Differential Revision: https://phab.mercurial-scm.org/D3148
      4335a75f
    • Gregory Szorc's avatar
      tests: run some largefiles and lfs tests with simple store · 556984ae
      Gregory Szorc authored
      Now that the simple store handles flags properly, a handful of
      the largefiles and lfs tests pass!
      
      Differential Revision: https://phab.mercurial-scm.org/D3147
      556984ae
    • Gregory Szorc's avatar
      commands: don't violate storage abstractions in `manifest --all` · 7b7ca9ba
      Gregory Szorc authored
      Previously, we asked the store to emit its data files. For modern
      repos, this would use fncache to resolve the set of files then would
      stat() each file. For my copy of the mozilla-unified repository, this
      took 3.3-10s depending on the state of my filesystem cache to render
      449,790 items.
      
      The previous behavior was a massive layering violation because it
      assumed tracked files would have specific filenames in specific
      directories. Alternate storage backends would violate this assumption.
      
      The new behavior scans the changelog entries for the set of files
      changed by each commit. It aggregates them into a set and then
      sorts and prints the result. This reliably takes ~16.3s on my
      machine. ~80% of the time is spent in zlib decompression.
      
      The performance regression is unfortunate. If we want to claw it
      back, we can create a proper storage API to query for the set of
      tracked files. I'm not opposed to doing that. But I'm in no hurry
      because I suspect ~0 people care about the performance of
      `hg manifest --all`.
      
      .. perf::
      
         `hg manifest --all` is likely slower due to changing its
         implementation to respect storage interface boundaries. If you
         are impacted by this regression in a meaningful way, please make
         noise on the development mailing list and it can be dealt with.
      
      Differential Revision: https://phab.mercurial-scm.org/D3119
      7b7ca9ba
    • Gregory Szorc's avatar
      commands: document the layering violation in `manifest --all` · 814e080a
      Gregory Szorc authored
      This commit fixes the last test failures when using the simple
      store extension!
      
      It turns out that `hg manifest --all` locks the repo and scans for
      revlogs. This feature was added by 71938479eff9 in 2011. I am
      debating changing the behavior. But that can occur in another
      commit.
      
      As part of debugging this, I realized that test-manifest.t is the
      only meaningful tester of `hg manifest --all` and that test was
      improperly disabled when bundlerepos aren't supported. The test is
      testing manifest behavior, not whether you can `hg pull` from a
      bundle. So I changed the test to `hg unbundle` instead.
      
      FWIW, I wasted a non-trivial amount of time tracking down this
      failure. I thought the issue involved Git, which is why I refactored
      the test to be more deterministic. Never in my mind would I have
      guessed that code in `hg manifest` would scan revlogs. I should have
      looked there to begin with. Doh.
      
      Differential Revision: https://phab.mercurial-scm.org/D3118
      814e080a
    • Gregory Szorc's avatar
      simplestore: correctly implement flag processors · 9d4f09bf
      Gregory Szorc authored
      There were a couple of bugs around the implementation of
      flags processing with the simple store. After these changes,
      test-flagprocessor.t now passes!
      
      test-flagprocessor.t was also updated to include explicit test
      coverage that pushed data is as expected on the server.
      
      The test extension used by test-flagprocessor.t has been updated
      so it monkeypatches the object returned from repo.file() instead
      of monkeypatching filelog.filelog. This allows it to work with
      extensions that return custom types from repo.file().
      
      The monkeypatching is rather hacky and probably is performance
      prohibitive for real repos. We should probably come up with a
      better mechanism for registering flag processors so monkeypatching
      isn't needed.
      
      Differential Revision: https://phab.mercurial-scm.org/D3116
      9d4f09bf
    • Gregory Szorc's avatar
      tests: `hg init` after resetting HGRCPATH · 0c1b8955
      Gregory Szorc authored
      Otherwise extensions loaded via --extra-config-opt could prevent
      access to the repo by introducing requirements file. This does mean
      that custom extensions loaded in this way won't impact this test.
      I'm fine with that.
      
      Differential Revision: https://phab.mercurial-scm.org/D3115
      0c1b8955
    • Gregory Szorc's avatar
      tests: work around potential repo incompatibility · 3fbd8b86
      Gregory Szorc authored
      test-run-tests.t invokes run-tests.py. But custom extensions providing
      new repo requirements may be in play and may not get inherited by the
      new run-tests.py. We ensure our repo is created with a vanilla config
      to mitigate extension-caused badness.
      
      Differential Revision: https://phab.mercurial-scm.org/D3114
      3fbd8b86
    • Gregory Szorc's avatar
      tests: disable test-keyword.t with simple store · 91687924
      Gregory Szorc authored
      The keyword extension is hooking into repo.file() and defining its
      own filelog class. It will likely require a more formal storage
      interface before keywords are usable with alternate storage backends.
      
      Differential Revision: https://phab.mercurial-scm.org/D3113
      91687924
    • Gregory Szorc's avatar
      tests: conditionalize test-treemanifest.t · 28ee8b28
      Gregory Szorc authored
      Parts of the test were assuming the use of revlogs with fnstore
      path encoding.
      
      Other parts of the test assumed we could create repos with different
      store encodings and that stream clone bundles worked.
      
      Make all of this conditional on running a revlog repo.
      
      Differential Revision: https://phab.mercurial-scm.org/D3112
      28ee8b28
    • Gregory Szorc's avatar
      tests: use unbundle in test-symlink-os-yes-fs-no.py · 5ac84b20
      Gregory Szorc authored
      The test (which should probably be rewritten as a .t test - the
      test was initially authored in 2009 and this may have predated
      some test harness features allowing us to implement it as a .t
      test) is verifying symlink behavior with regards to working
      directory operations. How it pulls bundle data into a repo is
      not relevant. So we can switch from pull to unbundle so we can
      support environments where bundlerepos don't work.
      
      Differential Revision: https://phab.mercurial-scm.org/D3111
      5ac84b20
  5. Apr 04, 2018
  6. Apr 05, 2018
  7. Apr 04, 2018
  8. Feb 27, 2018
    • Jörg Sonnenberger's avatar
      wireproto: allow direct stream processing for unbundle · 2d965bfe
      Jörg Sonnenberger authored
      Introduce a new option server.streamunbundle which starts a transaction
      immediately to apply a bundle instead of writing it to a temporary file
      first. This side steps the need for a large tmp directory at the cost of
      preventing concurrent pushes. This is a reasonable trade-off for many
      setups as concurrent pushes for the main branch at least are disallowed
      anyway. The option defaults to off to preserve existing behavior.
      
      Change the wireproto interface to provide a generator for reading the
      payload and make callers responsible for consuming all data.
      
      Differential Revision: https://phab.mercurial-scm.org/D2470
      2d965bfe
  9. Apr 06, 2018
  10. Apr 04, 2018
Loading