Skip to content
Snippets Groups Projects
  1. Oct 02, 2018
  2. Oct 01, 2018
  3. Sep 28, 2018
  4. Sep 30, 2018
  5. Nov 03, 2017
  6. Sep 26, 2018
    • Boris Feld's avatar
      cleanupnodes: drop special casing around prune markers (API) · 1857f50a
      Boris Feld authored
      The `cleanupnodes` has logic to skip the creation of "prune" markers if the
      changeset is already obsolete. This feels strange and gets in the way of code
      changes to tracks folds. Now that callers no longer request such prune, we can
      drop this logic.
      
      In many cases, pruning through cleanupnodes should be replaced by internal
      phase usage.
      1857f50a
  7. Sep 30, 2018
  8. Sep 29, 2018
  9. Sep 27, 2018
    • Boris Feld's avatar
      pullreport: rev duplicated and extinct into account · a89dd6d0
      Boris Feld authored
      If we already have some obsolete and hidden nodes locally and the server send
      them again to you, it seems useful to point it out instead of being silent about
      it.
      a89dd6d0
    • Boris Feld's avatar
      pullreport: issue a message about "extinct" pulled changesets · f9232b03
      Boris Feld authored
      Changeset pulled from a remote repository while already obsolete locally can
      end up hidden after the pull. Hiding obsolete changesets is a good behavior
      but silently "skipping" some of the pulled content can get confusing.
      
      We now detect this situation and emit a message about it. The message is
      simple and the wording could be improved, however, we focus on the detection
      here. Evolution is still an experimental feature, so the output is open to
      changes.
      
      In particular, we could point out at the latest successors of the obsolete
      changesets, however, it can get tricky is there are many of them. So we delay
      these improvements to another adventure.
      
      Another easy improvement would be to merge this message with the previous line
      about the new nodes and their phases.
      
      This is a good example of cases where we can only transmit a limited amount of
      data to users by default. We need some sort of "transaction journal" we could
      point the user to.
      f9232b03
    • Boris Feld's avatar
      pullreport: skip filtered revs instead of obsolete ones · a477679f
      Boris Feld authored
      Obsolescence is closely related to visibility but still a distinct concept. We
      can receive changesets that are obsolete but visible (eg: when pulling
      orphans). Such changeset should be reported too. In addition, the filtering
      level can be anything, we should respect it.
      a477679f
    • Boris Feld's avatar
      pullreport: add a test to show misreporting of visible changeset · 642cbc22
      Boris Feld authored
      The current code ignores all obsolete changesets including the visible one. We
      add a test showing this behavior before fixing the behavior.
      642cbc22
  10. Sep 26, 2018
  11. Sep 28, 2018
  12. Sep 20, 2018
    • Boris Feld's avatar
      shelve: find shelvedctx from bundle even if they are already in the repo · d9ba836f
      Boris Feld authored
      We use the new "duplicates" node tracking to find the tip of the bundle even if
      it already exists in the repository.
      
      Such logic is not supposed to be needed in theory. If the shelve was made using
      internal-phase, we already know its node. Otherwise, the bundle content should
      have been stripped. However, handling it makes the shelve code more robust and
      provide a good example of "revduplicates" usage.
      d9ba836f
  13. Sep 24, 2018
    • Gregory Szorc's avatar
      storageutil: move metadata parsing and packing from revlog (API) · 3e896b51
      Gregory Szorc authored
      Parsing and writing of revision text metadata is likely identical
      across storage backends. Let's move the code out of revlog so we
      don't need to import the revlog module in order to use it.
      
      Differential Revision: https://phab.mercurial-scm.org/D4754
      3e896b51
    • Gregory Szorc's avatar
      storageutil: new module for storage primitives (API) · f8eb71f9
      Gregory Szorc authored
      There will exist common code between storage backends. It would
      be nice to have a central place to put that code.
      
      This commit attempts to create that place by creating the
      "storageutil" module.
      
      The first thing we move is revlog.hash(), which is the function for
      computing the SHA-1 hash of revision fulltext and parents.
      
      Differential Revision: https://phab.mercurial-scm.org/D4753
      f8eb71f9
    • Gregory Szorc's avatar
      filelog: stop proxying deltaparent() (API) · a269fa55
      Gregory Szorc authored
      deltaparent() obtains the revision number of the base revision a
      delta in storage is stored against. It is highly revlog-centric and
      may not apply to other storage backends. As a result, it doesn't
      belong on the generic file storage interface.
      
      This method/proxy is no longer used in core. The last consumer was
      probably changegroup code and went away with the transition to
      emitrevisions().
      
      Differential Revision: https://phab.mercurial-scm.org/D4751
      a269fa55
    • Gregory Szorc's avatar
      filelog: stop proxying rawsize() (API) · d909c44d
      Gregory Szorc authored
      This method is no longer used by external consumers. The API is
      quite low-level and is effectively len(revision(raw=True)). I don't
      see a compelling reason to keep it around.
      
      Let's drop the API and make the file storage interface simpler.
      
      Differential Revision: https://phab.mercurial-scm.org/D4750
      d909c44d
    • Gregory Szorc's avatar
      filelog: stop proxying flags() (API) · 2ac4f3e9
      Gregory Szorc authored
      Per-revision storage flags are kinda a revlog-centric API. (Except for
      the fact that changegroup uses the same integer flags as revlog does
      and there's minimal verification that the server's flags map to the
      client's storage flags - but that's another problem.)
      
      The last user of flags() was in verify.py and that code was just moved
      into revlog.py and is accessed behind the verifyintegrity() file
      storage API.
      
      Since there are no more consumers, let's drop the proxy and remove
      the method from the file storage interface.
      
      This commit only drops the dedicated API for reading a single
      revision's storage flags: we still support reading and writing flags
      through the bulk data retrieval and add revision APIs. And since
      changegroups encode revlog integer flags over the wire, we'll always
      need to support flags at some level. The removal of individual storage
      flags may be too premature. But since flags() is now unused, I'd like
      to see how far we can get without that dedicated API - especially
      since it uses revision numbers instead of nodes.
      
      Differential Revision: https://phab.mercurial-scm.org/D4746
      2ac4f3e9
  14. Sep 26, 2018
  15. Sep 24, 2018
  16. Sep 21, 2018
    • Gregory Szorc's avatar
      tests: use more complex file storage test · c73f9f34
      Gregory Szorc authored
      The previous test was attempting to to test delta storage behavior.
      It didn't do a very good job at it because there was a good chance
      a delta wasn't being used in storage.
      
      Let's switch the test to yield a delta in storage so an upcoming
      change to delegate delta logic to storage has the desired effect.
      
      Differential Revision: https://phab.mercurial-scm.org/D4723
      c73f9f34
  17. Sep 24, 2018
    • Gregory Szorc's avatar
      changegroup: remove reordering control (BC) · db5501d9
      Gregory Szorc authored
      This logic - including the experimental bundle.reorder option -
      was originally added in a8e3931e3fb5 in 2011 and then later ported
      to changegroup.py.
      
      The intent of this option and associated logic is to control
      the ordering of revisions in deltagroups in changegroups. At the
      time it was implemented, only changegroup version 1 existed
      and generaldelta revlogs were just coming into the world. Changegroup
      version 1 requires that deltas be made against the last revision
      sent over the wire. Used with generaldelta, this created an
      impedance mismatch of sorts and resulted in changegroup producers
      spending a lot of time recomputing deltas.
      
      Revision reordering was introduced so outgoing revisions would be
      sent in "generaldelta order" and producers would be able to
      reuse internal deltas from storage.
      
      Later on, we introduced changegroup version 2. It supported denoting
      which revision a delta was against. So we no longer needed to
      sort outgoing revisions to ensure optimal delta generation from the
      producer. So, subsequent changegroup versions disabled reordering.
      
      We also later made the changelog not store deltas by default. And
      we also made the changelog send out deltas in storage order. Why we
      do this for changelog, I'm not sure. Maybe we want to preserve revision
      order across clones? It doesn't really matter for this commit.
      
      Fast forward to 2018. We want to abstract storage backends. And having
      changegroup code require knowledge about how deltas are stored
      internally interferes with that goal.
      
      This commit removes reordering control from changegroup generation.
      After this commit, the reordering behavior is:
      
      * The changelog is always sent out in storage order (no behavior
        change).
      * Non-changelog generaldelta revlogs are reordered to always be in DAG
        topological order (previously, generaldelta revlogs would be emitted
        in storage order for version 2 and 3 changegroups).
      * Non-changelog non-generaldelta revlogs are sent in storage order (no
        behavior change).
      * There exists no config option to override behavior.
      
      The big difference here is that generaldelta revlogs now *always* have
      their revisions sorted in DAG order before going out over the wire. This
      behavior was previously only done for changegroup version 1. Version 2
      and version 3 changegroups disabled reordering because the interchange
      format supported encoding arbitrary delta parents, so reordering wasn't
      strictly necessary.
      
      I can think of a few significant implications for this change.
      
      Because changegroup receivers will now see non-changelog revisions
      in DAG order instead of storage order, the internal storage order of
      manifests and files may differ substantially between producer and
      consumer. I don't think this matters that much, since the storage
      order of manifests and files is largely hidden from users. Only
      the storage order of changelog matters (because `hg log` shows the
      changelog in storage order). I don't think there should be any
      controversy here.
      
      The reordering of revisions has implications for changegroup producers.
      Previously, generaldelta revlogs would be emitted in storage order.
      And in the common case, the internally-stored delta could effectively
      be copied from disk into the deltagroup delta. This meant that emitting
      delta groups for generaldelta revlogs would be mostly linear read I/O.
      This is desirable for performance. With us now reordering generaldelta
      revlog revisions in DAG order, the read operations may use more random
      I/O instead of sequential I/O. This could result in performance
      loss. But with the prevalence of SSDs and fast random I/O, I'm not
      too worried. (Note: the optimal emission order for revlogs is actually
      delta encoding order. But the changegroup code wasn't doing that before
      or after this change. We could potentially implement that in a later
      commit.)
      
      Changegroups in DAG order will have implications for receivers.
      Previously, receiving storage order might mean seeing a number of
      interleaved branches. This would mean long delta chains, sparse
      I/O, and possibly more fulltext revisions instead of deltas, blowing
      up storage storage. (This is the same set of problems that sparse
      revlogs aims to address.) With the producer now sending revisions in DAG
      order, the receiver also stores revisions in DAG order. That means
      revisions for the same DAG branch are all grouped together. And this
      should yield better storage outcomes. In other words, sending the
      reordered changegroup allows the receiver to have better storage
      order and for the producer to not propagate its (possibly sub-optimal)
      internal storage order.
      
      On the mozilla-unified repository, this change influences bundle
      generation:
      
      $ hg bundle -t none-v2 -a
      before: time: real 355.680 secs (user 256.790+0.000 sys 16.820+0.000)
      after:  time: real 382.950 secs (user 281.700+0.000 sys 17.690+0.000)
      
      before: 7,150,228,967 bytes (uncompressed)
      after:  7,041,556,273 bytes (uncompressed)
      
      before: 1,669,063,234 bytes (zstd l=3)
      after:  1,628,598,830 bytes (zstd l=3)
      
      $ hg unbundle
      before: time: real 511.910 secs (user 466.750+0.000 sys 32.680+0.000)
      after:  time: real 487.790 secs (user 443.940+0.000 sys 30.840+0.000)
      
      00manifest.d size:
      source: 274,924,292 bytes
      before: 304,741,626 bytes
      after:  245,252,087 bytes
      
      .hg/store total file size:
      source: 2,649,133,490
      before: 2,680,888,130
      after:  2,627,875,673
      
      We see the bundle size drop. That's probably because if a revlog
      internally isn't storing a delta, it will choose to delta against
      the last emitted revision. And on repos with interleaved branches
      (like mozilla-unified), the previous revision could be an unrelated
      branch and therefore be a large delta. But with this patch, the
      previous revision is likely p1 or p2 and a delta should be small.
      
      We also see the manifest size drop by ~50 MB. It's worth noting that
      the manifest actually *increased* in size by ~25 MB in the old
      strategy and decreased ~25 MB from its source in the new strategy.
      Again, my explanation for this is that the DAG ordering in the
      changegroup is resulting in better grouping of revisions in the
      receiver, which results in more compact delta chains and higher
      storage efficiency.
      
      Unbundle time also dropped. I suspect this is due to the revlog having
      to work less to compute deltas since the incoming deltas are more
      optimal. i.e. the receiver spends less time resolving fulltext
      revisions as incoming deltas bounce around between DAG branches and
      delta chains.
      
      We also see bundle generation time increase. This is not desirable.
      However, the regression is only significant on the original repository:
      if we generate a bundle from the repository created from the new,
      always reordered bundles, we're close to baseline (if not at it with
      expected noise):
      
      $ hg bundle -t none-v2 -a
      before (original): time: real 355.680 secs (user 256.790+0.000 sys 16.820+0.000)
      after (original):  time: real 382.950 secs (user 281.700+0.000 sys 17.690+0.000)
      after (new repo):  time: real 362.280 secs (user 260.300+0.000 sys 17.700+0.000)
      
      This regression is a bit worrying because it will impact serving
      canonical repositories (that don't have optimal internal storage
      unless they are reordered - possibly as part of running
      `hg debugupgraderepo`). However, this regression will only be
      noticed by very large changegroups. And I'm guessing/hoping that
      any repository that large is using clonebundles to mitigate server
      load.
      
      Again, sending DAG order isn't the optimal send order for servers:
      sending in storage-delta order is. But in order to enable
      storage-optimal send order, we'll need a storage API that handles
      sorting. Future commits will introduce such an API.
      
      Differential Revision: https://phab.mercurial-scm.org/D4721
      db5501d9
  18. Sep 21, 2018
    • Gregory Szorc's avatar
      filelog: drop index attribute (API) · d9b3cc3d
      Gregory Szorc authored
      The previous commit removed the last consumer of the "index" attribute
      on the file storage interface. The index is an extremely low-level
      data structure that is revlog specific and isn't appropriate to
      expose as part of a generic storage API. There may be a market for
      an efficient data structure to obtain metadata on every revision for
      a file. But if there is, it should be designed using e.g. named
      attributes for lookup instead of a list-like of 8-tuples.
      
      Let's drop the attribute from filelog and remove the attribute from
      the file storage interface.
      
      Differential Revision: https://phab.mercurial-scm.org/D4720
      d9b3cc3d
  19. Sep 24, 2018
    • Gregory Szorc's avatar
      upgrade: report size of backing files, not internal storage size · 8dab7c8a
      Gregory Szorc authored
      upgrade.py is the only consumer of filelog.index, which I'd like
      to eliminate from the file storage interface.
      
      This commit changes the upgrade code to report the storage size
      of files by looking at the size of the files backing its storage
      instead of looking at the index.
      
      I'm not convinced the approach in this patch will live very long
      because it is relying on low-level attributes like "opener" and
      "files," which may behave very differently on non-revlog storage.
      But the data is only used for reporting purposes and it does get
      us one step closer to eliminating "index."
      
      A side-effect of this change is we now report the size of the revlog
      index data - not just the revision data. I think this is more
      accurate.
      
      Differential Revision: https://phab.mercurial-scm.org/D4717
      8dab7c8a
  20. Sep 21, 2018
    • Gregory Szorc's avatar
      lfs: access revlog directly · 62a53204
      Gregory Szorc authored
      LFS is monkeypatching filelog.filelog and is then accessing
      various filelog attributes in the monkeypatched function. This is all
      fine.
      
      But some of the attributes being accessed by LFS are revlog centric
      and shouldn't be exposed on the file storage interface.
      
      This commit changes the monkeypatched functions to access proxied
      attributes on self._revlog instead of self.
      
      This should be safe to do because non-revlog repositories should not
      be using filelog instances: instead they should have a separate class
      to represent file storage. So it is reasonable for LFS to assume the
      _revlog attribute exists and points to a revlog.
      
      Differential Revision: https://phab.mercurial-scm.org/D4714
      62a53204
  21. Sep 20, 2018
Loading