Skip to content
Snippets Groups Projects
  • Gregory Szorc's avatar
    db5501d93bcf
    changegroup: remove reordering control (BC) · db5501d93bcf
    Gregory Szorc authored
    This logic - including the experimental bundle.reorder option -
    was originally added in a8e3931e3fb5 in 2011 and then later ported
    to changegroup.py.
    
    The intent of this option and associated logic is to control
    the ordering of revisions in deltagroups in changegroups. At the
    time it was implemented, only changegroup version 1 existed
    and generaldelta revlogs were just coming into the world. Changegroup
    version 1 requires that deltas be made against the last revision
    sent over the wire. Used with generaldelta, this created an
    impedance mismatch of sorts and resulted in changegroup producers
    spending a lot of time recomputing deltas.
    
    Revision reordering was introduced so outgoing revisions would be
    sent in "generaldelta order" and producers would be able to
    reuse internal deltas from storage.
    
    Later on, we introduced changegroup version 2. It supported denoting
    which revision a delta was against. So we no longer needed to
    sort outgoing revisions to ensure optimal delta generation from the
    producer. So, subsequent changegroup versions disabled reordering.
    
    We also later made the changelog not store deltas by default. And
    we also made the changelog send out deltas in storage order. Why we
    do this for changelog, I'm not sure. Maybe we want to preserve revision
    order across clones? It doesn't really matter for this commit.
    
    Fast forward to 2018. We want to abstract storage backends. And having
    changegroup code require knowledge about how deltas are stored
    internally interferes with that goal.
    
    This commit removes reordering control from changegroup generation.
    After this commit, the reordering behavior is:
    
    * The changelog is always sent out in storage order (no behavior
      change).
    * Non-changelog generaldelta revlogs are reordered to always be in DAG
      topological order (previously, generaldelta revlogs would be emitted
      in storage order for version 2 and 3 changegroups).
    * Non-changelog non-generaldelta revlogs are sent in storage order (no
      behavior change).
    * There exists no config option to override behavior.
    
    The big difference here is that generaldelta revlogs now *always* have
    their revisions sorted in DAG order before going out over the wire. This
    behavior was previously only done for changegroup version 1. Version 2
    and version 3 changegroups disabled reordering because the interchange
    format supported encoding arbitrary delta parents, so reordering wasn't
    strictly necessary.
    
    I can think of a few significant implications for this change.
    
    Because changegroup receivers will now see non-changelog revisions
    in DAG order instead of storage order, the internal storage order of
    manifests and files may differ substantially between producer and
    consumer. I don't think this matters that much, since the storage
    order of manifests and files is largely hidden from users. Only
    the storage order of changelog matters (because `hg log` shows the
    changelog in storage order). I don't think there should be any
    controversy here.
    
    The reordering of revisions has implications for changegroup producers.
    Previously, generaldelta revlogs would be emitted in storage order.
    And in the common case, the internally-stored delta could effectively
    be copied from disk into the deltagroup delta. This meant that emitting
    delta groups for generaldelta revlogs would be mostly linear read I/O.
    This is desirable for performance. With us now reordering generaldelta
    revlog revisions in DAG order, the read operations may use more random
    I/O instead of sequential I/O. This could result in performance
    loss. But with the prevalence of SSDs and fast random I/O, I'm not
    too worried. (Note: the optimal emission order for revlogs is actually
    delta encoding order. But the changegroup code wasn't doing that before
    or after this change. We could potentially implement that in a later
    commit.)
    
    Changegroups in DAG order will have implications for receivers.
    Previously, receiving storage order might mean seeing a number of
    interleaved branches. This would mean long delta chains, sparse
    I/O, and possibly more fulltext revisions instead of deltas, blowing
    up storage storage. (This is the same set of problems that sparse
    revlogs aims to address.) With the producer now sending revisions in DAG
    order, the receiver also stores revisions in DAG order. That means
    revisions for the same DAG branch are all grouped together. And this
    should yield better storage outcomes. In other words, sending the
    reordered changegroup allows the receiver to have better storage
    order and for the producer to not propagate its (possibly sub-optimal)
    internal storage order.
    
    On the mozilla-unified repository, this change influences bundle
    generation:
    
    $ hg bundle -t none-v2 -a
    before: time: real 355.680 secs (user 256.790+0.000 sys 16.820+0.000)
    after:  time: real 382.950 secs (user 281.700+0.000 sys 17.690+0.000)
    
    before: 7,150,228,967 bytes (uncompressed)
    after:  7,041,556,273 bytes (uncompressed)
    
    before: 1,669,063,234 bytes (zstd l=3)
    after:  1,628,598,830 bytes (zstd l=3)
    
    $ hg unbundle
    before: time: real 511.910 secs (user 466.750+0.000 sys 32.680+0.000)
    after:  time: real 487.790 secs (user 443.940+0.000 sys 30.840+0.000)
    
    00manifest.d size:
    source: 274,924,292 bytes
    before: 304,741,626 bytes
    after:  245,252,087 bytes
    
    .hg/store total file size:
    source: 2,649,133,490
    before: 2,680,888,130
    after:  2,627,875,673
    
    We see the bundle size drop. That's probably because if a revlog
    internally isn't storing a delta, it will choose to delta against
    the last emitted revision. And on repos with interleaved branches
    (like mozilla-unified), the previous revision could be an unrelated
    branch and therefore be a large delta. But with this patch, the
    previous revision is likely p1 or p2 and a delta should be small.
    
    We also see the manifest size drop by ~50 MB. It's worth noting that
    the manifest actually *increased* in size by ~25 MB in the old
    strategy and decreased ~25 MB from its source in the new strategy.
    Again, my explanation for this is that the DAG ordering in the
    changegroup is resulting in better grouping of revisions in the
    receiver, which results in more compact delta chains and higher
    storage efficiency.
    
    Unbundle time also dropped. I suspect this is due to the revlog having
    to work less to compute deltas since the incoming deltas are more
    optimal. i.e. the receiver spends less time resolving fulltext
    revisions as incoming deltas bounce around between DAG branches and
    delta chains.
    
    We also see bundle generation time increase. This is not desirable.
    However, the regression is only significant on the original repository:
    if we generate a bundle from the repository created from the new,
    always reordered bundles, we're close to baseline (if not at it with
    expected noise):
    
    $ hg bundle -t none-v2 -a
    before (original): time: real 355.680 secs (user 256.790+0.000 sys 16.820+0.000)
    after (original):  time: real 382.950 secs (user 281.700+0.000 sys 17.690+0.000)
    after (new repo):  time: real 362.280 secs (user 260.300+0.000 sys 17.700+0.000)
    
    This regression is a bit worrying because it will impact serving
    canonical repositories (that don't have optimal internal storage
    unless they are reordered - possibly as part of running
    `hg debugupgraderepo`). However, this regression will only be
    noticed by very large changegroups. And I'm guessing/hoping that
    any repository that large is using clonebundles to mitigate server
    load.
    
    Again, sending DAG order isn't the optimal send order for servers:
    sending in storage-delta order is. But in order to enable
    storage-optimal send order, we'll need a storage API that handles
    sorting. Future commits will introduce such an API.
    
    Differential Revision: https://phab.mercurial-scm.org/D4721
    db5501d93bcf
    History
    changegroup: remove reordering control (BC)
    Gregory Szorc authored
    This logic - including the experimental bundle.reorder option -
    was originally added in a8e3931e3fb5 in 2011 and then later ported
    to changegroup.py.
    
    The intent of this option and associated logic is to control
    the ordering of revisions in deltagroups in changegroups. At the
    time it was implemented, only changegroup version 1 existed
    and generaldelta revlogs were just coming into the world. Changegroup
    version 1 requires that deltas be made against the last revision
    sent over the wire. Used with generaldelta, this created an
    impedance mismatch of sorts and resulted in changegroup producers
    spending a lot of time recomputing deltas.
    
    Revision reordering was introduced so outgoing revisions would be
    sent in "generaldelta order" and producers would be able to
    reuse internal deltas from storage.
    
    Later on, we introduced changegroup version 2. It supported denoting
    which revision a delta was against. So we no longer needed to
    sort outgoing revisions to ensure optimal delta generation from the
    producer. So, subsequent changegroup versions disabled reordering.
    
    We also later made the changelog not store deltas by default. And
    we also made the changelog send out deltas in storage order. Why we
    do this for changelog, I'm not sure. Maybe we want to preserve revision
    order across clones? It doesn't really matter for this commit.
    
    Fast forward to 2018. We want to abstract storage backends. And having
    changegroup code require knowledge about how deltas are stored
    internally interferes with that goal.
    
    This commit removes reordering control from changegroup generation.
    After this commit, the reordering behavior is:
    
    * The changelog is always sent out in storage order (no behavior
      change).
    * Non-changelog generaldelta revlogs are reordered to always be in DAG
      topological order (previously, generaldelta revlogs would be emitted
      in storage order for version 2 and 3 changegroups).
    * Non-changelog non-generaldelta revlogs are sent in storage order (no
      behavior change).
    * There exists no config option to override behavior.
    
    The big difference here is that generaldelta revlogs now *always* have
    their revisions sorted in DAG order before going out over the wire. This
    behavior was previously only done for changegroup version 1. Version 2
    and version 3 changegroups disabled reordering because the interchange
    format supported encoding arbitrary delta parents, so reordering wasn't
    strictly necessary.
    
    I can think of a few significant implications for this change.
    
    Because changegroup receivers will now see non-changelog revisions
    in DAG order instead of storage order, the internal storage order of
    manifests and files may differ substantially between producer and
    consumer. I don't think this matters that much, since the storage
    order of manifests and files is largely hidden from users. Only
    the storage order of changelog matters (because `hg log` shows the
    changelog in storage order). I don't think there should be any
    controversy here.
    
    The reordering of revisions has implications for changegroup producers.
    Previously, generaldelta revlogs would be emitted in storage order.
    And in the common case, the internally-stored delta could effectively
    be copied from disk into the deltagroup delta. This meant that emitting
    delta groups for generaldelta revlogs would be mostly linear read I/O.
    This is desirable for performance. With us now reordering generaldelta
    revlog revisions in DAG order, the read operations may use more random
    I/O instead of sequential I/O. This could result in performance
    loss. But with the prevalence of SSDs and fast random I/O, I'm not
    too worried. (Note: the optimal emission order for revlogs is actually
    delta encoding order. But the changegroup code wasn't doing that before
    or after this change. We could potentially implement that in a later
    commit.)
    
    Changegroups in DAG order will have implications for receivers.
    Previously, receiving storage order might mean seeing a number of
    interleaved branches. This would mean long delta chains, sparse
    I/O, and possibly more fulltext revisions instead of deltas, blowing
    up storage storage. (This is the same set of problems that sparse
    revlogs aims to address.) With the producer now sending revisions in DAG
    order, the receiver also stores revisions in DAG order. That means
    revisions for the same DAG branch are all grouped together. And this
    should yield better storage outcomes. In other words, sending the
    reordered changegroup allows the receiver to have better storage
    order and for the producer to not propagate its (possibly sub-optimal)
    internal storage order.
    
    On the mozilla-unified repository, this change influences bundle
    generation:
    
    $ hg bundle -t none-v2 -a
    before: time: real 355.680 secs (user 256.790+0.000 sys 16.820+0.000)
    after:  time: real 382.950 secs (user 281.700+0.000 sys 17.690+0.000)
    
    before: 7,150,228,967 bytes (uncompressed)
    after:  7,041,556,273 bytes (uncompressed)
    
    before: 1,669,063,234 bytes (zstd l=3)
    after:  1,628,598,830 bytes (zstd l=3)
    
    $ hg unbundle
    before: time: real 511.910 secs (user 466.750+0.000 sys 32.680+0.000)
    after:  time: real 487.790 secs (user 443.940+0.000 sys 30.840+0.000)
    
    00manifest.d size:
    source: 274,924,292 bytes
    before: 304,741,626 bytes
    after:  245,252,087 bytes
    
    .hg/store total file size:
    source: 2,649,133,490
    before: 2,680,888,130
    after:  2,627,875,673
    
    We see the bundle size drop. That's probably because if a revlog
    internally isn't storing a delta, it will choose to delta against
    the last emitted revision. And on repos with interleaved branches
    (like mozilla-unified), the previous revision could be an unrelated
    branch and therefore be a large delta. But with this patch, the
    previous revision is likely p1 or p2 and a delta should be small.
    
    We also see the manifest size drop by ~50 MB. It's worth noting that
    the manifest actually *increased* in size by ~25 MB in the old
    strategy and decreased ~25 MB from its source in the new strategy.
    Again, my explanation for this is that the DAG ordering in the
    changegroup is resulting in better grouping of revisions in the
    receiver, which results in more compact delta chains and higher
    storage efficiency.
    
    Unbundle time also dropped. I suspect this is due to the revlog having
    to work less to compute deltas since the incoming deltas are more
    optimal. i.e. the receiver spends less time resolving fulltext
    revisions as incoming deltas bounce around between DAG branches and
    delta chains.
    
    We also see bundle generation time increase. This is not desirable.
    However, the regression is only significant on the original repository:
    if we generate a bundle from the repository created from the new,
    always reordered bundles, we're close to baseline (if not at it with
    expected noise):
    
    $ hg bundle -t none-v2 -a
    before (original): time: real 355.680 secs (user 256.790+0.000 sys 16.820+0.000)
    after (original):  time: real 382.950 secs (user 281.700+0.000 sys 17.690+0.000)
    after (new repo):  time: real 362.280 secs (user 260.300+0.000 sys 17.700+0.000)
    
    This regression is a bit worrying because it will impact serving
    canonical repositories (that don't have optimal internal storage
    unless they are reordered - possibly as part of running
    `hg debugupgraderepo`). However, this regression will only be
    noticed by very large changegroups. And I'm guessing/hoping that
    any repository that large is using clonebundles to mitigate server
    load.
    
    Again, sending DAG order isn't the optimal send order for servers:
    sending in storage-delta order is. But in order to enable
    storage-optimal send order, we'll need a storage API that handles
    sorting. Future commits will introduce such an API.
    
    Differential Revision: https://phab.mercurial-scm.org/D4721