Skip to content
Snippets Groups Projects
  1. Oct 18, 2016
  2. Oct 11, 2016
  3. Oct 16, 2016
    • Gregory Szorc's avatar
      changegroup: increase write buffer size to 128k · 260af198
      Gregory Szorc authored
      By default, Python defers to the operating system for choosing the
      default buffer size on opened files. On my Linux machine, the default
      is 4k, which is really small for 2016.
      
      This patch bumps the write buffer size when writing
      changegroups/bundles to 128k. This matches the 128k read buffer
      we already use on revlogs.
      
      It's worth noting that this only impacts when writing to an explicit
      file (such as during `hg bundle`). Buffers when writing to bundle
      files via the repo vfs or to a temporary file are not impacted.
      
      When producing a none-v2 bundle file of the mozilla-unified repository,
      this change caused the number of write() system calls to drop from
      952,449 to 29,788. After this change, the most frequent system
      calls are fstat(), read(), lseek(), and open(). There were
      2,523,672 system calls after this patch (so a net decrease of
      ~950k is statistically significant).
      
      This change shows no performance change on my system. But I have a
      high-end system with a fast SSD. It is quite possible this change
      will have a significant impact on network file systems, where
      extra network round trips due to excessive I/O system calls could
      introduce significant latency.
      260af198
  4. Oct 13, 2016
    • Pierre-Yves David's avatar
      changegroup: skip delta when the underlying revlog do not use them · 6b0741d6
      Pierre-Yves David authored
      Revlog can now be configured to store full snapshot only. This is used on the
      changelog. However, the changegroup packing was still recomputing deltas to be
      sent over the wire.
      
      We now just reuse the full snapshot directly in this case, skipping delta
      computation. This provides use with a large speed up(-30%):
      
      # perfchangegroupchangelog on mercurial
      ! wall 2.010326 comb 2.020000 user 2.000000 sys 0.020000 (best of 5)
      ! wall 1.382039 comb 1.380000 user 1.370000 sys 0.010000 (best of 8)
      
      # perfchangegroupchangelog on pypy
      ! wall 5.792589 comb 5.780000 user 5.780000 sys 0.000000 (best of 3)
      ! wall 3.911158 comb 3.920000 user 3.900000 sys 0.020000 (best of 3)
      
      # perfchangegroupchangelog on mozilla central
      ! wall 20.683727 comb 20.680000 user 20.630000 sys 0.050000 (best of 3)
      ! wall 14.190204 comb 14.190000 user 14.150000 sys 0.040000 (best of 3)
      
      Many tests have to be updated because of the change in bundle content. All
      theses update have been verified.  Because diffing changelog was not very
      valuable, the resulting bundle have similar size (often a bit smaller):
      
      # full bundle of mozilla central
      with delta:    1142740533B
      without delta: 1142173300B
      
      So this is a win all over the board.
      6b0741d6
  5. Oct 14, 2016
  6. Oct 18, 2016
  7. Oct 17, 2016
  8. Oct 16, 2016
    • Gregory Szorc's avatar
      wireproto: compress data from a generator · d1051954
      Gregory Szorc authored
      Currently, the "getbundle" wire protocol command obtains a generator of
      data, converts it to a util.chunkbuffer, then converts it back to a
      generator via the protocol's groupchunks() implementation. For the SSH
      protocol, groupchunks() simply reads 4kb chunks then write()s the
      data to a file descriptor. For the HTTP protocol, groupchunks() reads
      32kb chunks, feeds those into a zlib compressor, emits compressed data
      as it is available, and that is sent to the WSGI layer, where it is
      likely turned into HTTP chunked transfer chunks as is or further
      buffered and turned into a larger chunk.
      
      For both the SSH and HTTP protocols, there is inefficiency from using
      util.chunkbuffer.
      
      For SSH, emitting consistent 4kb chunks sounds nice. However, the file
      descriptor it is writing to is almost certainly buffered. That means
      that a Python .write() probably doesn't translate into exactly what is
      written to the I/O layer.
      
      For HTTP, we're going through an intermediate layer to zlib compress
      data. So all util.chunkbuffer is doing is ensuring that the chunks we
      feed into the zlib compressor are of uniform size. This means more CPU
      time in Python buffering and emitting chunks in util.chunkbuffer but
      fewer function calls to zlib.
      
      This patch introduces and implements a new wire protocol abstract
      method: compresschunks(). It is like groupchunks() except it operates
      on a generator instead of something with a .read(). The SSH
      implementation simply proxies chunks. The HTTP implementation uses
      zlib compression.
      
      To avoid duplicate code, the HTTP groupchunks() has been reimplemented
      in terms of compresschunks().
      
      To prove this all works, the "getbundle" wire protocol command has been
      switched to compresschunks(). This removes the util.chunkbuffer from
      that command. Now, data essentially streams straight from the
      changegroup emitter to the wire, possibly through a zlib compressor.
      Generators all the way, baby.
      
      There were slim to no performance changes on the server as measured
      with the mozilla-central repository. This is likely because CPU
      time is dominated by reading revlogs, producing the changegroup, and
      zlib compressing the output stream. Still, this brings us a little
      closer to our ideal of using generators everywhere.
      d1051954
  9. Oct 17, 2016
  10. Oct 11, 2016
    • Gábor Stefanik's avatar
      copies: make _checkcopies handle copy sequences spanning the TCA (issue4028) · 1894c830
      Gábor Stefanik authored
      When working in a rotated DAG (for a graftlike merge), there can be files
      that are renamed both between the base and the topological CA, and between
      the TCA and the endpoint farther from the base. Such renames span the TCA
      (and thus need both passes of _checkcopies to be fully detected), but may
      not necessarily be divergent.
      
      Make _checkcopies return "incomplete copies" and "incomplete divergences"
      in this case, and let mergecopies recombine them once data from both passes
      of _checkcopies is available.
      
      With this patch, all known cases involving renames and grafts pass.
      
      (Developed together with Pierre-Yves David)
      1894c830
    • Gábor Stefanik's avatar
      checkcopies: add logic to handle remotebase · b94b92f0
      Gábor Stefanik authored
      As the two _checkcopies passes' ranges are separated by tca, not base,
      only one of the two passes will actually encounter the base.
      Pass "remotebase" to the other pass to let it know not to expect passing
      over the base. This is required for handling a few unusual rename cases.
      b94b92f0
  11. Oct 04, 2016
    • Gábor Stefanik's avatar
      mergecopies: add logic to process incomplete data · a005c33d
      Gábor Stefanik authored
      We first combine incomplete copies on the two sides of the topological CA
      into complete copies.
      Any leftover incomplete copies are then combined with the incomplete
      divergences to reconstruct divergences spanning over the topological CA.
      Finally we promote any divergences falsely flagged as incomplete to full
      divergences.
      
      Right now, there is nothing generating incomplete copy/divergence data,
      so this code does nothing. Changes to _checkcopies to populate these
      dicts are coming later in this series.
      a005c33d
  12. Oct 12, 2016
    • Gábor Stefanik's avatar
      checkcopies: handle divergences contained entirely in tca::ctx · 856ead83
      Gábor Stefanik authored
      During a graftlike merge, _checkcopies runs from ctx to tca, possibly
      passing over the merge base. If there is a rename both before and after
      the base, then we're actually dealing with divergent renames.
      If there is no rename on the other side of tca, then the divergence is
      contained entirely in the range of one _checkcopies invocation, and
      should be detected "in the loop" without having to rely on the other
      _checkcopies pass.
      856ead83
  13. Aug 25, 2016
    • Gábor Stefanik's avatar
      update: enable copy tracing for backwards and non-linear updates · a2804ddc
      Gábor Stefanik authored
      As a followup to the issue4028 series, this fixes a variant of the issue
      that can occur when updating with uncommited local changes.
      
      The duplicated .hgsub warning is coming from wc.dirty(). We would previously
      skip this call because it's only relevant when we're going to perform copy
      tracing, which we didn't do before.
      
      The change to the update summary line is because we now treat the rename as a
      proper rename (which counts as a change), rather than an add+delete pair
      (which counts as a change and a delete).
      a2804ddc
  14. Sep 26, 2016
  15. Aug 20, 2016
  16. Oct 13, 2016
    • Gábor Stefanik's avatar
      mergecopies: invoke _computenonoverlap for both base and tca during merges · 0accd5a5
      Gábor Stefanik authored
      The algorithm of _checkcopies can only walk backwards in the DAG, never
      forward. Because of this, the two _checkcopies patches need to run from
      their respective endpoints to the TCA to cover the entire subgraph where
      the merge is being performed. However, detection of files new in both
      endpoints, as well as directory rename detection, need to run with respect
      to the merge base, so we need lists of new files both from the TCA's and
      the merge base's viewpoint to correctly detect renames in a graft-like
      merge scenario.
      
      (Series reworked by Pierre-Yves David)
      0accd5a5
  17. Oct 17, 2016
  18. Oct 13, 2016
    • Gábor Stefanik's avatar
      copies: make _checkcopies handle simple renames in a rotated DAG · 88626de1
      Gábor Stefanik authored
      This introduces a distinction between "merge base" and
      "topological common ancestor". During a regular merge, these two are
      identical. Graft, however, performs a merge in a rotated DAG, where the
      merge base will not be a common ancestor at all in the
      original DAG.
      
      To correctly find copies in case of a graft, we need to take both the
      merge base and the topological CA into account, and track any renames
      between them in reverse. Fortunately we can detect this in advance,
      see comment in the code about "backwards".
      
      This patch only supports finding non-divergent renames contained entirely
      between the merge base and the topological CA. Further patches are coming
      to support more complex cases.
      
      (Pierre-Yves David was involved in the cleanup of this patch.)
      88626de1
    • Gábor Stefanik's avatar
      copies: compute a suitable TCA if base turns out to be unsuitable · 8c69c52c
      Gábor Stefanik authored
      This will be used later in an update to _checkcopies.
      
      (Pierre-Yves David was involved in the cleanup of this patch.)
      8c69c52c
  19. Oct 12, 2016
    • Gábor Stefanik's avatar
      copies: detect graft-like merges · 368e27eb
      Gábor Stefanik authored
      Right now, nothing changes as a result of this, but we want to handle
      grafts differently from ordinary merges later.
      
      (Series developed together with Pierre-Yves David)
      368e27eb
    • Gábor Stefanik's avatar
      tests: introduce tests for grafting through renames · 509d2925
      Gábor Stefanik authored
      These cover all currently known cases of renames being grafted,
      or changes being grafted through renames.
      
      Right now, most of these cases are broken. Later patches in this series
      will make them behave correctly.
      
      The testcases heavily rely on each other, which would make it very difficult
      to separate them and add them one-by-one for each case fixed by a patch.
      Separating them should perhaps be a 4.1 task, if it doesn't slow down
      the tests too much.
      
      (Developed together with Pierre-Yves David)
      509d2925
  20. Oct 17, 2016
    • Mads Kiilerich's avatar
      largefiles: fix 'deleted' files sometimes persistently appearing with R status · 328545c7
      Mads Kiilerich authored
      A code snippet that has been around since largefiles was introduced was wrong:
      Standins no longer found in lfdirstate has *not* been removed -
      they have probably just been deleted ... or not created.
      
      This wrong reporting did that 'up -C' didn't undo the change and didn't sync
      the two dirstates.
      
      Instead of reporting such files as removed, propagate the deletion to the
      standin file and report the file as deleted.
      328545c7
  21. Oct 16, 2016
    • Mads Kiilerich's avatar
      largefiles: more safe handling of interruptions while updating modifications · 56b93023
      Mads Kiilerich authored
      Largefiles are fragile with the design where dirstate and lfdirstate must be
      kept in sync.
      
      To be less fragile, mark all clean largefiles as unsure ("normallookup") before
      updating standins. After standins have been updated and we know exactly which
      largefile standins actually was changed, mark the unchanged largefiles back to
      clean ("normal").
      
      This will make the failure mode more safe. If interrupted, the next command
      will continue to perform extra hashing of all largefiles. That will do that all
      largefiles that are out of sync with their standin will be marked dirty and
      they will show up in status and can be cleaned with update --clean.
      56b93023
    • Mads Kiilerich's avatar
      largefiles: test coverage of fatal interruption of update · 4999c12c
      Mads Kiilerich authored
      Test using existing changesets in a clean working directory, revealing problems
      with files that don't show up as modified or do show up as removed when they
      just not have been written yet.
      4999c12c
  22. Oct 12, 2016
    • Gábor Stefanik's avatar
      checkcopies: add a sanity check against false-positive copies · 8a864844
      Gábor Stefanik authored
      When grafting a copy backwards through a rename, a copy is wrongly detected,
      which causes the graft to be applied inappropriately, in a destructive way.
      Make sure that the old file name really exists in the common ancestor,
      and bail out if it doesn't.
      
      This fixes the aggravated case of bug 5343, although the basic issue
      (failure to duplicate the copy information) still occurs.
      8a864844
  23. Oct 16, 2016
    • Gregory Szorc's avatar
      exchange: refactor APIs to obtain bundle data (API) · 3e86261b
      Gregory Szorc authored
      Currently, exchange.getbundle() returns either a cg1unpacker or a
      util.chunkbuffer (in the case of bundle2). This is kinda OK, as
      both expose a .read() to consumers. However, localpeer.getbundle()
      has code inferring what the response type is based on arguments and
      converts the util.chunkbuffer returned in the bundle2 case to a
      bundle2.unbundle20 instance. This is a sign that the API for
      exchange.getbundle() is not ideal because it doesn't consistently
      return an "unbundler" instance.
      
      In addition, unbundlers mask the fact that there is an underlying
      generator of changegroup data. In both cg1 and bundle2, this generator
      is being fed into a util.chunkbuffer so it can be re-exposed as a
      file object.
      
      util.chunkbuffer is a nice abstraction. However, it should only be
      used "at the edges." This is because keeping data as a generator is
      more efficient than converting it to a chunkbuffer, especially if we
      convert that chunkbuffer back to a generator (as is the case in some
      code paths currently).
      
      This patch refactors exchange.getbundle() into
      exchange.getbundlechunks(). The new API returns an iterator of chunks
      instead of a file-like object.
      
      Callers of exchange.getbundle() have been updated to use the new API.
      
      There is a minor change of behavior in test-getbundle.t. This is
      because `hg debuggetbundle` isn't defining bundlecaps. As a result,
      a cg1 data stream and unpacker is being produced. This is getting fed
      into a new bundle20 instance via bundle2.writebundle(), which uses
      a backchannel mechanism between changegroup generation to add the
      "nbchanges" part parameter. I never liked this backchannel mechanism
      and I plan to remove it someday. `hg bundle` still produces the
      "nbchanges" part parameter, so there should be no user-visible
      change of behavior. I consider this "regression" a bug in
      `hg debuggetbundle`. And that bug is captured by an existing
      "TODO" in the code to use bundle2 capabilities.
      3e86261b
  24. Oct 12, 2016
  25. Oct 11, 2016
  26. Oct 13, 2016
    • Mads Kiilerich's avatar
      cmdutil: satisfy expections in dirstateguard.__del__, even if __init__ fails · 144d8fe2
      Mads Kiilerich authored
      Python "delstructors" are terrible - this one because it assumed that __init__
      had completed before it was called. That would not necessarily be the case if
      the repository was read only or broken and saving the dirstate thus failed in
      unexpected ways. That could give confusing warnings about missing '_active'
      after failures.
      
      To fix that, make sure all member variables are "declared" before doing
      anything that possibly could fail. [Famous last words.]
      144d8fe2
    • Mads Kiilerich's avatar
      util: increase filechunkiter size to 128k · 7356e6b1
      Mads Kiilerich authored
      util.filechunkiter has been using a chunk size of 64k for more than 10 years,
      also in years where Moore's law still was a law. It is probably ok to bump it
      now and perhaps get a slight win in some cases.
      
      Also, largefiles have been using 128k for a long time. Specifying that size
      multiple times (or forgetting to do it) seems a bit stupid. Decreasing it to
      64k also seems unfortunate.
      
      Thus, we will set the default chunksize to 128k and use the default everywhere.
      7356e6b1
  27. Oct 12, 2016
    • Mads Kiilerich's avatar
      largefiles: always use filechunkiter when iterating files · 736f92c4
      Mads Kiilerich authored
      Before, we would sometimes use the default iterator over large files. That
      iterator is line based and would add extra buffering and use odd chunk sizes
      which could give some overhead.
      
      copyandhash can't just apply a filechunkiter as it sometimes is passed a
      genuine generator when downloading remotely.
      736f92c4
  28. Oct 14, 2016
  29. Oct 10, 2016
    • Yuya Nishihara's avatar
      revset: make follow() reject more than one start revisions · d61c42c1
      Yuya Nishihara authored
      Taking only the last revision is inconsistent because ancestors(set) follows
      all revisions given, and theoretically follow(startrev=set) == ancestors(set).
      I'm planning to add a support for multiple start revisions, but that won't fit
      to the 4.0 time frame. So reject multiple revisions now to avoid future BC.
      
      len(revs) might be slow if revs were large, but we don't care since a valid
      revs should have only one element.
      d61c42c1
  30. Oct 16, 2016
    • Gregory Szorc's avatar
      bundle2: only emit compressed chunks if they have data · 9626022f
      Gregory Szorc authored
      This is similar to 58467204cac0. Not all calls into the compressor
      return compressed data, as the compressor may buffer compressed
      output internally. It is cheaper to check for empty chunks than to
      send empty chunks through the generator.
      
      When generating a gzip-v2 bundle of the mozilla-unified repo, this
      change results in 50,093 empty chunks not being sent through the
      generator (out of 1,902,996 total input chunks).
      9626022f
Loading