Skip to content
Snippets Groups Projects
  1. Oct 17, 2016
  2. Nov 04, 2016
    • Durham Goode's avatar
      manifest: add __nonzero__ method · b19291e5d506
      Durham Goode authored
      This adds a __nonzero__ method to manifestdict. This isn't strictly necessary in
      the vanilla Mercurial implementation, since Python will handle nonzero checks by
      using __len__, but having it implemented here makes it easier for alternative
      implementations to implement __nonzero__ and have them be plug-n-play with the
      normal implementation.
      b19291e5d506
  3. Nov 05, 2016
    • Pulkit Goyal's avatar
      py3: have bytes version of sys.argv · a2f2f694dce9
      Pulkit Goyal authored
      sys.argv returns unicodes on Python 3. We need a bytes version for us.
      There was also a python bug/feature request which wanted then to implement
      one. They rejected and it is quoted in one of the comments that we can use
      fsencode() to get a bytes version of sys.argv. Though not sure about its
      correctness.
      
      Link to the comment: http://bugs.python.org/issue8776#msg217416
      
      After this patch we will have pycompat.sysargv which will return us bytes
      version of sys.argv. If this patch goes in, i will like to make transformer
      rewrite sys.argv with pycompat.argv because there are lot of occurences.
      a2f2f694dce9
  4. Oct 09, 2016
  5. Oct 17, 2016
  6. Oct 20, 2016
  7. Nov 07, 2016
    • Gregory Szorc's avatar
      bdiff: replace hash algorithm · e1d6aa0e4c3a
      Gregory Szorc authored
      This patch replaces lyhash with the hash algorithm used by diffutils.
      The algorithm has its origins in Git commit 2e9d1410, which is all the
      way back from 1992. The license header in the code at that revision
      in GPL v2.
      
      I have not performed an extensive analysis of the distribution
      (and therefore buckets) of hash output. However, `hg perfbdiff`
      gives some clear wins. I'd like to think that if it is good enough
      for diffutils it is good enough for us?
      
      From the mozilla-unified repository:
      
      $ perfbdiff -m 3041e4d59df2
      ! wall 0.053271 comb 0.060000 user 0.060000 sys 0.000000 (best of 100)
      ! wall 0.035827 comb 0.040000 user 0.040000 sys 0.000000 (best of 100)
      
      $ perfbdiff 0e9928989e9c --alldata --count 100
      ! wall 6.204277 comb 6.200000 user 6.200000 sys 0.000000 (best of 3)
      ! wall 4.309710 comb 4.300000 user 4.300000 sys 0.000000 (best of 3)
      
      From the hg repo:
      
      $ perfbdiff 35000 --alldata --count 1000
      ! wall 0.660358 comb 0.660000 user 0.660000 sys 0.000000 (best of 15)
      ! wall 0.534092 comb 0.530000 user 0.530000 sys 0.000000 (best of 19)
      
      Looking at the generated assembly and statistical profiler output
      from the kernel level, I believe there is room to make this function
      even faster. Namely, we're still consuming data character by character
      instead of at the word level. This translates to more loop iterations
      and more instructions.
      
      At this juncture though, the real performance killer is that we're
      hashing every line. We should get a significant speedup if we change
      the algorithm to find the longest prefix, longest suffix, treat those
      as single "lines" and then only do the line splitting and hashing on
      the parts that are different. That will require a lot of C code,
      however. I'm optimistic this approach could result in a ~2x speedup.
      e1d6aa0e4c3a
  8. Nov 05, 2016
    • Gregory Szorc's avatar
      profiling: make statprof the default profiler (BC) · 3fd53cc1aad8
      Gregory Szorc authored
      The statprof sampling profiler runs with significantly less overhead.
      Its data is therefore more useful. Furthermore, its default output
      shows the hotpath by default, which I've found to be way more useful
      than the default profiler's function time table.
      
      There is one behavioral regression with this change worth noting:
      the statprof profiler currently doesn't profile individual hgweb
      requests like lsprof does. This is because the current implementation
      of statprof only profiles the thread that started profiling.
      
      The ability for lsprof to profile individual hgweb requests is
      relatively new and likely not widely used. Furthermore, I have plans
      to modify statprof to support profiling multiple threads. I expect
      that change to go through several iterations. I'm submitting this
      patch first so there is more time to test statprof. Perfect is the
      enemy of good.
      3fd53cc1aad8
    • Gregory Szorc's avatar
      profiling: use vendored statprof and upstream enhancements (BC) · faf1b8923da2
      Gregory Szorc authored
      Now that the statprof module is vendored and suitable for use, we
      switch our statprof profiler to use it. This required some minor
      changes because of drift between the official statprof profiler
      and the vendored copy.
      
      We also incorporate Facebook's improvements from the "statprofext"
      extension at
      https://bitbucket.org/facebook/hg-experimental, notably support for
      different display formats.
      
      Because statprof output is different, this is marked as BC. Although
      most users likely won't notice since most users don't profile.
      faf1b8923da2
  9. Oct 20, 2016
  10. Nov 06, 2016
    • Gregory Szorc's avatar
      bdiff: don't check border condition in loop · d500ddae7494
      Gregory Szorc authored
      `plast = a + len - 1`. So, this "for" loop iterates from "a" to "plast",
      inclusive. So, `p == plast` can only be true on the final iteration
      of the loop. So checking for it on every loop iteration is wasteful.
      
      This patch simply decreases the upper bound of the loop by 1 and
      adds an explicit check after iteration for the `p == plast` case.
      We can't simply add 1 to the initial value for "i" because that
      doesn't do the correct thing on empty input strings.
      
      `perfbdiff -m 3041e4d59df2` on the Firefox repo becomes significantly
      faster:
      
      ! wall 0.072763 comb 0.070000 user 0.070000 sys 0.000000 (best of 100)
      ! wall 0.053221 comb 0.060000 user 0.060000 sys 0.000000 (best of 100)
      
      For the curious, this code has its origins in 8b067bde6679, which is
      the changeset that introduced bdiff.c in 2005.
      
      Also, GNU diffutils is able to perform a similar line-based diff in
      under 20ms. So there's likely more perf wins to be found in this code.
      One of them is the hashing algorithm. But it looks like mpm spent
      some time testing hash collisions in d0c48891dd4a. I'd like to do the
      same before switching away from lyhash, just to be on the safe side.
      d500ddae7494
    • Gregory Szorc's avatar
      perf: add perfbdiff · c8fa7ad1ff90
      Gregory Szorc authored
      bdiff shows up a lot in profiling. I think it would be useful to have
      a perf command that runs bdiff over and over so we can find hot spots.
      c8fa7ad1ff90
    • Pulkit Goyal's avatar
      help: show help for disabled extensions (issue5228) · 5581b294f3c6
      Pulkit Goyal authored
      This patch does not exactly solve issue5228 but it results in a better
      condition on this issue. For disabled extensions, we used to parse the
      module and get the first occurrences of docstring and then return the first
      line of that as an introductory heading of extension. This is what we get
      today.
      
      This patch returns the whole docstring of the module as a help for extension,
      which is more informative. There are some modules which don't have much
      docstring at top level except the heading so those are unaffected by this
      change. To follow the existing trend of showing commands either we have to
      load the extension or have a very ugly parsing method which don't even assure
      correctness.
      5581b294f3c6
  11. Nov 05, 2016
    • Pulkit Goyal's avatar
      py3: make scmutil.rcpath() return bytes · af7c60988f6e
      Pulkit Goyal authored
      This patch make sure scmutil.rcpath() returns bytes independent of
      which platform is used on Python 3. If we want to change type for windows we
      can just conditionalize the return variable.
      af7c60988f6e
    • Pulkit Goyal's avatar
      py3: use pycompat.ossep at certain places · ba2c04059317
      Pulkit Goyal authored
      Certain instances of os.sep has been converted to pycompat.ossep where it was
      sure to use bytes only. There are more such instances which needs some more
      attention and will get surely.
      ba2c04059317
    • Pulkit Goyal's avatar
      py3: have pycompat.ospathsep and pycompat.ossep · ad40d307a9f0
      Pulkit Goyal authored
      We needed bytes version of os.sep and os.pathsep in py3 as they return
      unicodes.
      ad40d307a9f0
    • Pulkit Goyal's avatar
      py3: add a bytes version of os.name · 3874ddba1ab4
      Pulkit Goyal authored
      os.name returns unicodes on py3. Most of our checks are like
          os.name == 'nt'
      
      Because of the transformer, on the right hand side we have b'nt'. The
      condition will never satisfy even if os.name returns 'nt' as that will be an
      unicode.
      We either need to encode every occurence of os.name or have a
      new variable which is much cleaner. Now we have pycompat.osname.
      There are around 53 occurences of os.name in the codebase which needs to
      be replaced by pycompat.osname to support Python 3.
      3874ddba1ab4
  12. Nov 06, 2016
    • Pulkit Goyal's avatar
      py3: make util.datapath a bytes variable · 8321b083a83d
      Pulkit Goyal authored
      In this patch we make util.datapath a bytes variable, but we have to pass a
      unicode to gettext.translation otherwise it will cry. Used pycompat.fsdecode()
      to decode it back to unicode as it was converted to bytes using
      pycompat.fsencode().
      8321b083a83d
  13. Nov 05, 2016
  14. Nov 03, 2016
    • Durham Goode's avatar
      manifest: remove manifest.readshallowdelta · f65faa4422c8
      Durham Goode authored
      This removes manifest.readshallowdelta and converts its one consumer to use
      manifestlog instead.
      f65faa4422c8
    • Durham Goode's avatar
      manifest: get rid of manifest.readshallowfast · bce79dfcf5e4
      Durham Goode authored
      This removes manifest.readshallowfast and converts it's one user to use
      manifestlog instead.
      bce79dfcf5e4
    • Durham Goode's avatar
      manifest: add shallow option to treemanifestctx.readdelta and readfast · 78f3c7166f0d
      Durham Goode authored
      The old manifest had different functions for performing shallow reads, shallow
      readdeltas, and shallow readfasts. Since a lot of the code is duplicate (and
      since those functions don't make sense on a normal manifestctx), let's unify
      them into flags on the existing readdelta and readfast functions.
      
      A future diff will change consumers of these functions to use the manifestctx
      versions and will delete the old apis.
      78f3c7166f0d
    • Durham Goode's avatar
      manifest: change manifestlog mancache to be directory based · d4b340bf68c5
      Durham Goode authored
      In the last patch we added a get() function that allows fetching directory level
      treemanifestctxs. It didn't handle caching at directory level though, so we need to
      change our mancache to support multiple directories.
      d4b340bf68c5
    • Durham Goode's avatar
      manifest: add manifestlog.get to obtain subdirectory instances · dc21ea3323c4
      Durham Goode authored
      Previously manifestlog only allowed obtaining root level manifests. Future
      patches will need direct access to subdirectory manifests as part of changegroup
      creation, so let's add a get() function that knows how to deal with
      subdirectories.
      dc21ea3323c4
    • Durham Goode's avatar
      manifest: throw LookupError if node not in revlog · 1a0c1ad57833
      Durham Goode authored
      When accessing a manifest via manifestlog[node], let's verify that the node
      actually exists and throw a LookupError if it doesn't. This matches the old read
      behavior, so we don't accidentally return invalid manifestctxs.
      
      We do this in manifestlog instead of in the manifestctx/treemanifestctx
      constructors because the treemanifest code currently relies on the fact that
      certain code paths can produce treemanifests without touching the revlogs (and
      it has tests that verify things work if certain revlogs are missing entirely, so
      they break if we add validation that tries to read them).
      1a0c1ad57833
  15. Oct 23, 2016
    • Gregory Szorc's avatar
      revlog: optimize _chunkraw when startrev==endrev · 1f92056c4066
      Gregory Szorc authored
      In many cases, _chunkraw() is called with startrev==endrev. When
      this is true, we can avoid an extra index lookup and some other
      minor operations.
      
      On the mozilla-unified repo, `hg perfrevlogchunks -c` says this
      has the following impact:
      
      ! read w/ reused fd
      ! wall 0.371846 comb 0.370000 user 0.350000 sys 0.020000 (best of 27)
      ! wall 0.337930 comb 0.330000 user 0.300000 sys 0.030000 (best of 30)
      
      ! read batch w/ reused fd
      ! wall 0.014952 comb 0.020000 user 0.000000 sys 0.020000 (best of 197)
      ! wall 0.014866 comb 0.010000 user 0.000000 sys 0.010000 (best of 196)
      
      So, we've gone from ~25x slower than batch to ~22.5x slower.
      
      At this point, there's probably not much else we can do except
      implement an optimized function in the index itself, including in C.
      1f92056c4066
Loading