Skip to content
Snippets Groups Projects
  1. Dec 14, 2017
  2. Dec 13, 2017
  3. Dec 12, 2017
    • Wojciech Lis's avatar
      lfs: using workers in lfs prefetch · f98fac24
      Wojciech Lis authored
      This significantly speeds up lfs prefetch. With fast network we are
      seeing ~50% improvement of overall prefetch times
      Because of worker's API in posix we do lose finegrained progress update and only
      see progress when a file finished downloading.
      
      Test Plan:
      Run tests:
      ./run-tests.py -l test-lfs*
      ....
      # Ran 4 tests, 0 skipped, 0 failed.
      Run commands resulting in lfs prefetch e.g. hg sparse --enable-profile
      
      Differential Revision: https://phab.mercurial-scm.org/D1568
      f98fac24
  4. Dec 01, 2017
    • Wojciech Lis's avatar
      worker: make windows workers daemons · 86b8cc1f
      Wojciech Lis authored
      The windows workers weren't daemons and were not correctly killed when ctrl-c'd from the terminal. Withi this change when the main thread is killed, all daemons get killed as well.
      I also reduced the time we give to workers to cleanup nicely to not have people ctrl-c'ing when they get inpatient.
      
      The output when threads clened up nicely:
        PS C:\<dir>> hg.exe sparse --disable-profile SparseProfiles/<profile>.sparse
        interrupted!
      The output when threads don't clenup in 1 sec:
        PS C:\<dir> hg.exe sparse --enable-profile SparseProfiles/<profile>.sparse
        failed to kill worker threads while handling an exception
        interrupted!
        Exception in thread Thread-4 (most likely raised during interpreter shutdown):
        PS C:\<dir>>
      
      Test Plan:
      Run hg command on windows (pull/update/sparse). Ctrl-C'd sparse --enable-profile command that was using threads and observed in proces explorer that all threads got killed.
      ran tests on CentOS
      
      Differential Revision: https://phab.mercurial-scm.org/D1564
      86b8cc1f
  5. Dec 17, 2017
    • Wojciech Lis's avatar
      workers: add config to enable/diable workers · 471918fa
      Wojciech Lis authored
      This adds config to disable/enable workers with default being enabled.
      
      Test Plan:
      enabled profile without updaing .hg/hgrc (the default should be to use workers) and ran
      hg sprase --enable-profile <profile>.sparse
      Watched in the proces explorer that hg started 12 new threads for materializing files (this is my worker.numcpus) value
      
      Added
        [worker]
        enabled = False
      to the .hg/hgrc and re ran the command. This time hg didn't spawn any new threads for matreializing of files
      
      Differential Revision: https://phab.mercurial-scm.org/D1460
      471918fa
  6. Nov 20, 2017
    • Wojciech Lis's avatar
      workers: handling exceptions in windows workers · 71427ff1
      Wojciech Lis authored
      This adds handling of exceptions from worker threads and resurfaces them as if the function ran without workers.
      If any of the threads throws, the main thread kills all running threads giving them 5 sec to handle the interruption and raises the first exception received.
      
      We don't have to join threads if is_alive() is false
      
      Test Plan:
      Ran multiple updates/enable/disable sparse profile and things worked well
      
      Ran test on CentOS- all tests passing on @ passed here
      
      Added a forged exception into the worker code and got it properly resurfaced and the rest of workers killed: P58642088
      PS C:\open\<repo>> ..\facebook-hg-rpms\build\hg\hg.exe --config extensions.fsmonitor=! sparse --enable-profile <profile>
      updating [==>                                                                                                                ]  1300/39166 1m57sException in thread Thread-3:
      Traceback (most recent call last):
        File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\threading.py", line 801, in __bootstrap_inner
          self.run()
        File "C:\open\facebook-hg-rpms\build\hg\mercurial\worker.py", line 244, in run
          raise e
      Exception: Forged exception
      
      Exception in thread Thread-2:
      Traceback (most recent call last):
        File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\threading.py", line 801, in __bootstrap_inner
          self.run()
        File "C:\open\facebook-hg-rpms\build\hg\mercurial\worker.py", line 244, in run
          raise e
      Exception: Forged exception
      <...>
      Traceback (most recent call last):
        File "C:\open\facebook-hg-rpms\build\hg\hgexe.py", line 41, in <module>
          dispatch.run()
        File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 85, in run
          status = (dispatch(req) or 0) & 255
        File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 173, in dispatch
          ret = _runcatch(req)
        File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 324, in _runcatch
          return _callcatch(ui, _runcatchfunc)
        File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 332, in _callcatch
          return scmutil.callcatch(ui, func)
        File "C:\open\facebook-hg-rpms\build\hg\mercurial\scmutil.py", line 154, in callcatch
          return func()
        File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 314, in _runcatchfunc
          return _dispatch(req)
        File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 951, in _dispatch
          cmdpats, cmdoptions)
        File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\remotefilelog\__init__.py", line 415, in runcommand
          return orig(lui, repo, *args, **kwargs)
        File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\undo.py", line 118, in _runcommandwrapper
          result = orig(lui, repo, cmd, fullargs, *args)
        File "C:\open\facebook-hg-rpms\build\hg\hgext\journal.py", line 84, in runcommand
          return orig(lui, repo, cmd, fullargs, *args)
        File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\perftweaks.py", line 268, in _tracksparseprofiles
          res = runcommand(lui, repo, *args)
        File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\perftweaks.py", line 256, in _trackdirstatesizes
          res = runcommand(lui, repo, *args)
        File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\copytrace.py", line 144, in _runcommand
          return orig(lui, repo, cmd, fullargs, ui, *args, **kwargs)
        File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbamend\hiddenoverride.py", line 119, in runcommand
          result = orig(lui, repo, cmd, fullargs, *args)
        File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 712, in runcommand
          ret = _runcommand(ui, options, cmd, d)
        File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 959, in _runcommand
          return cmdfunc()
        File "C:\open\facebook-hg-rpms\build\hg\mercurial\dispatch.py", line 948, in <lambda>
          d = lambda: util.checksignature(func)(ui, *args, **strcmdopt)
        File "C:\open\facebook-hg-rpms\build\hg\mercurial\util.py", line 1183, in check
          return func(*args, **kwargs)
        File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbsparse.py", line 860, in sparse
          disableprofile=disableprofile, force=force)
        File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbsparse.py", line 949, in _config
          len, _refresh(ui, repo, oldstatus, oldsparsematch, force))
        File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\hgext3rd\fbsparse.py", line 1116, in _refresh
          mergemod.applyupdates(repo, typeactions, repo[None], repo['.'], False)
        File "C:\open\facebook-hg-rpms\build\hg\hg-python\lib\site-packages\remotefilelog\__init__.py", line 311, in applyupdates
          return orig(repo, actions, wctx, mctx, overwrite, labels=labels)
        File "C:\open\facebook-hg-rpms\build\hg\mercurial\merge.py", line 1464, in applyupdates
          for i, item in prog:
        File "C:\open\facebook-hg-rpms\build\hg\mercurial\worker.py", line 286, in _windowsworker
          raise t.exception
      Exception: Forged exception
      PS C:\open\ovrsource>
      
      Differential Revision: https://phab.mercurial-scm.org/D1459
      71427ff1
    • Wojciech Lis's avatar
      workers: implemented worker on windows · 02b36e86
      Wojciech Lis authored
      This change implements thread based worker on windows.
      The handling of exception from within threads will happen in separate diff.
      
      The worker is for now used in mercurial/merge.py and in lfs extension
      
      After multiple tests and milions of files materiealized, thousands lfs fetched
      it seems that neither merge.py nor lfs/blobstore.py is thread unsafe. I also
      looked through the code and besides the backgroundfilecloser (handled in base
      of this) things look good.
      
      The performance boost of this on windows is
      
      ~50% for sparse --enable-profile
      * Speedup of hg up/rebase - not exactly measured
      
      Test Plan:
      Ran 10s of hg sparse --enable-profile and --disable-profile operations on large profiles and verified that workers are running. Used sysinternals suite to see that all threads are spawned and run as they should
      
      Run various other operations on the repo including update and rebase
      
      Ran tests on CentOS and all tests that pass on @ pass here
      
      Differential Revision: https://phab.mercurial-scm.org/D1458
      02b36e86
  7. Dec 12, 2017
  8. Dec 15, 2017
  9. Dec 14, 2017
  10. Dec 11, 2017
  11. Dec 13, 2017
  12. Dec 15, 2017
    • Martin von Zweigbergk's avatar
      tests: avoid echo with backslash escapes · 417fa230
      Martin von Zweigbergk authored
      Some shells' echo (e.g. Bash's) do not interpret backslash escapes by
      default, while others shells' do (e.g. Dash's). This led to a failure
      in test-run-tests.t. We could switch to the more compatible printf,
      but using a heredoc seems simpler, so that's what this patch does.
      
      For anyone who wants to test this for themselves, remove the "unset
      HGTEST_SHELL" line in helper-runtests.sh and pass
      e.g. "--shell=/bin/dash" to the test runner.
      
      Differential Revision: https://phab.mercurial-scm.org/D1699
      417fa230
    • Anton Shestakov's avatar
      hgweb: stop using HTML comments in <script> · 56854848
      Anton Shestakov authored
      Once upon a time, in 1995, there were browsers that didn't understand <script>
      tags and they would simply show the code inside as text. This started a
      tradition of wrapping everything inside <script> in <!-- HTML comments -->.
      Nowadays, it's not only not needed, but can be considered harmful[1]:
      
      - within XHTML documents, the source will actually be hidden from all browsers
        and rendered useless
      
      - `--` is not allowed within HTML comments, so any decrement operations in
        script are invalid
      
      [1]: http://www.javascripttoolbox.com/bestpractices/#comments
      56854848
  13. Dec 14, 2017
  14. Oct 01, 2017
  15. Dec 14, 2017
  16. Dec 13, 2017
    • Phil Cohen's avatar
      rebase: fix for hgsubversion · 83014fa9
      Phil Cohen authored
      5c25fe7fb1e6 broke something in the hgsubversion test path, causing it raise an
      abort (Abort: nothing to merge) during a perfectly good rebase. I tracked it
      down to this change. It's probably not hgsubversion related.
      
      I suspect that using the same `wctx` from before the initial update causes
      problems with the wctx's cached manifest property. I noticed we also sometimes
      stick random gunk on the wctx object in other places (like in `copies.py`) so
      it's probably best to reset it for now.
      
      The line I added before was actually useless since we don't pass wctx to the
      initial `merge.update`, so it defaults to `repo[None]`. So I just removed it.
      
      Differential Revision: https://phab.mercurial-scm.org/D1679
      83014fa9
  17. Dec 11, 2017
    • Anton Shestakov's avatar
      hgweb: render next pages on /graph incrementally · f84b0125
      Anton Shestakov authored
      Previously, when user scrolled down to see the next page on /graph, all hgweb
      did was re-render everything that would be visible (by simply incrementing
      revcount). It was not efficient at all, and this patch makes /graph page behave
      similarly to the regular /log: every new page only consists of new changesets,
      no duplication, and only jsdata is based on the full set of changesets required
      to build accurate graph.
      
      This is achieved by adding "?graphtop=<node>" to the next page URL template,
      effectively remembering where the graph started, and using that value to create
      the new `tree` that covers the whole visible graph. That variable is then used
      to produce jsdata for redrawing graph client-side.
      
      nextentry is used for the same purpose as on /log page (to format the next page
      URL), but it's not a part of the graph.
      f84b0125
    • Anton Shestakov's avatar
      hgweb: split graphdata() into jsdata() and nodes() · a48af499
      Anton Shestakov authored
      nodes keyword passed to the template can be any iterator, but jsdata needs to
      be a list because it gets JSONified.
      a48af499
  18. Dec 10, 2017
    • Anton Shestakov's avatar
      hgweb: calculate <canvas> width and height client-side · 27ab3150
      Anton Shestakov authored
      hgweb determines and passes to templates some variables related to graph
      appearance, like bg_height, canvaswidth and canvasheight. bg_height was and
      still is used for graph.scale() call in graph.tmpl, and the two latter
      variables were used in <canvas> element as width and height properties, and
      they were set before JS code got to run. Setting these properties server-side
      doesn't make a lot of sense, because a graph that has been scaled should
      calculate things like width and height on its own when being rendered.
      
      Let's move (re)sizing <canvas> to JavaScript (to Graph.render function) and
      stop parsing HTML with regular expressions just to know new width and height.
      That extra loop that only counts cols is required because <canvas> can't
      be resized after or in the process of rendering (or it gets cleared).
      Incidentally, SVG doesn't have this problem and I'm hoping to switch graph to
      using it in future.
      
      There also was truecanvasheight, but according to hg grep --all it was never
      used, see d490edc71146.
      27ab3150
  19. Dec 08, 2017
    • Anton Shestakov's avatar
      hgweb: filter graphmod.colored() output before iterating over it · 76dcdc4e
      Anton Shestakov authored
      Consumers in this function use output of graphmod.colored(), but only want
      items with type == CHANGESET, so let's filter it early.
      
      This is primarily just a refactoring, but it also fixes a potential small bug
      with `rows = len(tree)` (this variable is used for "Rows shown" line in
      raw-graph) if there are items of other types.
      76dcdc4e
  20. Dec 11, 2017
  21. Dec 10, 2017
  22. Dec 11, 2017
    • Martin von Zweigbergk's avatar
      memfilectx: make changectx argument mandatory in constructor (API) · 8a0cac20
      Martin von Zweigbergk authored
      committablefilectx has three subclasses: workingfilectx, memfilectx,
      and overlayfilectx. committablefilectx takes an optional (change) ctx
      instance to its constructor. If it's provided, it's set on the
      instance as self._changectx. If not, that property is supposed to be
      defined by the class. However, only workingfilectx does that. The
      other two will have the property undefined if it's not passed in the
      constructor. That seems bad to me. This patch makes the changectx
      argument to the memfilectx constructor mandatory because that fixes
      the failure I ran into. It seems like we should also fix the
      overlayfilectx case.
      
      Differential Revision: https://phab.mercurial-scm.org/D1658
      8a0cac20
  23. Dec 09, 2017
  24. Dec 13, 2017
  25. Nov 12, 2017
  26. Dec 11, 2017
    • Matt Harbison's avatar
      tests: remove (glob) annotations that were only for '\' matches · 4441705b
      Matt Harbison authored
      # skip-blame because this was mechanically rewritten the following script.  I
      ran it on both *.t and *.py, but none of the *.py changes were proper.  All *.t
      ones appear to be, and they run without addition failures on both Windows and
      Linux.
      
        import argparse
        import os
        import re
      
        ap = argparse.ArgumentParser()
        ap.add_argument('path', nargs='+')
        opts = ap.parse_args()
      
        globre = re.compile(r'^(.*) \(glob\)(.*)$')
      
        for p in opts.path:
            tmp = p + '.tmp'
            with open(p, 'rb') as src, open(tmp, 'wb') as dst:
                for line in src:
                    m = globre.match(line)
                    if not m or '$LOCALIP' in line or '*' in line:
                        dst.write(line)
                        continue
                    if '?' in line[:-3] or ('?' in line[:-3] and line[-3:] != '(?)'):
                        dst.write(line)
                        continue
                    dst.write(m.group(1) + m.group(2) + '\n')
            os.unlink(p)
            os.rename(tmp, p)
      4441705b
Loading