Skip to content
Snippets Groups Projects
  1. Feb 21, 2022
  2. Mar 04, 2022
    • Gregory Szorc's avatar
      global: bulk replace simple pycompat.iteritems(x) with x.items() · f254fc73d956
      Gregory Szorc authored
      pycompat.iteritems() just calls .items().
      
      This commit applies a regular expression search and replace to convert
      simple instances of pycompat.iteritems() with .items(). There are still
      a handful of calls to pycompat.iteritems() remaining. But these all have
      more complicated expressions that I wasn't comfortable performing an
      automated replace on. In addition, some simple replacements were withheld
      because they broke pytype. These will be handled by their own changesets.
      
      Differential Revision: https://phab.mercurial-scm.org/D12318
      f254fc73d956
  3. Mar 03, 2022
  4. Jan 07, 2022
  5. Sep 28, 2021
    • Martin von Zweigbergk's avatar
      errors: raise InputError on bad revset to revrange() iff provided by the user · 5ced12cfa41b
      Martin von Zweigbergk authored
      Most callers of `scmutil.revrange()` pass in a revset provided by the
      user. If there are problems resolving that, it should result in an
      `InputError` and exit code 10 (when using detailed exit
      codes). However, there are also some callers that pass in revsets not
      provided by the user. `InputError` is not appropriate in those
      cases. This patch therefore introduces a wrapper around
      `scmutil.revrange()` that simply converts the exception type. I put it
      in `logcmdutil.py` since that seems to be the lowest-level module in
      the (poorly defined) UI layer.
      
      Differential Revision: https://phab.mercurial-scm.org/D11560
      5ced12cfa41b
  6. Mar 28, 2021
  7. Apr 12, 2021
  8. Jan 18, 2021
    • Raphaël Gomès's avatar
      sidedata: move to new sidedata storage in revlogv2 · 3d740058b467
      Raphaël Gomès authored
      The current (experimental) sidedata system uses flagprocessors to signify the
      presence and store/retrieve sidedata from the raw revlog data. This proved to be
      quite fragile from an exchange perspective and a lot more complex than simply
      having a dedicated space in the new revlog format.
      
      This change does not handle exchange (ironically), so the test for amend - that
      uses a bundle - is broken. This functionality is split into the next patches.
      
      Differential Revision: https://phab.mercurial-scm.org/D9993
      3d740058b467
  9. Dec 31, 2020
  10. Dec 01, 2020
  11. Nov 27, 2020
    • durin42's avatar
      formating: upgrade to black 20.8b1 · 89a2afe31e82
      durin42 authored
      This required a couple of small tweaks to un-confuse black, but now it
      works. Big formatting changes come from:
      
       * Dramatically improved collection-splitting logic upstream
       * Black having a strong (correct IMO) opinion that """ is better than '''
      
      Differential Revision: https://phab.mercurial-scm.org/D9430
      89a2afe31e82
  12. Oct 15, 2020
    • Pierre-Yves David's avatar
      clonebundles: move a bundle of clone bundle related code to a new module · 74271829ddc0
      Pierre-Yves David authored
      In the process on general clone bundle automatically, we need to make some
      function available more widely. This is a good opportunity to extract a
      significant amount of code from `mercurial.exchange` into a new
      `mercurial.bundlecaches`. This make `mercurial.exchange` move under the 3K line
      range (hooray…).
      
      The module is called `bundlecaches` because I expect it to be eventually useful
      for more than just clone bundle (like pull bunbles).
      
      Differential Revision: https://phab.mercurial-scm.org/D9208
      74271829ddc0
  13. Apr 14, 2020
    • Pulkit Goyal's avatar
      scmutil: add writereporequirements() and route requires writing through it · a03c177a4679
      Pulkit Goyal authored
      In upcoming patches, to implement Share Safe plan we will be introducing
      requires file in store. We need to route all callers to a single function
      to check for a share-safe requirement and if present, write requirements to
      .hg/store/requires instead.
      
      After this patch, callers directly calling scmutil.writerequires() are only
      those where we don't have the repo object, for example when initializing
      the repository object itself.
      
      Differential Revision: https://phab.mercurial-scm.org/D8631
      a03c177a4679
  14. Jul 10, 2020
  15. Feb 09, 2020
    • Matt Harbison's avatar
      lfs: use str for the open() mode when opening a blob for py3 · 234001d22ba6
      Matt Harbison authored
      The other fix for this was to leave the mode as bytes, and import
      `pycompat.open()` like a bunch of other modules do.  But I think it's confusing
      to still use bytes at the python boundary, and obviously error prone.  Grepping
      for ` open\(.+, ['"][a-z]+['"]\)` and ` open\(.+, b['"][a-z]+['"]\)` outside of
      `tests`, there are 51 and 87 uses respectively, so it's not like this is a rare
      direct usage.
      
      Differential Revision: https://phab.mercurial-scm.org/D8099
      234001d22ba6
  16. Jan 21, 2020
    • Matt Harbison's avatar
      lfs: enable workers by default · 87167caa2989
      Matt Harbison authored
      With the stall issue seemingly fixed, there's no reason not to use workers.  The
      setting is left for now to keep the test output deterministic, and in case other
      issues come up.  If none do, this can be converted to a developer setting for
      usage with the tests.
      
      Differential Revision: https://phab.mercurial-scm.org/D7963
      87167caa2989
    • Matt Harbison's avatar
      lfs: fix the stall and corruption issue when concurrently uploading blobs · 43eea17ae7b3
      Matt Harbison authored
      We've avoided the issue up to this point by gating worker usage with an
      experimental config.  See 10e62d5efa73, and the thread linked there for some of
      the initial diagnosis, but essentially some data was being read from the blob
      before an error occurred and `keepalive` retried, but didn't rewind the file
      pointer.  So the leading data was lost from the blob on the server, and the
      connection stalled, trying to send more data than available.
      
      In trying to recreate this, I was unable to do so uploading from Windows to
      CentOS 7.  But it reproduced every time going from CentOS 7 to another CentOS 7
      over https.
      
      I found recent fixes in the FaceBook repo to address this[1][2].  The commit
      message for the first is:
      
          The KeepAlive HTTP implementation is bugged in it's retry logic, it supports
          reading from a file pointer, but doesn't support rewinding of the seek cursor
          when it performs a retry. So it can happen that an upload fails for whatever
          reason and will then 'hang' on the retry event.
      
          The sequence of events that get triggered are:
           - Upload file A, goes OK. Keep-Alive caches connection.
           - Upload file B, fails due to (for example) failing Keep-Alive, but LFS file
             pointer has been consumed for the upload and fd has been closed.
           - Retry for file B starts, sets the Content-Length properly to the expected
             file size, but since file pointer has been consumed no data will be uploaded,
             causing the server to wait for the uploaded data until either client or
             server reaches a timeout, making it seem as our mercurial process hangs.
      
          This is just a stop-gap measure to prevent this behavior from blocking Mercurial
          (LFS has retry logic). A proper solutions need to be build on top of this
          stop-gap measure: for upload from file pointers, we should support fseek() on
          the interface. Since we expect to consume the whole file always anyways, this
          should be safe. This way we can seek back to the beginning on a retry.
      
      I ported those two patches, and it works.  But I see that `url._sendfile()` does
      a rewind on `httpsendfile` objects[3], so maybe it's better to keep this all in
      one place and avoid a second seek.  We may still want the first FaceBook patch
      as extra protection for this problem in general.  The other two uses of
      `httpsendfile` are in the wire protocol to upload bundles, and to upload
      largefiles.  Neither of these appear to use a worker, and I'm not sure why
      workers seem to trigger this, or if this could have happened without a worker.
      
      Since `httpsendfile` already has a `close()` method, that is dropped.  That
      class also explicitly says there's no `__len__` attribute, so that is removed
      too.  The override for `read()` is necessary to avoid the progressbar usage per
      file.
      
      [1] https://github.com/facebookexperimental/eden/commit/c350d6536d90c044c837abdd3675185644481469
      [2] https://github.com/facebookexperimental/eden/commit/77f0d3fd0415e81b63e317e457af9c55c46103ee
      [3] https://www.mercurial-scm.org/repo/hg/file/5.2.2/mercurial/url.py#l176
      
      Differential Revision: https://phab.mercurial-scm.org/D7962
      43eea17ae7b3
    • Matt Harbison's avatar
      lfs: add a method to the local blobstore to convert OIDs to file paths · 06de4a673f48
      Matt Harbison authored
      This is less ugly than passing an open callback to the `httpsendfile`
      constuctor.
      
      Differential Revision: https://phab.mercurial-scm.org/D7961
      06de4a673f48
    • Matt Harbison's avatar
      lfs: move the initialization of the upload request into the try block · 46c8f15fb2b4
      Matt Harbison authored
      This (almost) guarantees that the file is closed in the case of an exception.
      The one hole is if the `seek(SEEK_END)`/`tell()`/`seek(0)` sequence fails.  But
      that's going to go away when subclassing `httpconnection.httpsendfile` to fix
      the worker problem, so I'm not going to worry too much.  (And that class appears
      to have the same problem.)
      
      Differential Revision: https://phab.mercurial-scm.org/D7959
      46c8f15fb2b4
    • Matt Harbison's avatar
      lfs: drop an unnecessary r'' prefix · b2408acaa4c9
      Matt Harbison authored
      No longer necessary since the source transformer was removed.
      
      # skip-blame for changing string prefixes
      
      Differential Revision: https://phab.mercurial-scm.org/D7958
      b2408acaa4c9
    • Matt Harbison's avatar
      lfs: explicitly close the file handle for the blob being uploaded · 2ad4e8aefcf4
      Matt Harbison authored
      The previous code relied on reading the blob fully to close it.  The obvious
      problem is if an error occurs before that point.  But there is also a problem
      when using workers where the data may need to be re-read, which can't happen
      once it is closed.  This eliminates the surprising behavior before attempting to
      fix the worker problem.
      
      Differential Revision: https://phab.mercurial-scm.org/D7957
      2ad4e8aefcf4
    • Matt Harbison's avatar
      lfs: drop the unused progressbar code in the `filewithprogress` class · 5f841daf3b41
      Matt Harbison authored
      This has been unused since f98fac24b757, which added worker based transfers for
      concurrency, shifting the progressbar maintenance to the single thread waiting
      on the worker to complete.  Since the name no longer fits, rename the class.
      
      Differential Revision: https://phab.mercurial-scm.org/D7956
      5f841daf3b41
  17. Jan 15, 2020
  18. Jan 14, 2020
  19. Dec 23, 2019
    • Matt Harbison's avatar
      verify: allow the storage to signal when renames can be tested on `skipread` · b9e174d4ed11
      Matt Harbison authored
      This applies the new marker in the lfs handler to show it in action, and adds
      the test mentioned at the beginning of the series to show that fulltext isn't
      necessary in the LFS case.
      
      The existing `skipread` isn't enough, because it is also set if an error occurs
      reading the revlog data, or the data is censored.  It could probably be cleared,
      but then it technically violates the interface contract.  That wouldn't matter
      for the existing verify algorithm, but it isn't clear how that will change as
      alternate storage support is added.
      
      The flag is probably pretty revlog specific, given the comments in verify.py.
      But there's already filelog specific stuff in there and I'm not sure what future
      storage will bring, so I don't want to over-engineer this.  Likewise, I'm not
      sure that we want the verify method for each storage type to completely drive
      the bus when it comes to detecting renames, so I don't want to go down the
      rabbithole of having verifyintegrity() return metadata hints at this point.
      
      Differential Revision: https://phab.mercurial-scm.org/D7713
      b9e174d4ed11
    • Matt Harbison's avatar
      lfs: don't skip locally available blobs when verifying · 1a6dd50cd0db
      Matt Harbison authored
      The `skipflags` config was introduced in a2ab9ebcd85b, which specifically calls
      out downloading and storing all blobs as potentially too expensive.  But I don't
      see any reason to skip blobs that are already available locally.  Hashing the
      blob is the only way to indirectly verify the rawdata content stored in the
      revlog.
      
      (The note in that commit about skipping renamed is still correct, but the reason
      given about needing fulltext isn't.)
      
      Differential Revision: https://phab.mercurial-scm.org/D7712
      1a6dd50cd0db
  20. Dec 20, 2019
  21. Dec 23, 2019
  22. Nov 08, 2019
    • Augie Fackler's avatar
      cleanup: remove pointless r-prefixes on single-quoted strings · 9f70512ae2cf
      Augie Fackler authored
      This is the promised second step on single-quoted strings. These had
      existed because our source transformer didn't turn r'' into b'', so we
      had tagged some strings as r-strings to get "native" strings on both
      Pythons. Now that the transformer is gone, we can dispense with this
      nonsense.
      
      Methodology:
      
      I ran
      
          hg locate 'set:added() or modified() or clean()' | egrep '.*\.py$'  | xargs egrep --color=never -n  -- \[\^b\]\[\^a-z\]r\'\[\^\'\\\\\]\*\'\[\^\'\
      
      in an emacs grep-mode buffer, and then used a keyboard macro to
      iterate over the results and remove the r prefix as needed.
      
      # skip-blame removing unneeded r prefixes left over from Python 3 migration.
      
      Differential Revision: https://phab.mercurial-scm.org/D7306
      9f70512ae2cf
  23. Nov 07, 2019
    • Augie Fackler's avatar
      cleanup: remove pointless r-prefixes on double-quoted strings · 313e3a279828
      Augie Fackler authored
      This is only double-quoted strings. I'll do single-quoted strings as a
      second step. These had existed because our source transformer didn't
      turn r"" into b"", so we had tagged some strings as r-strings to get
      "native" strings on both Pythons. Now that the transformer is gone, we
      can dispense with this nonsense.
      
      Methodology:
      
      I ran
      
          hg locate 'set:added() or modified() or clean()' | egrep '.*\.py$'  | xargs egrep --color=never -n  -- \[\^a-z\]r\"\[\^\"\\\\\]\*\"\[\^\"\]
      
      in an emacs grep-mode buffer, and then used a keyboard macro to
      iterate over the results and remove the r prefix as needed.
      
      # skip-blame removing unneeded r prefixes left over from Python 3 migration.
      
      Differential Revision: https://phab.mercurial-scm.org/D7305
      313e3a279828
  24. Nov 02, 2019
    • Gregory Szorc's avatar
      py3: define and use json.loads polyfill · 579672b347d2
      Gregory Szorc authored
      Python 3.5's json.loads() requires a str. Only Python 3.6+
      supports passing a bytes or bytearray.
      
      This commit implements a json.loads() polyfill on Python 3.5
      so that we can use bytes. The added function to detect encodings
      comes verbatim from Python 3.7.
      579672b347d2
  25. Oct 21, 2019
    • Denis Laxalde's avatar
      lfs: dedent documentation section about .hglfs file · 26caf96a5fa9
      Denis Laxalde authored
      The leading ".hglfs::" is interpreted as a macro in generated man page
      and, as it is unknown, the whole section does not render. Also, having
      the section marked as preformatted (::) makes it render verbatim in
      HTML, which is not desired as the text contains formatting markers. So
      we just dedent the section and remove the ".hglfs::" line. The example
      file is still indented and rendered preformatted.
      26caf96a5fa9
  26. Oct 07, 2019
  27. Oct 06, 2019
    • Gregory Szorc's avatar
      py3: define and use pycompat.iteritems() for hgext/ · 649d3ac37a12
      Gregory Szorc authored
      .iteritems() -> .items() is the last source transform being performed.
      But it is also the most widely used.
      
      This commit adds a pycompat.iteritems symbol and imports it in place
      of .iteritems() for usage in hgext/. I chose to stop at just hgext/
      because the patch will be large and it is an easy boundary to stop at
      since we can disable source transformation on a per-package basis.
      
      There are places where the type does implement items() and we could
      call items() directly. However, this would require critical thought
      and I thought it would be easier to just blindly change the code. We
      know which call sites need to be audited in the future because they
      have "pycompat.iteritems."
      
      With this change, we no longer perform source transformation on
      hgext!
      
      Differential Revision: https://phab.mercurial-scm.org/D7014
      649d3ac37a12
Loading