- Jan 21, 2020
-
-
Matt Harbison authored
No longer necessary since the source transformer was removed. # skip-blame for changing string prefixes Differential Revision: https://phab.mercurial-scm.org/D7958
-
Matt Harbison authored
The previous code relied on reading the blob fully to close it. The obvious problem is if an error occurs before that point. But there is also a problem when using workers where the data may need to be re-read, which can't happen once it is closed. This eliminates the surprising behavior before attempting to fix the worker problem. Differential Revision: https://phab.mercurial-scm.org/D7957
-
Matt Harbison authored
This has been unused since f98fac24b757, which added worker based transfers for concurrency, shifting the progressbar maintenance to the single thread waiting on the worker to complete. Since the name no longer fits, rename the class. Differential Revision: https://phab.mercurial-scm.org/D7956
-
- Jan 15, 2020
-
-
Matt Harbison authored
This is also adapted from the Facebook repo[1]. Unlike there, we were already reading the download stream in chunks and immediately writing it to disk, so we basically avoided the problem on download. There shouldn't be a lot of data to read on upload, but it's better to get rid of this pattern. [1] https://github.com/facebookexperimental/eden/commit/82df66ffe97e21f3ee73dfec093c87500fc1f6a7 Differential Revision: https://phab.mercurial-scm.org/D7882
-
Matt Harbison authored
Adapted from the Facebook repo[1]. The intent is to distinguish between the connection dying and getting served a corrupt blob. The original message: HTTP makes no provision to tell your client that you failed halfway through producing your response and won't have the answer they're looking for. So, if a LFS server fails while producing a response, then we'll report an OID mismatch. We can do a little better and disambiguate between "the server sent us the wrong blob" (very scary) and "the server crashed" (merely annoying) by looking at the content length of the response we got back. If it's not what was advertised, we can reasonably safely assume the server crashed. [1] https://github.com/facebookexperimental/eden/commit/2a4a6fab4e882ed89b948bfc1e7d56d7c3c99dd2 Differential Revision: https://phab.mercurial-scm.org/D7881
-
- Jan 14, 2020
-
-
Matt Harbison authored
This is the response object, not a request. Differential Revision: https://phab.mercurial-scm.org/D7880
-
Matt Harbison authored
Facebook also passes `util.bytecount()` as a pretty formatter here, but our progress bar doesn't support that. Differential Revision: https://phab.mercurial-scm.org/D7872
-
- Dec 23, 2019
-
-
Matt Harbison authored
This applies the new marker in the lfs handler to show it in action, and adds the test mentioned at the beginning of the series to show that fulltext isn't necessary in the LFS case. The existing `skipread` isn't enough, because it is also set if an error occurs reading the revlog data, or the data is censored. It could probably be cleared, but then it technically violates the interface contract. That wouldn't matter for the existing verify algorithm, but it isn't clear how that will change as alternate storage support is added. The flag is probably pretty revlog specific, given the comments in verify.py. But there's already filelog specific stuff in there and I'm not sure what future storage will bring, so I don't want to over-engineer this. Likewise, I'm not sure that we want the verify method for each storage type to completely drive the bus when it comes to detecting renames, so I don't want to go down the rabbithole of having verifyintegrity() return metadata hints at this point. Differential Revision: https://phab.mercurial-scm.org/D7713
-
Matt Harbison authored
The `skipflags` config was introduced in a2ab9ebcd85b, which specifically calls out downloading and storing all blobs as potentially too expensive. But I don't see any reason to skip blobs that are already available locally. Hashing the blob is the only way to indirectly verify the rawdata content stored in the revlog. (The note in that commit about skipping renamed is still correct, but the reason given about needing fulltext isn't.) Differential Revision: https://phab.mercurial-scm.org/D7712
-
- Dec 20, 2019
-
-
Matt Harbison authored
Trying to validate the fulltext of an external revision causes missing blobs to be downloaded and cached. Since the downloads aren't batch prefetched[1] and aren't compressed, this can be expensive both in terms of time and space. I made this a tri-state instead of a simple bool because there's an existing (undocumented) config to handle this, and it would be weird if `hg verify` were to suddenly start ignoring that config but an `hg recover` initiated verify honors it. Since this uses the same config setting, it too will skip rename verification (which requires fulltext, but not for LFS). [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2018-April/116118.html Differential Revision: https://phab.mercurial-scm.org/D7708
-
- Dec 23, 2019
-
-
Matt Harbison authored
This conceptually broke in 1541e1a8e87d when the filelog isa revlog relationship was changed to containment of the revlog. It was made more obvious in 62a532045e71 and related API simplification. It's resolved in favor of passing a revlog because the revlog verification code doesn't have a reference to a filelog. Differential Revision: https://phab.mercurial-scm.org/D7711
-
- Nov 08, 2019
-
-
Augie Fackler authored
This is the promised second step on single-quoted strings. These had existed because our source transformer didn't turn r'' into b'', so we had tagged some strings as r-strings to get "native" strings on both Pythons. Now that the transformer is gone, we can dispense with this nonsense. Methodology: I ran hg locate 'set:added() or modified() or clean()' | egrep '.*\.py$' | xargs egrep --color=never -n -- \[\^b\]\[\^a-z\]r\'\[\^\'\\\\\]\*\'\[\^\'\ in an emacs grep-mode buffer, and then used a keyboard macro to iterate over the results and remove the r prefix as needed. # skip-blame removing unneeded r prefixes left over from Python 3 migration. Differential Revision: https://phab.mercurial-scm.org/D7306
-
- Nov 07, 2019
-
-
Augie Fackler authored
This is only double-quoted strings. I'll do single-quoted strings as a second step. These had existed because our source transformer didn't turn r"" into b"", so we had tagged some strings as r-strings to get "native" strings on both Pythons. Now that the transformer is gone, we can dispense with this nonsense. Methodology: I ran hg locate 'set:added() or modified() or clean()' | egrep '.*\.py$' | xargs egrep --color=never -n -- \[\^a-z\]r\"\[\^\"\\\\\]\*\"\[\^\"\] in an emacs grep-mode buffer, and then used a keyboard macro to iterate over the results and remove the r prefix as needed. # skip-blame removing unneeded r prefixes left over from Python 3 migration. Differential Revision: https://phab.mercurial-scm.org/D7305
-
- Nov 02, 2019
-
-
Gregory Szorc authored
Python 3.5's json.loads() requires a str. Only Python 3.6+ supports passing a bytes or bytearray. This commit implements a json.loads() polyfill on Python 3.5 so that we can use bytes. The added function to detect encodings comes verbatim from Python 3.7.
-
- Oct 21, 2019
-
-
Denis Laxalde authored
The leading ".hglfs::" is interpreted as a macro in generated man page and, as it is unknown, the whole section does not render. Also, having the section marked as preformatted (::) makes it render verbatim in HTML, which is not desired as the text contains formatting markers. So we just dedent the section and remove the ".hglfs::" line. The example file is still indented and rendered preformatted.
-
- Oct 07, 2019
-
-
Martin von Zweigbergk authored
Differential Revision: https://phab.mercurial-scm.org/D7029
-
- Oct 06, 2019
-
-
Gregory Szorc authored
.iteritems() -> .items() is the last source transform being performed. But it is also the most widely used. This commit adds a pycompat.iteritems symbol and imports it in place of .iteritems() for usage in hgext/. I chose to stop at just hgext/ because the patch will be large and it is an easy boundary to stop at since we can disable source transformation on a per-package basis. There are places where the type does implement items() and we could call items() directly. However, this would require critical thought and I thought it would be easier to just blindly change the code. We know which call sites need to be audited in the future because they have "pycompat.iteritems." With this change, we no longer perform source transformation on hgext! Differential Revision: https://phab.mercurial-scm.org/D7014
-
Gregory Szorc authored
Now that we don't byte transform string literals, we no longer need this transform. While we're here, we also drop some superfluous u'' prefix in existing callers. Differential Revision: https://phab.mercurial-scm.org/D7011
-
Gregory Szorc authored
The march continues. Differential Revision: https://phab.mercurial-scm.org/D7009
-
Gregory Szorc authored
Continuing to eliminate the implicit import of symbols in the Python 3 source transformer so we can eliminate it. Differential Revision: https://phab.mercurial-scm.org/D7007
-
Augie Fackler authored
Done with python3.7 contrib/byteify-strings.py -i $(hg files 'set:mercurial/**.py - mercurial/thirdparty/** + hgext/**.py - hgext/fsmonitor/pywatchman/** - mercurial/__init__.py') black -l 80 -t py33 -S $(hg files 'set:**.py - mercurial/thirdparty/** - "contrib/python-zstandard/**" - hgext/fsmonitor/pywatchman/**') # skip-blame mass-reformatting only Differential Revision: https://phab.mercurial-scm.org/D6972
-
Augie Fackler authored
This is using my patch to black (https://github.com/psf/black/pull/826) so we don't un-wrap collection literals. Done with: hg files 'set:**.py - mercurial/thirdparty/** - "contrib/python-zstandard/**"' | xargs black -S # skip-blame mass-reformatting only # no-check-commit reformats foo_bar functions Differential Revision: https://phab.mercurial-scm.org/D6971
-
- Sep 03, 2019
-
-
Pierre-Yves David authored
If we want some flag processors to be able to store sidedata it needs to be actually fed that data. Differential Revision: https://phab.mercurial-scm.org/D6816
-
Pierre-Yves David authored
This makes it possible for flag processors to -read- flag data. Differential Revision: https://phab.mercurial-scm.org/D6813
-
- Aug 17, 2019
-
-
Pulkit Goyal authored
I was trying to understand current interfaces and write new ones and I realized we need to improve how current interfaces are organised. This creates a dedicated folder for defining interfaces and move `repository.py` which defines all the current interfaces inside it. Differential Revision: https://phab.mercurial-scm.org/D6741
-
- Aug 07, 2019
-
-
Pierre-Yves David authored
We update callers incrementally because this help bisecting failures. This was useful during development, so we expect it might be useful again in the future.
-
- Jul 14, 2019
-
-
sliquister authored
I want to change the computation of the list of files modified by a commit. In principle, this would simply change a cache. But since this information is stored in commits rather than a cache, changing it means changing commit hashes (going forward). Some users rely on the convert extension from hg to hg not changing hashes when nothing changes (usually). Allow these users to preserve hashes despite changes to the changelog files computation by reusing these files lists when the manifest is unchanged (since these files list are derived from the manifest). Differential Revision: https://phab.mercurial-scm.org/D6643
-
- Jun 14, 2019
-
-
Matt Harbison authored
-
- May 04, 2019
-
-
Matt Harbison authored
This is a cleaned up and reorganized list of items I sent out about a year ago. But tracking this in the repo (like the narrow extension) gives more visibility in case anyone wants to help out.
-
- Feb 07, 2019
-
-
Matt Harbison authored
I ran into a problem pushing to an old Apache server- the normal outgoing traffic occurred, the Batch API request and response occurred, and then things suddenly halted. 5 minutes later, a 500 was returned, and the server log had a timeout reading 32K from `self._req.bodyfh` in hgweb.request.sendresponse(). Watching in WireShark, the Batch API got a 401, retried properly, then proceeded to PUT the blob (without authentication headers). This got a 401, but the client never retried with authentication. Worse, the blob was sent over the wire in the failed attempt. This kills digests for both the Batch API and the Transfer API. While in theory we could have the Batch API provide external URLs to a place that supports Basic Authentication, the LFS spec actually calls out using Basic Authentication[1]. It's not clear to me if they've been able to shoehorn in other methods. But let's keep it simple until somebody needs it. If we only had to support python2, we could just not add the handler for digest authentication. However in python3, AbstractBasicAuthHandler raises ValueError if it sees a scheme other than Basic. So we need to intercept all other schemes before it gets to that point. # no-check-commit because of urllib2.OpenerDirector foo_bar calling conventions [1] https://github.com/git-lfs/git-lfs/blob/master/docs/api/authentication.md
-
- Jan 29, 2019
-
-
Matt Harbison authored
This was the reason for test-lfs-test-server.t#git-server complaining about an "invalid byte in chunk length". For some reason if this isn't explicitly added, py3.7.1 is adding `transfer-encoding: chunked` as well as `Content-length: x`. Wireshark flagged this as malformed. However, if this is set, it doesn't bother with `transfer-encoding`. Before this patch with py3: PUT /objects/31cf46fbc4ecd458a0943c5b4881f1f5a6dd36c53d6167d5b69ac45149b38e5b HTTP/1.1 Accept-Encoding: identity Content-length: 12 accept: application/vnd.git-lfs content-type: application/octet-stream host: localhost:20062 transfer-encoding: chunked user-agent: git-lfs/2.3.4 (Mercurial 4.9rc0+149-7eb7637e34bf) Before this patch with py27: PUT /objects/31cf46fbc4ecd458a0943c5b4881f1f5a6dd36c53d6167d5b69ac45149b38e5b HTTP/1.1 Accept-Encoding: identity accept: application/vnd.git-lfs content-type: application/octet-stream content-length: 12 host: localhost:20062 user-agent: git-lfs/2.3.4 (Mercurial 4.9rc0+149-7eb7637e34bf+20190128) With this patch and py3, the content is the same as the py27 example. RFC2616 says to ignore `Content-Length` if `Transfer-Encoding` is present, so maybe there's nothing to do in the hg-server side (though I'm not sure which it is using if presented both). Maybe chunked encoding is better to do? If someone knows how to suppress the `Content-Length`, we can try that instead.
-
- Jan 28, 2019
-
-
Matt Harbison authored
For reasons unknown, py3 is adding an extra '\n' before the headers print out. This makes the output the same as py2.
-
- Jan 27, 2019
-
-
Matt Harbison authored
It got too confusing juggling r'' vs b'' across several functions.
-
Matt Harbison authored
This is (almost?) entirely from Augie's work. I'm a bit surprised that the JSON data is being encoded with ASCII via `pycompat.bytesurl()`- I would have thought UTF-8.
-
Matt Harbison authored
This is almost entirely b'' prefixing, with a couple of exceptions forced to bytes. Much of this is also borrowed from Augie's code. There's an HTTPError.read() that I flagged that I assume needs to be converted to bytes, but I can't find confirmation. Handling the deserialized JSON object over several functions made r'' vs b'' accesses confusing, so this assumes that the JSON object will be converted to bytes immediately. That will be done in the following commits, so it's not buried in these trivial changes.
-
Matt Harbison authored
Some of this is based on code written by Augie. I'm slightly unsure if these are the correct pycompat bytes <-> str conversion methods.
-
Matt Harbison authored
There are a ton of these changes in the blobstore, as well as r'' prefixing. So separating these out hopefully makes review easier. # skip-blame for b'' prefixing
-
- Jan 17, 2019
-
-
Matt Harbison authored
Yuya pointed out that this goes against the typical advice to not add attributes to classes[1]. The evolve extension still uses this a handful of times, so maybe it should be brought back in the future if a general use is found. But it isn't nice to have a new helper API that can lead to easy problems. [1] https://www.mercurial-scm.org/pipermail/mercurial-devel/2018-December/126330.html
-
- Dec 30, 2018
-
-
Martin von Zweigbergk authored
When a repo lock is released, we try to persist the manifest cache. That involves getting the narrowmatcher for the manifestlog. That should not fail if the store and working copy narrowspecs are out of sync. Without this patch, the later patches in this series will fail because of that. Differential Revision: https://phab.mercurial-scm.org/D5508
-
- Dec 28, 2018
-
-
Matt Harbison authored
Same mechanism as revsets earlier in this series. The LFS extension is updated to provide test coverage.
-