- Apr 08, 2018
-
-
Matt Harbison authored
Unlike pull, the blobs are uploaded within the exchange.push() window, so simply wrap it and swap in a properly configured remote store. The '_subtoppath' field shouldn't be available during this window, but give the passed path priority for clarity. At one point I hit an AttributeError in one of the convert tests when trying to save the original remote blobstore when the swap was run unconditionally. I wrapped it in a util.safehasattr(), but then today I wasn't able to reproduce it. But now the whole thing is tucked under the requirement guard because without the requirement, there are no blobs in the repo, even if the extension is loaded.
-
Matt Harbison authored
I don't see any easier way to do this because the update part of `hg pull -u` happens outside exchange.pull(), and commands.postincoming() doesn't take a path. So (ab)use the mechanism used by subrepos to redirect where subrepos are pulled from when an explicit path is given. As a bonus, this should allow lfs blobs to be pulled into a subrepo when it is checked out. An explicit push path can be handled within exchange.push(). That can be done next, outside of this dirty hack.
-
- Apr 11, 2018
-
-
Matt Harbison authored
The previous code worked on Windows, but not on Unix, and a pending patch's test failed. The url being used was something like "/tmp/.../client1/null://", courtesy of ui.configpath(). Looking at the doc comment, this seems like it's maybe not the right function to call (why should a relative cache path be expanded relative to the repo root or config file?), but largefiles has been using it since 8b8dd13295db (Oct 2011). It was introduced in 1b591f9b7fd2 (Jan 2011) without comment or callers. A grep over the whole history shows that only largefiles used it until lfs and infinitepush came along recently. It looks like if the `if not os.path.isabs(v) or "://" not in v` in configpath() is changed to an 'and', both Linux and Windows are happy. I'm guessing that "://" is to pick off URLs, so that seems reasonable. But I'm not sure why it isn't explicitly "file://", and I thought that "file://foo" is relative anyway. (At least, there are doctests for file:///tmp in util.url.) There is no mention of this setting in the help, but it is referenced on the wiki page for largefiles. (There's no mention that this is intended to be a URL, and the example uses an absolute path.) I don't want this blocking the rest of the lfs server discovery stuff. It was also wrong to allow a file:// URL here, but not in largefiles.
-
- Apr 04, 2018
-
-
Pulkit Goyal authored
The current tests in test-infinitepush-ci.t showed that `hg pull -r <rev>` does not work. Digging in code, I found that we have logic for pulling from bundlestore without having client side logic. This patch adds test demonstrating that pulling from bundlestore works when working over wire. Pulling from bundlestore when the peer is a localpeer still does not works. Differential Revision: https://phab.mercurial-scm.org/D3072
-
- Mar 31, 2018
-
-
Danny Hooper authored
Differential Revision: https://phab.mercurial-scm.org/D2988
-
- Apr 11, 2018
-
-
Pulkit Goyal authored
Differential Revision: https://phab.mercurial-scm.org/D3244
-
Gregory Szorc authored
With the new handshake defined and in place on the server, we can now implement it on the client. The HTTP handshake mechanism has been taught to add headers advertising its support for the new capabilities response. Response handling has been adjusted to allow CBOR responses through. And makepeer() has been taught to instantiate a mutually supported peer. The HTTPv2 peer class doesn't implement the full peer interface. So HTTPv2 is not yet usable as a peer. Like the server side, we support registering handlers for different API services. This allows extensions to easily implement API services and peers. A practical use case for this is to provide a previous implementation of the experimental version 2 wire protocol to a future version of Mercurial. We know there will be BC breaks after 4.6 ships. But someone could take the peer and server code from 4.6, drop it in an extension, and allow its use indefinitely. Differential Revision: https://phab.mercurial-scm.org/D3243
-
- Apr 10, 2018
-
-
Gregory Szorc authored
When clients connect to repositories over HTTP, they issue a request to the well-known URL "?cmd=capabilities" to fetch the repository capabilities. This is the handshake portion of the HTTP protocol. This commit defines a mechanism to use that HTTP request to return information about modern server features. If a client sends an X-HgUpgrade-* header containing a list of client-supported API names, the server responds with a response containing information about available services. This includes the normal capabilities string. So if the server doesn't support any newer services, the client can easily fall back. By advertising supported services from clients, server operators can see and log what client support exists in the wild. This will also help with debugging. The response contains the base path to API services. We know there are potential issues with the <repo>/api/ URL space conflicting with hgwebdir and subrepos. By making the API URL dynamic from the perspective of the client, the URL for APIs is not subject to backwards compatibility concerns - at least as long as a ?cmd=capabilities request is made. We've also defined the ``cbor`` client capability for the X-HgProto-* header. This MUST be sent in order to get the modern response from "?cmd=capabilities". During implementation, I initially always sent an application/mercurial-cbor response. However, the handshake mechanism will be more future compatible if the client is in charge of which formats to request. We already perform content negotiation from X-HgProto-*, so keying off this for the capabilities response feels appropriate. In addition, I initially used application/cbor. However, it is conceivable that a non-Mercurial server could serve application/cbor. To rule out this possibility, I've invented a new media type that is Mercurial specific and can't be confused for generic CBOR. Differential Revision: https://phab.mercurial-scm.org/D3242
-
- Apr 11, 2018
-
-
Gregory Szorc authored
We don't need to be advertising client protocol parameters as part of the capabilities request during the handshake because nothing in version 1 of the wire protocol will use this data. i.e. the advertisement is wasteful. Differential Revision: https://phab.mercurial-scm.org/D3241
-
- Apr 10, 2018
-
-
Gregory Szorc authored
Before, we manually updated the Vary header value for each header contributing to it. All X-Hg* headers are reserved for the Mercurial protocol and could have caching implications. So it makes sense to always add these headers to Vary. A test revealed that X-HgArgs-Post wasn't being added to Vary. This is only sent on POST requests. POST requests generally aren't cacheable. However, it is possible if the server sends the appropriate headers. Mercurial shouldn't be sending those headers. But let's not take any chances. Differential Revision: https://phab.mercurial-scm.org/D3240
-
Gregory Szorc authored
Versions of Mercurial older than 1.0 emitted the text/plain and application/hg-changegroup media types in response to wire protocol commands. Way back in 8760d0c83b9b in 2005, the code validating these media types was added, presumably for backwards compatibility. 0b245edec124 a short time before that commit changed things from text/plain and application/hg-changegroup to application/mercurial-0.1 and application/hg-0.1. 8760d0c83b9b seemed to indicate ("for now") that the BC compatibility was temporary. But that code has lived until this day. It has been more than 10 years and nobody should be running pre 1.0 servers. Pretty much the only risk to this is if there's a server somewhere advertising the old media types or server software is interfering and not letting Mercurial send the proper Content-Type header. I think the chances are rare. The wire protocol docs were created (by me) from reading existing code. So the deletions don't constitute a spec change as much as reflecting the reality of how things have been for years. .. bc:: The HTTP client no longer accepts text/plain and application/hg-changegroup Content-Type values as a valid Mercurial command response. These should only be encountered on pre 1.0 Mercurial servers. Differential Revision: https://phab.mercurial-scm.org/D3239
-
Gregory Szorc authored
This allows us to use makepeer() in `hg debugwireproto`. Differential Revision: https://phab.mercurial-scm.org/D3238
-
Gregory Szorc authored
Previously, we constructed an httppeer then always ran _fetchcaps() to issue the capabilities command. We want to issue the capabilities command before constructing a peer instance so we can construct an appropriate peer instance depending on the capabilities result. With the code for making and sending requests moved out of httppeer, it is now possible to send command requests without an httppeer. This commit creates a new function for making the capabilities request and calls it as part of makepeer(). This code should be functionality equivalent to what existed before. Differential Revision: https://phab.mercurial-scm.org/D3237
-
Gregory Szorc authored
This allows the common redirect detection, content type validation, and decompression wrapping to be usable outside of httppeer instances. Differential Revision: https://phab.mercurial-scm.org/D3236
-
Gregory Szorc authored
This is common for all HTTP requests. It should be part of sendrequest(). Differential Revision: https://phab.mercurial-scm.org/D3235
-
Gregory Szorc authored
Some of this feels awkward, such as having to pass in a function to evaluate a capability. And this code is generally pretty difficult to read. I didn't want to perform too much refactoring as part of the code move since it would make review more difficult. Differential Revision: https://phab.mercurial-scm.org/D3234
-
Gregory Szorc authored
This is generic and doesn't need to live as a method of httppeer. Differential Revision: https://phab.mercurial-scm.org/D3233
-
Gregory Szorc authored
Upcoming commits will move the initial ?cmd=capabilities handshake request out of httppeer so the handshake can be performed before a peer instance is constructed. In order to do this, we'll need to refactor code for making HTTP requests. The type used to construct HTTP requests is configurable. If we'll be making HTTP requests outside of httppeer, we should be able to use a custom request builder. So move the definition of that type into makepeer(). Extensions can monkeypatch the function and override the argument value. Differential Revision: https://phab.mercurial-scm.org/D3232
-
Gregory Szorc authored
This is relatively straightforward. As part of this, we introduced a local @wireprotocommand that wraps the main one and defines a v2 only policy by default. Because the hacky HTTPv2 peer isn't using capabilities response yet, we had to move some code around to force import of wireprotov2server so commands are registered. This is super hacky. But this code will go away once the HTTPv2 peer is using the capabilities response to derive permissions. Differential Revision: https://phab.mercurial-scm.org/D3231
-
Gregory Szorc authored
wireprotoserver has generic and version 1 server code. The wireproto module also has both version 1 and version 2 command implementations. Upcoming work I want to do will make it difficult for this code to live in the current locations. Plus, it kind of makes sense for the version 2 code to live in an isolated module. This commit copies the HTTPv2 bits from wireprotoserver into a new module. We do it as a file copy to preserve history. A future commit will be copying wire protocol commands into this module as well. But there is little history of that code, so it makes sense to take history for wireprotoserver. Differential Revision: https://phab.mercurial-scm.org/D3230
-
- Apr 09, 2018
-
-
Gregory Szorc authored
We can now feed received frames into the client reactor and it will validate their sanity, dispatch them appropriately. The hacky HTTP peer has been updated to use the new code. No existing tests changed, somewhat proving the code works as expected. Rudimentary unit tests for the new functionality have been implemented. Differential Revision: https://phab.mercurial-scm.org/D3224
-
Gregory Szorc authored
We have a nice state machine of sorts for reacting to server-side events. Now it is time to implement the client equivalent. We introduce a "clientreactor." It allows callers to request that commands be issued. It has multiple modes of operation to reflect what the underlying transport supports. e.g. for SSH, we can perform wire sends immediately but for HTTP we need to buffer sends until all command requests are received. In addition, SSH allows sending multiple requests as long as the connection is open. But HTTP/1.1 only allows sending request data once. For SSH, we'll have one reactor per connection. For HTTP, we'll have one reactor per HTTP request. But because code that calls wire protocol commands should not be aware of how the underlying transport works, this will all be abstracted away by the peer interface. Our crude HTTP peer has been updated to use the reactor instead of formulating frames directly. No behavior should have changed here and tests seem to confirm that. Basic unit tests for the reactor behavior have been added. Differential Revision: https://phab.mercurial-scm.org/D3223
-
Gregory Szorc authored
I was lazy when I put these in test-wireproto-serverreactor.py. Let's do it properly. Differential Revision: https://phab.mercurial-scm.org/D3222
-
Gregory Szorc authored
I think it will be more trouble than it is worth to code version 1 and version 2 command handlers to the same interface. It will feel awkward to shoehorn functionality into e.g. the version 1 protocol handler interface. This would likely constrain the ability for version 2 to evolve. Previous commits introduced a clean separation between command handlers for version 1 and version 2 transports. This commit reinforces that separation by dropping support for having a single command handler service both version 1 and version 2 transports. Differential Revision: https://phab.mercurial-scm.org/D3208
-
Gregory Szorc authored
For backwards compatibility reasons. We want extension provided commands to opt in to version 2 rather than get inherited automatically. This will facilitate a clean break between the protocols. As part of this, we duplicate some commands used in tests so there are different command handlers per transport. Differential Revision: https://phab.mercurial-scm.org/D3207
-
Gregory Szorc authored
These are the most complicated wire protocol commands. I don't want to deal with porting them just yet. Let's disable both of them on version 2 transports so we drive the final wedge between command handlers and start to evolve version 2 command handlers more. Differential Revision: https://phab.mercurial-scm.org/D3206
-
- Apr 07, 2018
-
-
Gregory Szorc authored
This is pretty straightforward. We don't yet handle errors because we don't have an error handling mechanism in place yet. I'm also tempted to fold this into `known`. We'll come back to this later. Differential Revision: https://phab.mercurial-scm.org/D3205
-
Gregory Szorc authored
It doesn't do output redirection yet. And I'd love to generally overhaul the pushkey protocol for wire protocol version 2. But this will be a bit of effort. Let's do it as a follow-up. Differential Revision: https://phab.mercurial-scm.org/D3204
-
Gregory Szorc authored
This may make a comeback in wire protocol version 2. The feature definitely needs to be carried forward. But at this juncture, I'm flirting with the idea of implementing this via a "redirect" mechanism at the command response level itself rather than something that requires one-off client support for querying and handling. i.e. I want to make it so servers can say "fetch this first and then come back" and clients handle that automatically. This would not only support clone bundles, but would also support piece-meal "pull bundles." Whatever happens, we can deal with it down the road. Differential Revision: https://phab.mercurial-scm.org/D3203
-
Gregory Szorc authored
Exposing the set of argument names is cool. But with wire protocol version 2, we're using CBOR to transport arguments and this allows us to have typing for arguments. Typed arguments are much nicer because they will cut down on transfer overhead and processing overhead for decoding values. This commit teaches @wireprotocommand to accept a dictionary for arguments. The arguments registered for version 2 transports are canonically stored as dictionaries rather than a space-delimited string. It is an error to defined arguments with a dictionary for commands using version 1 transports. This reinforces my intent to fully decouple command handlers for version 2 transports. Differential Revision: https://phab.mercurial-scm.org/D3202
-
- Apr 06, 2018
-
-
Gregory Szorc authored
I have plans to implement stream clone using a better mechanism than this existing command. Let's not carry it forward to wire protocol version 2. Differential Revision: https://phab.mercurial-scm.org/D3201
-
- Apr 09, 2018
-
-
Gregory Szorc authored
The capabilities mechanism for wire protocol version 2 represents a clean break from version 1. Instead of effectively exchanging a set of capabilities, we're exchanging a rich data structure. This data structure currently contains information about every available command, including its accepted arguments. It also contains information about supported compression formats. Exposing information about supported commands will allow clients to automatically generate bindings to the server. Clients will be able to do things like detect when they are attempting to run a command that isn't known to the server. Exposing the required permissions to run a command can be used by clients to determine if they have privileges to call a command before actually calling it. We could potentially even have clients send credentials preemptively without waiting for the server to deny the command request. Lots of potential here. The data returned by this command will likely evolve heavily. So we shouldn't bikeshed the implementation just yet. Differential Revision: https://phab.mercurial-scm.org/D3200
-
- Apr 08, 2018
-
-
Martin von Zweigbergk authored
It's close to code freeze, and dropping support for repo['123'] and repo ['my-bookmark'] and repo['deadbeef'] is pretty dispruptive, so this just adds deprecation warnings so extensions can easily find the places they need to fix. Differential Revision: https://phab.mercurial-scm.org/D3197
-
Martin von Zweigbergk authored
The only remaining cases where we were delegating unhandled symbols to repo.__getitem__ should now be when the symbol could not be found. In that case we just delegated to repo.__getitem__ for the error message. Let's just copy the error message instead. If there were any cases where we got e.g. a binary nodeid or an integer revnum into revsymbol() (e.g. via repo.lookup()), we'd now start raising an exception instead. That is why this is marked (API). This affects one test case, but the new behavior seems better to me. I can't tell if the old behavior was desired or if the test was just there to document how it happened to work. Differential Revision: https://phab.mercurial-scm.org/D3196
-
Martin von Zweigbergk authored
Similar reasoning as previous patches. Differential Revision: https://phab.mercurial-scm.org/D3195
-
- Apr 07, 2018
-
-
Martin von Zweigbergk authored
Similar reasoning as previous patches. Differential Revision: https://phab.mercurial-scm.org/D3194
-
Martin von Zweigbergk authored
This is a bit unfortunate, but it enables moving other pieces out of changectx's constructor without affecting the order in which we look up things (e.g. hex nodeid before bookmark). We convert nodeid to revnum before calling repo.__getitem__, even though that will result in converting back to nodeid later. This is so we can handle the LookupError and attempt to interpret the string as something else (e.g. a bookmark). We also need to start handling WdirUnsupported now, since the full hex nodeid "ffff..." represents the working directory. The exception is raised by the revlog layer. Differential Revision: https://phab.mercurial-scm.org/D3193
-
Martin von Zweigbergk authored
This patch copies the handling of stringified ints from changectx's constructor. It then calls repo.__getitem__ with the int. Since that method only interprets integers as revnums the first thing it does, this will not be redoing any of the work already done. We leave the old code in place so we can later deprecate it instead of breaking extensions. Differential Revision: https://phab.mercurial-scm.org/D3146
-
- Apr 10, 2018
-
-
Pulkit Goyal authored
# skip-blame because just b'' prefix Differential Revision: https://phab.mercurial-scm.org/D3220
-
Pulkit Goyal authored
Comparing None with integers was fine in Python 2 but returns error in Python 3 which is nice. This patch replaces None with -1 where sorting is done and some related logic. Differential Revision: https://phab.mercurial-scm.org/D3219
-