Skip to content
Snippets Groups Projects
  1. Apr 08, 2018
    • Matt Harbison's avatar
      lfs: infer the blob store URL from an explicit push dest or default-push · 31a4ea77
      Matt Harbison authored
      Unlike pull, the blobs are uploaded within the exchange.push() window, so simply
      wrap it and swap in a properly configured remote store.  The '_subtoppath' field
      shouldn't be available during this window, but give the passed path priority for
      clarity.
      
      At one point I hit an AttributeError in one of the convert tests when trying to
      save the original remote blobstore when the swap was run unconditionally.  I
      wrapped it in a util.safehasattr(), but then today I wasn't able to reproduce
      it.  But now the whole thing is tucked under the requirement guard because
      without the requirement, there are no blobs in the repo, even if the extension
      is loaded.
      31a4ea77
    • Matt Harbison's avatar
      lfs: infer the blob store URL from an explicit pull source · be1cc65b
      Matt Harbison authored
      I don't see any easier way to do this because the update part of `hg pull -u`
      happens outside exchange.pull(), and commands.postincoming() doesn't take a
      path.  So (ab)use the mechanism used by subrepos to redirect where subrepos are
      pulled from when an explicit path is given.  As a bonus, this should allow lfs
      blobs to be pulled into a subrepo when it is checked out.
      
      An explicit push path can be handled within exchange.push().  That can be done
      next, outside of this dirty hack.
      be1cc65b
  2. Apr 11, 2018
    • Matt Harbison's avatar
      lfs: special case the null:// usercache instead of treating it as a url · e5cd8d1a
      Matt Harbison authored
      The previous code worked on Windows, but not on Unix, and a pending patch's test
      failed.  The url being used was something like "/tmp/.../client1/null://",
      courtesy of ui.configpath().  Looking at the doc comment, this seems like it's
      maybe not the right function to call (why should a relative cache path be
      expanded relative to the repo root or config file?), but largefiles has been
      using it since 8b8dd13295db (Oct 2011).  It was introduced in 1b591f9b7fd2 (Jan
      2011) without comment or callers.  A grep over the whole history shows that only
      largefiles used it until lfs and infinitepush came along recently.
      
      It looks like if the `if not os.path.isabs(v) or "://" not in v` in configpath()
      is changed to an 'and', both Linux and Windows are happy.  I'm guessing that
      "://" is to pick off URLs, so that seems reasonable.  But I'm not sure why it
      isn't explicitly "file://", and I thought that "file://foo" is relative anyway.
      (At least, there are doctests for file:///tmp in util.url.)  There is no mention
      of this setting in the help, but it is referenced on the wiki page for
      largefiles.  (There's no mention that this is intended to be a URL, and the
      example uses an absolute path.)
      
      I don't want this blocking the rest of the lfs server discovery stuff.  It was
      also wrong to allow a file:// URL here, but not in largefiles.
      e5cd8d1a
  3. Apr 04, 2018
  4. Mar 31, 2018
  5. Apr 11, 2018
    • Pulkit Goyal's avatar
      c4a0626f
    • Gregory Szorc's avatar
      httppeer: support protocol upgrade · 8a731322
      Gregory Szorc authored
      With the new handshake defined and in place on the server, we can
      now implement it on the client.
      
      The HTTP handshake mechanism has been taught to add headers advertising
      its support for the new capabilities response. Response handling
      has been adjusted to allow CBOR responses through. And makepeer()
      has been taught to instantiate a mutually supported peer.
      
      The HTTPv2 peer class doesn't implement the full peer interface. So
      HTTPv2 is not yet usable as a peer.
      
      Like the server side, we support registering handlers for
      different API services. This allows extensions to easily implement
      API services and peers. A practical use case for this is to
      provide a previous implementation of the experimental version 2
      wire protocol to a future version of Mercurial. We know there will
      be BC breaks after 4.6 ships. But someone could take the peer and
      server code from 4.6, drop it in an extension, and allow its use
      indefinitely.
      
      Differential Revision: https://phab.mercurial-scm.org/D3243
      8a731322
  6. Apr 10, 2018
    • Gregory Szorc's avatar
      wireproto: define and implement HTTP handshake to upgrade protocol · 734515ac
      Gregory Szorc authored
      When clients connect to repositories over HTTP, they issue a request
      to the well-known URL "?cmd=capabilities" to fetch the repository
      capabilities. This is the handshake portion of the HTTP protocol.
      
      This commit defines a mechanism to use that HTTP request to return
      information about modern server features.
      
      If a client sends an X-HgUpgrade-* header containing a list of
      client-supported API names, the server responds with a response
      containing information about available services. This includes
      the normal capabilities string. So if the server doesn't support
      any newer services, the client can easily fall back.
      
      By advertising supported services from clients, server operators
      can see and log what client support exists in the wild. This will
      also help with debugging.
      
      The response contains the base path to API services. We know there
      are potential issues with the <repo>/api/ URL space conflicting with
      hgwebdir and subrepos. By making the API URL dynamic from the
      perspective of the client, the URL for APIs is not subject to backwards
      compatibility concerns - at least as long as a ?cmd=capabilities request
      is made.
      
      We've also defined the ``cbor`` client capability for the X-HgProto-*
      header. This MUST be sent in order to get the modern response from
      "?cmd=capabilities". During implementation, I initially always sent
      an application/mercurial-cbor response. However, the handshake
      mechanism will be more future compatible if the client is in charge
      of which formats to request. We already perform content negotiation
      from X-HgProto-*, so keying off this for the capabilities response
      feels appropriate.
      
      In addition, I initially used application/cbor. However, it is
      conceivable that a non-Mercurial server could serve application/cbor.
      To rule out this possibility, I've invented a new media type that
      is Mercurial specific and can't be confused for generic CBOR.
      
      Differential Revision: https://phab.mercurial-scm.org/D3242
      734515ac
  7. Apr 11, 2018
  8. Apr 10, 2018
    • Gregory Szorc's avatar
      httppeer: always add x-hg* headers to Vary header · 930c433e
      Gregory Szorc authored
      Before, we manually updated the Vary header value for each
      header contributing to it.
      
      All X-Hg* headers are reserved for the Mercurial protocol and
      could have caching implications. So it makes sense to always add
      these headers to Vary.
      
      A test revealed that X-HgArgs-Post wasn't being added to Vary.
      This is only sent on POST requests. POST requests generally
      aren't cacheable. However, it is possible if the server sends
      the appropriate headers. Mercurial shouldn't be sending those
      headers. But let's not take any chances.
      
      Differential Revision: https://phab.mercurial-scm.org/D3240
      930c433e
    • Gregory Szorc's avatar
      httppeer: don't accept very old media types (BC) · 301a1d2e
      Gregory Szorc authored
      Versions of Mercurial older than 1.0 emitted the text/plain
      and application/hg-changegroup media types in response to wire
      protocol commands.
      
      Way back in 8760d0c83b9b in 2005, the code validating these media
      types was added, presumably for backwards compatibility. 0b245edec124
      a short time before that commit changed things from text/plain and
      application/hg-changegroup to application/mercurial-0.1 and
      application/hg-0.1. 8760d0c83b9b seemed to indicate ("for now") that
      the BC compatibility was temporary. But that code has lived until
      this day.
      
      It has been more than 10 years and nobody should be running pre 1.0
      servers.
      
      Pretty much the only risk to this is if there's a server somewhere
      advertising the old media types or server software is interfering
      and not letting Mercurial send the proper Content-Type header. I
      think the chances are rare.
      
      The wire protocol docs were created (by me) from reading existing
      code. So the deletions don't constitute a spec change as much as
      reflecting the reality of how things have been for years.
      
      .. bc::
      
         The HTTP client no longer accepts text/plain and
         application/hg-changegroup Content-Type values as a valid Mercurial
         command response. These should only be encountered on pre 1.0
         Mercurial servers.
      
      Differential Revision: https://phab.mercurial-scm.org/D3239
      301a1d2e
    • Gregory Szorc's avatar
      httppeer: allow opener to be passed to makepeer() · 6b08cf6b
      Gregory Szorc authored
      This allows us to use makepeer() in `hg debugwireproto`.
      
      Differential Revision: https://phab.mercurial-scm.org/D3238
      6b08cf6b
    • Gregory Szorc's avatar
      httppeer: perform capabilities request in makepeer() · 8b8a845c
      Gregory Szorc authored
      Previously, we constructed an httppeer then always ran _fetchcaps()
      to issue the capabilities command.
      
      We want to issue the capabilities command before constructing a
      peer instance so we can construct an appropriate peer instance
      depending on the capabilities result.
      
      With the code for making and sending requests moved out of httppeer,
      it is now possible to send command requests without an httppeer.
      
      This commit creates a new function for making the capabilities
      request and calls it as part of makepeer().
      
      This code should be functionality equivalent to what existed before.
      
      Differential Revision: https://phab.mercurial-scm.org/D3237
      8b8a845c
    • Gregory Szorc's avatar
      httppeer: extract common response handling into own function · 946eb204
      Gregory Szorc authored
      This allows the common redirect detection, content type
      validation, and decompression wrapping to be usable outside of
      httppeer instances.
      
      Differential Revision: https://phab.mercurial-scm.org/D3236
      946eb204
    • Gregory Szorc's avatar
      httppeer: move error handling and response wrapping into sendrequest · b5862ee0
      Gregory Szorc authored
      This is common for all HTTP requests. It should be part of
      sendrequest().
      
      Differential Revision: https://phab.mercurial-scm.org/D3235
      b5862ee0
    • Gregory Szorc's avatar
      httppeer: extract code for creating a request into own function · 66d1001e
      Gregory Szorc authored
      Some of this feels awkward, such as having to pass in a function
      to evaluate a capability. And this code is generally pretty difficult
      to read. I didn't want to perform too much refactoring as part of
      the code move since it would make review more difficult.
      
      Differential Revision: https://phab.mercurial-scm.org/D3234
      66d1001e
    • Gregory Szorc's avatar
      httppeer: extract code for performing an HTTP request · 8e7a4435
      Gregory Szorc authored
      This is generic and doesn't need to live as a method of httppeer.
      
      Differential Revision: https://phab.mercurial-scm.org/D3233
      8e7a4435
    • Gregory Szorc's avatar
      httppeer: move requestbuilder defaults into makepeer() argument · 835ccc2a
      Gregory Szorc authored
      Upcoming commits will move the initial ?cmd=capabilities handshake
      request out of httppeer so the handshake can be performed before a
      peer instance is constructed. In order to do this, we'll need to
      refactor code for making HTTP requests.
      
      The type used to construct HTTP requests is configurable. If we'll
      be making HTTP requests outside of httppeer, we should be able to
      use a custom request builder. So move the definition of that type
      into makepeer().
      
      Extensions can monkeypatch the function and override the argument
      value.
      
      Differential Revision: https://phab.mercurial-scm.org/D3232
      835ccc2a
    • Gregory Szorc's avatar
      wireproto: move version 2 command handlers to wireprotov2server · 3a2367e6
      Gregory Szorc authored
      This is relatively straightforward.
      
      As part of this, we introduced a local @wireprotocommand that
      wraps the main one and defines a v2 only policy by default.
      
      Because the hacky HTTPv2 peer isn't using capabilities response
      yet, we had to move some code around to force import of
      wireprotov2server so commands are registered. This is super
      hacky. But this code will go away once the HTTPv2 peer is using
      the capabilities response to derive permissions.
      
      Differential Revision: https://phab.mercurial-scm.org/D3231
      3a2367e6
    • Gregory Szorc's avatar
      wireproto: extract HTTP version 2 code to own module · 93397c46
      Gregory Szorc authored
      wireprotoserver has generic and version 1 server code. The wireproto
      module also has both version 1 and version 2 command implementations.
      Upcoming work I want to do will make it difficult for this code to
      live in the current locations. Plus, it kind of makes sense for the
      version 2 code to live in an isolated module.
      
      This commit copies the HTTPv2 bits from wireprotoserver into a new
      module. We do it as a file copy to preserve history. A future
      commit will be copying wire protocol commands into this module
      as well. But there is little history of that code, so it makes
      sense to take history for wireprotoserver.
      
      Differential Revision: https://phab.mercurial-scm.org/D3230
      93397c46
  9. Apr 09, 2018
    • Gregory Szorc's avatar
      wireproto: client reactor support for receiving frames · 55b5ba8d
      Gregory Szorc authored
      We can now feed received frames into the client reactor and it will
      validate their sanity, dispatch them appropriately.
      
      The hacky HTTP peer has been updated to use the new code. No
      existing tests changed, somewhat proving the code works as
      expected.
      
      Rudimentary unit tests for the new functionality have been
      implemented.
      
      Differential Revision: https://phab.mercurial-scm.org/D3224
      55b5ba8d
    • Gregory Szorc's avatar
      wireproto: introduce a reactor for client-side state · 01361be9
      Gregory Szorc authored
      We have a nice state machine of sorts for reacting to server-side
      events. Now it is time to implement the client equivalent.
      
      We introduce a "clientreactor." It allows callers to request
      that commands be issued. It has multiple modes of operation to
      reflect what the underlying transport supports. e.g. for SSH,
      we can perform wire sends immediately but for HTTP we need to
      buffer sends until all command requests are received. In addition,
      SSH allows sending multiple requests as long as the connection is
      open. But HTTP/1.1 only allows sending request data once.
      
      For SSH, we'll have one reactor per connection. For HTTP, we'll
      have one reactor per HTTP request. But because code that calls
      wire protocol commands should not be aware of how the underlying
      transport works, this will all be abstracted away by the peer
      interface.
      
      Our crude HTTP peer has been updated to use the reactor instead
      of formulating frames directly. No behavior should have changed
      here and tests seem to confirm that.
      
      Basic unit tests for the reactor behavior have been added.
      
      Differential Revision: https://phab.mercurial-scm.org/D3223
      01361be9
    • Gregory Szorc's avatar
      tests: extract wire protocol framing tests to own file · 1ec5ce21
      Gregory Szorc authored
      I was lazy when I put these in test-wireproto-serverreactor.py. Let's
      do it properly.
      
      Differential Revision: https://phab.mercurial-scm.org/D3222
      1ec5ce21
    • Gregory Szorc's avatar
      wireproto: disallow commands handlers for multiple transport versions · 3e5e3720
      Gregory Szorc authored
      I think it will be more trouble than it is worth to code version 1
      and version 2 command handlers to the same interface. It will feel
      awkward to shoehorn functionality into e.g. the version 1 protocol
      handler interface. This would likely constrain the ability for version
      2 to evolve.
      
      Previous commits introduced a clean separation between command handlers
      for version 1 and version 2 transports. This commit reinforces that
      separation by dropping support for having a single command handler
      service both version 1 and version 2 transports.
      
      Differential Revision: https://phab.mercurial-scm.org/D3208
      3e5e3720
    • Gregory Szorc's avatar
      wireproto: make @wireprotocommand version 1 only by default · 693cb376
      Gregory Szorc authored
      For backwards compatibility reasons. We want extension provided
      commands to opt in to version 2 rather than get inherited
      automatically. This will facilitate a clean break between the
      protocols.
      
      As part of this, we duplicate some commands used in tests so
      there are different command handlers per transport.
      
      Differential Revision: https://phab.mercurial-scm.org/D3207
      693cb376
    • Gregory Szorc's avatar
      wireproto: only expose "getbundle" and "unbundle" to v1 transports · 4a0d58d6
      Gregory Szorc authored
      These are the most complicated wire protocol commands. I don't want
      to deal with porting them just yet. Let's disable both of them on
      version 2 transports so we drive the final wedge between command
      handlers and start to evolve version 2 command handlers more.
      
      Differential Revision: https://phab.mercurial-scm.org/D3206
      4a0d58d6
  10. Apr 07, 2018
    • Gregory Szorc's avatar
      wireproto: port lookup to wire protocol v2 · 89fed81b
      Gregory Szorc authored
      This is pretty straightforward. We don't yet handle errors because we
      don't have an error handling mechanism in place yet.
      
      I'm also tempted to fold this into `known`. We'll come back to this
      later.
      
      Differential Revision: https://phab.mercurial-scm.org/D3205
      89fed81b
    • Gregory Szorc's avatar
      wireproto: port pushkey command to wire protocol version 2 · be5d4749
      Gregory Szorc authored
      It doesn't do output redirection yet. And I'd love to generally overhaul
      the pushkey protocol for wire protocol version 2. But this will be a bit
      of effort. Let's do it as a follow-up.
      
      Differential Revision: https://phab.mercurial-scm.org/D3204
      be5d4749
    • Gregory Szorc's avatar
      wireproto: only expose "clonebundles" to version 1 transports · 2003da12
      Gregory Szorc authored
      This may make a comeback in wire protocol version 2. The feature
      definitely needs to be carried forward. But at this juncture, I'm
      flirting with the idea of implementing this via a "redirect"
      mechanism at the command response level itself rather than something
      that requires one-off client support for querying and handling.
      i.e. I want to make it so servers can say "fetch this first and
      then come back" and clients handle that automatically. This would
      not only support clone bundles, but would also support piece-meal
      "pull bundles." Whatever happens, we can deal with it down the
      road.
      
      Differential Revision: https://phab.mercurial-scm.org/D3203
      2003da12
    • Gregory Szorc's avatar
      wireproto: define and expose types of wire command arguments · 69e46c18
      Gregory Szorc authored
      Exposing the set of argument names is cool. But with wire protocol
      version 2, we're using CBOR to transport arguments and this allows us
      to have typing for arguments.
      
      Typed arguments are much nicer because they will cut down on transfer
      overhead and processing overhead for decoding values.
      
      This commit teaches @wireprotocommand to accept a dictionary for
      arguments. The arguments registered for version 2 transports are
      canonically stored as dictionaries rather than a space-delimited string.
      
      It is an error to defined arguments with a dictionary for commands using
      version 1 transports. This reinforces my intent to fully decouple command
      handlers for version 2 transports.
      
      Differential Revision: https://phab.mercurial-scm.org/D3202
      69e46c18
  11. Apr 06, 2018
  12. Apr 09, 2018
    • Gregory Szorc's avatar
      wireproto: implement capabilities for wire protocol v2 · df498549
      Gregory Szorc authored
      The capabilities mechanism for wire protocol version 2 represents a
      clean break from version 1.
      
      Instead of effectively exchanging a set of capabilities, we're
      exchanging a rich data structure.
      
      This data structure currently contains information about
      every available command, including its accepted arguments. It also
      contains information about supported compression formats.
      
      Exposing information about supported commands will allow clients
      to automatically generate bindings to the server. Clients will be
      able to do things like detect when they are attempting to run a
      command that isn't known to the server. Exposing the required
      permissions to run a command can be used by clients to determine if
      they have privileges to call a command before actually calling it.
      We could potentially even have clients send credentials
      preemptively without waiting for the server to deny the command
      request. Lots of potential here.
      
      The data returned by this command will likely evolve heavily. So we
      shouldn't bikeshed the implementation just yet.
      
      Differential Revision: https://phab.mercurial-scm.org/D3200
      df498549
  13. Apr 08, 2018
  14. Apr 07, 2018
  15. Apr 10, 2018
Loading