Skip to content
Snippets Groups Projects
  1. Aug 30, 2023
    • Georges Racinet's avatar
      rhgitaly::streaming::stream_chunks · d124dd52
      Georges Racinet authored
      This Rust version of Python `hgitaly.util.chunked` should be useful
      for many gRPC methods. It provide the means to distinguish the first
      response, but does not implement the pagination protocol.
      d124dd52
  2. Aug 18, 2023
  3. Aug 12, 2023
  4. Aug 07, 2023
  5. Jul 31, 2023
  6. Aug 07, 2023
    • Georges Racinet's avatar
      rhgitaly::oid: methods for tree handling · eb6763d7
      Georges Racinet authored
      There are for now completely identical to methods for blobs,
      but they could well become different (e.g, different disk caches or
      whatever).
      eb6763d7
    • Georges Racinet's avatar
      rhgitaly::mercurial::ls_path: listing a subset of manifest · 721ac5a4
      Georges Racinet authored
      This will be useful to implement `CommitService.TreeEntry`.
      There is plenty of room for performance improvement, notably:
      
      - the position of the first relevant entry could be found
        by binary search instead of a linear scan
      - we could try and clone less
      
      But it is possible that this would not be really needed, given
      the performance and scalability boost given by RHGitaly over
      HGitaly, before Mercurial manifests start using a more efficient
      layout, making this code obsolete.
      721ac5a4
    • Georges Racinet's avatar
      RHGitaly: allowing to stream TreeEntryResponse from BlobResponseChunk · 87b6970c
      Georges Racinet authored
      We will soon implement `CommitService.TreeEntry`, which has to
      behave like `GetBlobs` on one `RevisionPath` in case it turns out
      the path is a blob.
      87b6970c
  7. Jul 31, 2023
    • Georges Racinet's avatar
      rhgitaly::mercurial: preparation for CommitService.TreeEntry · 562c5d46
      Georges Racinet authored
      We split `lookup_blob()` in two functions, so that
      parts can be reused without reopening changelogs, manifests etc.
      Notably, the implementation of `TreeEntry` will use `read_blob`
      in case it is serving a blob and not a tree: in this case
      the manifest entry has already been found, no need to reopen and
      rescan anything.
      562c5d46
  8. Aug 06, 2023
    • Georges Racinet's avatar
      RHGitaly: bumped MSRV to 1.65 · 2acf3e5f
      Georges Racinet authored
      Rust 1.65 is currently the version generally available in GitLab
      context (upstream usage appeared recently) and it is also able
      to build tokei 12.1.2, whereas 1.61 cannot.
      
      This leads up to fix the following Clippy error (not in 1.61):
      "you are deriving `PartialEq` and can implement `Eq`".
      2acf3e5f
  9. Jul 31, 2023
    • Georges Racinet's avatar
      RHGitaly implement HasLocalBranches · e0fbe488
      Georges Racinet authored
      This method is mostly used to check if the repository is emtpy (and
      that is in the case of Mercurial logically equivalent),
      but it's better to implement it using the GitLab branches file as it
      avoids actually opening the repository, spawning a thread for
      blocking iteration etc.
      e0fbe488
  10. Jul 29, 2023
    • Georges Racinet's avatar
      RHGitaly: implementing BlobService.GetBlobs · fd8aaff5
      Georges Racinet authored
      This finally puts together all the common logic already used in
      `GetBlob`, and we can switch on the Comparison tests (one single
      test function for both gRPC methods).
      
      Using `blocking_gitlab_revision_node_prefix` is not very elegant,
      we could probably resolve all the revisions before entering the blocking
      thread (and perhaps also avoid scanning the state files many times),
      but this is deemed not so important for the time being, compared to
      the huge efficiency boost provided to Heptapod by
      implementing this method in Rust.
      
      As is now customary, the tracing part has us create a wrapper struct
      and implement `Debug` because of the bytes in `RevisionPath`. At
      some point we'll have to come up with something more generic for this.
      fd8aaff5
    • Georges Racinet's avatar
      RHGitaly: blocking variant of revision resolution · 81cb41f4
      Georges Racinet authored
      This will be used once in a thread created by `spawn_blocking`,
      typically with the closure given to the likes of
      `repository::load_repo_and_stream`.
      
      It is a bit annoying to introduce the dependency to
      `futures::executor` for this, but we don't feel at ease
      using `tokio::runtime::Runtime.block_on`, as this is really
      meant as an entry point (looks like we'd need to create a new
      runtime and it's not clear whether it would interfere with the
      current running one or not).
      81cb41f4
    • Georges Racinet's avatar
      RHGitaly: BlobService.GetBlob · 03355820
      Georges Racinet authored
      This builds on the previously introduced repository file
      extraction and conversion to `GetBlobResponse`.
      
      It is important to emit at least one response, even if the
      resulting data is empty (usually because of `limit=0`).
      03355820
    • Georges Racinet's avatar
      RHGitaly: Blob gRPC messages definitions and adapters · d274bd92
      Georges Racinet authored
      This handles the fact that `GetBlobResponse` and `GetBlobsResponse` have
      some similar logic (including metadata only on the first message for a
      given blob) and some differences (more metadata in the latter).
      
      We enclose the common logic in the new `message::BlobResponseChunk` trait,
      and implement it for both response types.
      
      The intermixing with data is not very natural in the Rust ownership
      context, we try and make it not too painful. There is possibly room
      for improvement in reducing cloning/allocations, perhaps using the
      `Bytes`-oriented variant of `prost`.
      d274bd92
    • Georges Racinet's avatar
      RHGitaly: repository files extraction with metadata · 76f69a84
      Georges Racinet authored
      This provides the base logic to access file data, and a common
      representation for metadata that will be enough to implement
      the `GetBlob` and `GetBlobs` gRPC request, and that we hope to be
      reusable for the likes of `TreeEntry` and `GetTreeEntries`.
      
      The `rhgitaly::message` module will provide the necessary conversions
      to actual Gitaly response types.
      76f69a84
    • Georges Racinet's avatar
      RHGitaly WRITE_BUFFER_SIZE lazy static · 93c0c47c
      Georges Racinet authored
      Same as in `hgitaly.streaming`, and ultimately as in Gitaly,
      this is used as a constant that can be overridden by environment
      variable.
      93c0c47c
    • Georges Racinet's avatar
  11. Jul 28, 2023
  12. Jul 25, 2023
  13. Jul 29, 2023
  14. Jul 05, 2023
  15. Jun 29, 2023
    • Georges Racinet's avatar
      RHGitaly: CommitService.FindCommit implementation · 6e654158
      Georges Racinet authored
      This is now a direct application of `gitlab_revision_node_prefix()`.
      Perhaps worth of notice is the fact thay we stay purely async until
      the revision is resolved as a `NodePrefix`, and only spawn a thread
      once it is really needed (access to repository `Changelog`).
      6e654158
    • Georges Racinet's avatar
      RHGitaly: message::commit_for_node_prefix · 6b54aa74
      Georges Racinet authored
      This method is fairly simple, but it has the tremendous advantage
      to flatten three possible `RevlogError` into one, considerably
      simplifying error treatment for the caller.
      6b54aa74
    • Georges Racinet's avatar
      RHGitaly: repository loading helper for unary RPC methods · 8b6fdf35
      Georges Racinet authored
      This is in particular useful because it makes the return type of
      the `spawn_blocking` call explicit, a hint that the compiler would
      request even for so simple a method as `FindCommit`.
      
      Like `load_repo_and_stream()` it takes care of converting some errors
      into the proper `Status`, still letting the caller define what happens
      with `RepoSpecError`, which may have to differ from one request to the other.
      8b6fdf35
    • Georges Racinet's avatar
      RHGitaly: comment typo · 12caab4c
      Georges Racinet authored
      12caab4c
  16. Jun 16, 2023
    • Georges Racinet's avatar
      RHGitaly: GitLab revision resolution as node. · 734564f0
      Georges Racinet authored
      The `gitlab_revision_node_prefix()` function will be used in all
      gRPC methods that accept arguments that are single GitLab revisions.
      Even those that take ranges would use it, just less directly.
      
      `CommitService.FindCommit` would be the most straightforward case,
      but there are many more.
      
      Closes #126
      734564f0
    • Georges Racinet's avatar
      RHGitaly: reading the GitLab default branch file · c81b8f78
      Georges Racinet authored
      As noted in comment, this should really be in the `store` subdirectory,
      but that is a fight (data migration) for another day.
      c81b8f78
    • Georges Racinet's avatar
      RHGitaly: utilities to resolve refs as Nodes · 2c80b078
      Georges Racinet authored
      `gitlab::state::lookup_typed_ref_as_node` is the specialization of
      `map_lookup_typed_ref` to return `Node`s, hence it is working on a single
      stream of typed refs, whereas the higher level `gitlab::revision::full_ref_node`
      is the specialization of `map_full_ref` that works for any ref given by
      its full path.
      2c80b078
  17. Jun 15, 2023
    • Georges Racinet's avatar
      RHGitaly: implement RefService.RefExists · cfb11341
      Georges Racinet authored
      Perhaps the simplest useful application of `map_full_ref`.
      Testing provided by Comparison tests.
      
      We have to introduce another wrapper for propper logging of
      the `revision` field (otherwise it is almost impossible to follow, even
      if one knows one's ASCII). This will be needed by many other methods.
      
      Closes #128
      cfb11341
  18. Jun 14, 2023
    • Georges Racinet's avatar
      RHGitaly: gitlab::revision, only with `map_full_ref()` for now. · 5ac1a849
      Georges Racinet authored
      Of course it will grow the equivalenbt of Python's `gitlab_revision_changeset`,
      which is one of the most common helper functions in HGitaly, making
      the `FindCommit` gRPC method almost trivial.
      5ac1a849
    • Georges Racinet's avatar
      RHGitaly: reading keep-arounds state file · 725e8c52
      Georges Racinet authored
      This time, because there is no arbitrary name to consider, hence
      no bytes string, Tokio's `LinesStream` is perfectly suitable for
      our needs.
      725e8c52
    • Georges Racinet's avatar
      RHGitaly: reading GitLab ref files · bcab7028
      Georges Racinet authored
      This follows the file format and conventions defined in the server side
      `heptapod` Python package.
      
      Since everything is async, we actually provide Streams of TypedRef objects,
      and the `map_lookup_typed_ref` for the numerous use cases when one is looking
      for a specific ref.
      
      The file format is very primitive, so all we can do at this point is to
      perform a full scan, but it has room (version preamble) for more advanced
      formats in the future (e.g, append-only persistent hash map or similar).
      
      A preliminary version of this was using Tokio's `LineStream`, but this
      has several drawbacks:
      
      - it is unicode-centric, yielding `String`s
      - we had to implement our own `Stream`, keeping the `LineStream` as
        a field. Hence because `poll_next()` takes `&mut Pin<Self>` we had
        to write a pinning accessor, which is unsafe (see the doc for `std::pin`
        for more details).
      bcab7028
    • Georges Racinet's avatar
      RHGitaly: mapping file not found errors to `None` · 29fc9a2b
      Georges Racinet authored
      When a file contains a list of values, it is a common pattern that
      the absence of the file is equivalent to the list being empty.
      This `io_error_not_found_as_none` will help keeping duplication low
      (even if fairly trivial).
      29fc9a2b
  19. Apr 26, 2023
    • Georges Racinet's avatar
      RHGitaly: avoid logging all metadata · 32b0304d
      Georges Racinet authored
      Logging all invocation metadata is verbose and contains secrets
      (authentication header). Of these, only the `correlation_id` is
      interesting to us at the moment.
      
      To achieve this, we split the service method with an inner method
      taking the inner request and the correlation id. Only the inner
      method is now instrumented.
      
      This is heavy for the implementation.
      We tried not to split, using the `fields` modifier like this:
      
      ```
          #[instrument(skip(self, request), fields(request = request.get_ref()))]
      ```
      
      and it ended up giving "`RepositoryRequest` does not implement tracing::Value`"
      
      The splitting will be good enough for now.
      32b0304d
  20. Apr 13, 2023
Loading