Skip to content
Snippets Groups Projects
  1. Sep 21, 2023
    • Georges Racinet's avatar
      RHGitaly: resynced protocol with general Gitaly protocol · 14ed2552
      Georges Racinet authored
      Parent changeset removed two messages, and previous bump of Gitaly
      protocol had some other changes that don't affect RHGitaly (or introduce new
      options that are not implemented yet), but should be applied for clarity:
      RHGitaly protocol definitions should have all messages and a subset
      of the methods.
      14ed2552
    • Georges Racinet's avatar
      Bump Gitaly protocol and remove FindAll{Tags,Branch}Names methods · 9de6c722
      Georges Racinet authored
      In Gitaly 91b69d050, these deprecated methods are finally removed.
      Of course clients such as the Rails app have been using `patterns` in
      `ListRefs` for a while, which we also demonstrate in the adapted
      `test_ref.test_tags`. These use-cases of `ListRefs` were also already
      tested.
      9de6c722
  2. Sep 20, 2023
    • Georges Racinet's avatar
      CommitService.GetTreeEntries: remove root_oid · eef65d4b
      Georges Racinet authored
      Of course this requires an update of the base image, to get a Gitaly past
      the point when `root_oid` is removed from protocol. We could run locally
      the Comparison tests with 16.0.0, which is simpler to reference than an
      exact commit, so we'll try this.
      
      The length of the first response in `test_compare_get_tree_entries_pagination`
      by Gitaly changes, perhaps because each `TreeEntry` is a bit smaller (we've long
      suspected the streaming logic was based on actual size hints), but we did not
      check that.
      
      Closes #151
      eef65d4b
    • Georges Racinet's avatar
      RefService: remove PackRefs method · 60137290
      Georges Racinet authored
      It is removed from the protocol in Gitaly v16.0, all what we were doing
      is avoiding to return an error to the sender anyway.
      60137290
    • Georges Racinet's avatar
      Merged stable branch into default · a6238b82
      Georges Racinet authored
      a6238b82
  3. Sep 19, 2023
    • Georges Racinet's avatar
      SearchFilesByContent: improving memory efficiency · e40148cc
      Georges Racinet authored
      This iterator-based approach just keeps a small window of up to 3
      lines in RAM, instead of the full list previously obtained with
      splitlines()`).
      
      It uses `collections.deque` (natively implemented), to make sure the
      window will never be reallocated. On the other hand, matching lines
      are allocated afresh, which is expected not to be a problem.
      
      At this point, further improvements should take the form of an implementation
      in RHGitaly.
      e40148cc
  4. Sep 17, 2023
  5. Sep 04, 2023
  6. Aug 31, 2023
    • Georges Racinet's avatar
      GetTreeEntries: testing and fixing some corner cases · 45749dd3
      Georges Racinet authored
      Testing the case of `limit=0` and revision unknown uncovered a
      small difference between HGitaly and Gitaly (RHGitaly was already
      compliant): HGitaly was returning a response, with an empty list
      of entries.
      
      Decided to change `chunked()` to avoid yielding an empty chunk,
      but it turned out that `RepositoryService.SearchFilesByName` was
      actually expected to yield responses with empty lists, hence we
      hade to make it optional.
      
      The check in `CommitService` to avoid empty chunks having become
      default, we had to remove it to kepp coverage.
      45749dd3
  7. Aug 30, 2023
    • Georges Racinet's avatar
      RHGitaly: implement CommitService.GetTreeEntries · 63daae2a
      Georges Racinet authored
      This builds on the previously introduced elements: iterators for the
      three cases, helpers to stream in paginated chunks and activates the
      Gitaly Comparison tests for this method.
      
      The actual call to the iterators look to be identical, but they are
      not, as monomorphisation will give us three different
      `stream_get_tree_entries_from_iterator`.
      63daae2a
    • Georges Racinet's avatar
      rhgitaly::mercurial::DirIteratorWithFlatPaths · aba79850
      Georges Racinet authored
      This is the engine behind the non-recursive case of
      `CommitService.GetTreeEntries`, if `skip_flat_paths` is `false`.
      
      Like the Python reference HGitaly implementation, we are interpreting
      the "flat path" to be the greatest common path of all entries equal or
      inside the given entry (see `hgitaly.manifest` Python module for details about
      this)
      aba79850
  8. Sep 01, 2023
  9. Aug 30, 2023
  10. Aug 31, 2023
    • Georges Racinet's avatar
      rhgitaly::mercurial::DirIteratorWithoutFlatPaths · 74c6bbd8
      Georges Racinet authored
      This iterator will be the core engine of `CommitService.GetTreeEntries` in
      the simplest case: non-recursive without flat paths computation.
      
      The flat paths computation is not as expensive in the Mercurial case as it is
      in the Git case, because we are iterating over the entire manifest anyway (a
      later version might use a binary search to find the starting point of the
      requested directory, but all files within the directory will have to be scanned),
      but not doing it enables this simple implementation: namely we can yield top-level
      directories immediately, and hence to have at most one `TreeEntry` to yield per run
      of the loop.
      74c6bbd8
  11. Aug 30, 2023
    • Georges Racinet's avatar
      rhgitaly::util::common_subpath_split · b4465e96
      Georges Racinet authored
      This utility method finds the greatest common denominator of
      two paths, and helps using the remainder in one of them.
      
      It takes care of the various edge cases (strict equality,
      trailing slashes) and will be used several times in the
      `GetTreeEntries` implementation.
      b4465e96
    • Georges Racinet's avatar
      rhgitaly::mercurial: an iterator over a directory of a manifest · 831f85fc
      Georges Racinet authored
      This will be useful to implement the various cases of `GetTreeEntries`.
      It could also be extended to be used in `ls_path`, but that would
      require treating the special case where the given path is actually a file.
      
      Consider sending this UPSTREAM.
      831f85fc
    • Georges Racinet's avatar
      rhgitaly::streaming::stream_with_pagination · 13486a81
      Georges Racinet authored
      This generic helper in `rhgitaly::streaming` is for the case
      of streamed responses with `repeated` items, the whole stream
      being the current page. In all cases known to us, GitLab derives
      some information from the last item of the last chunk to be
      the `next_cursor`, hence we introduce a trait for the item
      to represent that, and use it in the implementation.
      
      The first use-case will be the implementation of
      `CommitService.GetTreeEntries`, but this also makes
      the likes of `RefService.FindLocalBranches` essentially trivial.
      13486a81
    • Georges Racinet's avatar
      rhgitaly::streaming::stream_chunks · d124dd52
      Georges Racinet authored
      This Rust version of Python `hgitaly.util.chunked` should be useful
      for many gRPC methods. It provide the means to distinguish the first
      response, but does not implement the pagination protocol.
      d124dd52
  12. Aug 18, 2023
  13. Aug 12, 2023
  14. Aug 11, 2023
  15. Aug 09, 2023
  16. Aug 07, 2023
  17. Jul 31, 2023
  18. Aug 07, 2023
    • Georges Racinet's avatar
      rhgitaly::oid: methods for tree handling · eb6763d7
      Georges Racinet authored
      There are for now completely identical to methods for blobs,
      but they could well become different (e.g, different disk caches or
      whatever).
      eb6763d7
    • Georges Racinet's avatar
      rhgitaly::mercurial::ls_path: listing a subset of manifest · 721ac5a4
      Georges Racinet authored
      This will be useful to implement `CommitService.TreeEntry`.
      There is plenty of room for performance improvement, notably:
      
      - the position of the first relevant entry could be found
        by binary search instead of a linear scan
      - we could try and clone less
      
      But it is possible that this would not be really needed, given
      the performance and scalability boost given by RHGitaly over
      HGitaly, before Mercurial manifests start using a more efficient
      layout, making this code obsolete.
      721ac5a4
    • Georges Racinet's avatar
      RHGitaly: allowing to stream TreeEntryResponse from BlobResponseChunk · 87b6970c
      Georges Racinet authored
      We will soon implement `CommitService.TreeEntry`, which has to
      behave like `GetBlobs` on one `RevisionPath` in case it turns out
      the path is a blob.
      87b6970c
  19. Jul 31, 2023
    • Georges Racinet's avatar
      rhgitaly::mercurial: preparation for CommitService.TreeEntry · 562c5d46
      Georges Racinet authored
      We split `lookup_blob()` in two functions, so that
      parts can be reused without reopening changelogs, manifests etc.
      Notably, the implementation of `TreeEntry` will use `read_blob`
      in case it is serving a blob and not a tree: in this case
      the manifest entry has already been found, no need to reopen and
      rescan anything.
      562c5d46
  20. May 03, 2023
    • Georges Racinet's avatar
      CommitService.CommitLanguages implementation with tokei · 77e37a03
      Georges Racinet authored
      This introduces a new binary dependency on the
      [Tokei](https://crates.io/crates/tokei) code statistics tool.
      
      The path to the Tokei executable is configurable
      as `hgitaly.tokei-executable`, defaulting to `tokei`.
      
      Strict comparison with Gitaly is not fully possible, as Tokei
      does not produce the very same results as GitHub Linguist (or
      `go-enry` which is the library Gitaly is currently moving to).
      For instance `robots.txt` is detected as the `robots.txt` language
      by Linguist and as `Plain Text` by tokei. That being said, we
      can compare more frequent results, namely Python and Ruby.
      
      We vendor the `languages.json` file from GitHub Linguist to add the
      color information (not part of Tokei results), with the same defaulting
      logic as Gitaly (see the detailed docstrings), based on a hash of the
      language name. We were lucky to find a programming language (COBOL) to
      base our tests on.
      
      It is important to sort the results in decreasing order, because
      the Rails app considers only the 5 most frequent languages in the
      repository, and does so by truncating at the 5 first elements.
      77e37a03
Loading