1. 02 Nov, 2020 1 commit
  2. 25 Nov, 2020 2 commits
  3. 18 Nov, 2020 2 commits
  4. 16 Nov, 2020 1 commit
  5. 17 Nov, 2020 2 commits
  6. 16 Nov, 2020 1 commit
    • Georges Racinet's avatar
      Branch acces checks: direct identification of legit merges · 948dac54b949
      Georges Racinet authored
      Closes #368
      To let merges done from the Rails application be checked
      against the distinct set of merge rules rather that push
      rules, GitLab looks for MRs matching the pre-receive changes by
      comparing with the merge commit sha that's always available
      (and set by the Repository Model) before the actual branch move.
      Because the Mercurial change can be atomic, we have no other
      option than passing a special parameter down from
      `Repository#merge` through the Mercurial backend, with the
      `HEPTAPOD_ACCEPT_MERGE_ID` environment variable.
      In turn the Mercurial process will add it to its pre-receive
      hook payload, and the target branch can be compared to switch
      from (direct) push to (Rails) merge mode.
      In the case of merges intended from the very beginning to be
      fast forwards, the commit sha is already known in advance,
      but we're using the same mechanism for consistency in the
      Finally, there was a good dose of wrestling with those RSpec
      examples that care about merging to a protected branch, because
      of the on-demand nature of instantiation.
      branch : heptapod-stable
  7. 15 Nov, 2020 1 commit
  8. 09 Nov, 2020 2 commits
    • Georges Racinet's avatar
      HGitayRepository: fixing Diff invocations · 92500bee7f38
      Georges Racinet authored
      Although they go trough `gitaly_commit_client` they actually
      call the `DiffService`, not currently implemented by HGitaly at
      all (obscure generic NoMethod gRPC error, hard to track which
      caller it was).
      We need to
      - use @git_storage to call the regular Gitaly
      - convert SHAs to Git SHAs
      This gives us our first working Commit page in the Web UI, but only
      for Commits without parents. In the (frequent, yes) general case,
      we still need at least BatchByOids to display the Commit parents list
      (must go partly through Markdown rendering). This last part will be
      done after 0.17.0rc1
      branch : heptapod
    • Georges Racinet's avatar
      HGitalyRepository: fix cached_exists? and conversions for vcs-cloned · b188426a2147
      Georges Racinet authored
      The `#cached_exists` method can issue a Gitaly call. Of course if
      it's running for a temporary vcs-cloned as `GitalyCommitClient` uses,
      that makes for double VCS prefixing and it fails.
      We correct this, and hence its various conversion callers by
      subclassing and doing the right thing explicitely.
      branch : heptapod
  9. 08 Nov, 2020 2 commits
    • Georges Racinet's avatar
      HGitalyRepository: Git-based list_last_commit_for_tree · e77cd6a5f21f
      Georges Racinet authored
      Arguably, this method has nothing to do with Trees – it
      uses paths, not actual Git Tree objects and could therefore
      be implemented directly in HGitaly.
      To be frank, if we had come to that conclusion earlier we
      probably would have made it that way, but now that this code
      is there, we can as well turn to other pressing matters.
      A consequence is the `hpd_back_convert!` method, but it could
      turn out to be useful for something else. Of course, like its
      cousins in the `Tree` and `Blob` class, we'd prefer not to
      need it at all.
      branch : heptapod
    • Georges Racinet's avatar
      Repository classes: gitsha methods, preparing for HGitaly1 · 312149ad6346
      Georges Racinet authored
      For the integration of native Mercurial repositories to reach
      milestone HGitaly1 (mixed mode still relying on Git conversion for
      some data access), we will need to be able to convert internally
      to Git SHAs. Of course, this conversion is trivial for
      `Git::Repository` and `Mercurial::HgGitRepository`.
      Various classes in the `Gitlab::Git` module will make use of
      these methods, `Blob` and `Tree` being the first to come to mind.
      branch : heptapod
  10. 07 Nov, 2020 2 commits
    • Georges Racinet's avatar
      gitaly_client: plugging in HGitaly in CommitService · d3f9a208d676
      Georges Racinet authored
      This is more complicated than what we've done earlier for
      - Upstream passes down storage to `GitalyClient.call` as
        `@repository.storage`. We don't want to change all methods
        of CommitService because it wouldn't apply readily to new
        methods from upstream.
        Hence we actually re-instantiate `@repository`,
        with the VCS-qualified storage, with the new `vcs_storage_clone`
        method. On the side, this also requires a small change
        in HgGitRepository to make the value admissible.
      - only a subset of Commit service methods are actually involved
        in our first round of HGitaly implementation (getting rid of
        Git SHAs, see HGitaly1 milestone). We're deciding here that
        the other methods will use directly the original storage name,
        that we store as a new instance variable, `@git_storage`.
        In particular, we're trying to avoid the implementation of the
        various methods for Commit content (files, trees, diffs…) for
        the HGitaly1 milestone. This part is WIP (we'll need to
        convert back to Git revisions to complete it).
      - Finally, this hack also requires to make sure that the resulting
        Commit instances refer to the original Repository, so that further
        (H)Gitaly calls done from these Commits don't reuse a VCS-qualified
        storage, which would defeat the very purpose of separation.
      This is all very hacky. The only thing that makes it admissible is
      that it is temporary scaffolding. For the HGitaly2 milestone, all
      methods will go to HGitaly, and perhaps we can convince upstream to
      use a `@storage` instance variable here for the sake of uniformity.
      branch : heptapod
    • Georges Racinet's avatar
      HGitalyRepository: comments about SHA/ID conversion methods · 3639ef277c7e
      Georges Racinet authored
      branch : heptapod
  11. 30 Oct, 2020 1 commit
  12. 29 Oct, 2020 1 commit
  13. 25 Oct, 2020 3 commits
    • Georges Racinet on mutations.racinet.fr's avatar
      Commit: introducing hpd_display_id · b10c60a26265
      In some cases it makes sense to display a different commit ID than the
      one used for addressing. Of course our existing `HgGitRepository` is
      a prime example of that, but it's not hard to imagine other ones.
      Up to now, we've changed some templates and user-facing to code to
      use directly some 'hgid'. For dual VCS support, as well as intermediate
      steps towards Mercurial native support, we'll need something more
      abstract than that.
      The method name, `hpd_display_id`, is voluntary obnoxious: we want
      to recognize at first glance that it's Heptapod-specific, and we want
      it to be easy to find all occurrences in the code base, in preparation
      for potential future times when we won't be needing it any more.
      branch : heptapod
    • Georges Racinet on mutations.racinet.fr's avatar
      archive_sha: fixing HGitalyRepository · e457db6b841d
      The position of the overriding method was just wrong. This went
      unnoticed because the Mercurial archive feature is for now tested
      with HgGitRepository only.
      branch : heptapod
    • Georges Racinet on mutations.racinet.fr's avatar
      archive_sha: tests for Git::Repository and Mercurial::HgGitRepository · ab8bb59acad1
      We're also using the `override` helper, which makes sure that there
      is a method with the same name to be overridden, hence providing
      protection in the case of renames in the base class.
      We don't provide tests in the HGitalyRepository case in this commit:
      this will be the point of the next one.
      branch : heptapod
  14. 19 Oct, 2020 9 commits
  15. 18 Oct, 2020 1 commit
    • Georges Racinet's avatar
      HgGitRepository#hg_pull · 667641b1a538
      Georges Racinet authored
      This will be an interesting low-level helper, being exposed
      on the `Repository` model automatically, like all "raw"
      repository methods.
      Like all write methods, a valid `user` argument must be passed.
      branch : heptapod
  16. 31 Oct, 2020 1 commit
  17. 14 Oct, 2020 1 commit
    • Georges Racinet's avatar
      Repository archives: dispatching to HGitaly for Mercurial repos · 2e2da0334c73
      Georges Racinet authored
      For archive production, the Rails app duty is to return
      direct connection information to Workhorse, so that it can call
      Gitaly directly.
      We're reusing the dispatching logic for our existing (yet unplugged)
      HGitaly integration: by prefixing the storage (shard) with `hg:` for
      Mercurial repositories, we have the `GitalyClient.address` return
      connection information for the HGitaly server.
      For this to work, we need `archive_metadata` method of the
      Mercurial Repository classes to provide the Mercurial SHA, which we do
      by intoducing the dedicated `archive_sha` method.
      branch : heptapod
  18. 09 Oct, 2020 1 commit
    • Georges Racinet's avatar
      HgGitRepository: low level hg_tag method · 74b64d425d85
      Georges Racinet authored
      This is for now mostly intended as a helper for tests.
      We're using one of the HGitaly tags test to make sure it works,
      and in turn that test gets simplified.
      There's a catch: `hg_git_repository.hg_tag` would not
      work before `subject` has been used at least once, so that the
      repository is indeed initialized on disk.
      If we were in need to use `hg_git_repository` here, we'd have
      to switch to `subject!` instead of just `subject`.
      branch : heptapod
  19. 24 Sep, 2020 2 commits
  20. 14 Sep, 2020 1 commit
    • Georges Racinet's avatar
      Hgitaly: repository class, using the Ref service · d08e4ed93de0
      Georges Racinet authored
      For now, the `HGitalyRepository` inherits from `HgGitRepository`,
      so that we can gradually transition.
      The granularity of the transition is at the service class level,
      and we start the movement by passing the expected `vcs: 'hg'`
      parameter to `GitalyClient::RefService`.
      The RSpec test in turn validates the round-trip with HGitaly,
      as an absolute expected result, as well as a comparision with
      the value returned by `HgGitRepository`.
      This particular example demonstrates the need for such an approach:
      hgitaly!3 was actually needed for the test to pass.
      branch : heptapod
  21. 10 Sep, 2020 1 commit
    • Georges Racinet's avatar
      Markdown renderings: lookup for Mercurial short nodes · 31146d7c450d
      Georges Racinet authored
      The `CommitReferenceFilter` class is the one responsible for
      the rendering.
      We're *adding* resolution of truncated Mercurial Node ids for
      the case of the `hg_git` VCS type, relying on the existing
      map of full Mercurial SHA (Node id) to Git SHA.
      This is very inefficient, and proper calls with low startup
      overhead to a persistent Mercurial nodemap would do much better
      above a certain repository size (persistent nodemap is provided
      by Rust native extensions, and low overhead would be achieved only
      by a fastpath without Python or a long-running process, such as
      While the `Git::Commit.batch_by_oid` method
      doesn't seem to be called except from the `#commits_by` method of the
      `Repository` model, the latter is used in several places
      of the application, with risks of breakage and/or severe performance
      degradation. For `hg_git` VCS type, most callers don't need the Mercurial
      resolution. That's why we're executing the new lookup only if
      the new `hg_full_resolution` argument is set to `true`.
      Preliminary performance analysis (non scientific, on my workstation)
      shows that in the 100k changesets range (pypy repo) this naive lookup takes
      about 30ms, whereas `hg log -T` is in the 100-200ms ballpark.
      Around 500k changesets (mozilla-central repo), times have the same order
      of magnitude (around 100-200ms again). To insist, this is with the hg startup
      overhead and without the Rust persistent nodemap (would take less than 1ms).
      We fully expect the current hg<->git maps to be an unbereable performance
      problem in Heptapod anyway around 500k to 1M changesets anyway, only solved
      by HGitaly.
      All in all, the performance question seems to be acceptable in the current
      inefficient context *for Note rendering*.
      HGitaly would provide an efficient RPC call anyway.
      branch : heptapod-stable
  22. 27 Aug, 2020 2 commits
    • Georges Racinet's avatar
      Forwarding actual permissions to Mercurial processes · 4c45dfe84231
      Georges Racinet authored
      Closes #334
      Normally, the abstraction layers in GitLab mean that the Project
      should not be accessible from the low level repository. However,
      we need it to pass it down to the `hg` processes. Doing it in
      `hg_env_for_write` is the way to make it really robust, since all
      write operations basically have to use it.
      To mitigate somewhat this breach of abstraction, we make the
      attribute writable only, and we give it a more specific name than
      just '@project'. Aesthetically, it would be better to store only
      a factory for the `UserAccess` instance that we really need, but
      that could be done separately.
      This required running some of `HgAccess` RSpec tests with empty
      repositories, because in them, the project owner does not have enough
      rights to create a first commit any more, and there's not even
      an admin user around for that.
      branch : heptapod
    • Georges Racinet's avatar
      HgGitRepository: removed duplication · 4fb9222b604b
      Georges Racinet authored
      We still have a deprecated compatibility alias, and the
      rebase tests prove that it works.
      branch : heptapod