1. 16 Jul, 2020 4 commits
    • Georges Racinet's avatar
      HgGitRepository#get_hg_heptapod_config can ignore global conf · 05ed90b0af75
      Georges Racinet authored
      This will be useful for an incremental approach of configuration
      setting, and spares us INI format parsing in Ruby (GitLab does not
      come with a gem for this).
      
      --HG--
      branch : heptapod-stable
      05ed90b0af75
    • Georges Racinet's avatar
      HgGitRepository: make get_hg_heptapod_config work if empty · e70d3a5eb8d7
      Georges Racinet authored
      Because `hg config SECTION` exits with code 1 if no matching
      setting have been found, we previously had an error if this
      method (and the corresponding API endpoint) was called before
      any local setting is set in the `[heptapod]` section.
      
      This can be considered part of #299, too
      
      Also fixed the exception raising, Ruby has its quirks, not the
      same as Python's.
      
      --HG--
      branch : heptapod-stable
      e70d3a5eb8d7
    • Georges Racinet's avatar
      HgGitRepository: don't crash if no managed hgrc exists yet · c23adf095138
      Georges Racinet authored
      Closes #299
      
      We'll return content to be `nil` in that case, which will be
      conveyed as `null` in the JSON API.
      
      --HG--
      branch : heptapod-stable
      c23adf095138
    • Georges Racinet's avatar
      HgGitRepository: taking Mercurial repo size into account · 2d5d3aff9555
      Georges Racinet authored
      Closes #314
      
      Previously only the Git repository was considered.
      
      Note that this is cached in the `projects_statistics` table,
      and can be invalidated by doing
      
      ```
         project.statistics.refresh!
      ```
      
      This seems in turn to be triggered by `UpdateProjectStatisticsWorker`,
      iself called by `ProjectCacheWorker`, from the likes of `PostReceiveWorker`
      etc.
      
      --HG--
      branch : heptapod-stable
      2d5d3aff9555
  2. 14 Jun, 2020 2 commits
    • Georges Racinet's avatar
      Post-receives for repository files and merges with changeset · 376d59c06826
      Georges Racinet authored
      Post-receive notificiations are necessary to trigger pipelines and
      other post branch update actions that must be taken on the GitLab
      side
      
      Previously, we enabled them in merges with changeset performed by the
      Rails app itself, but we had only the publication step send them
      to avoid unnecessary consequences like redundant pipelines, but that
      was wrong: if there's already an added commit on `branch/default`
      a publication doesn't trigger any change on `branch/default`. The
      only post-receive it can send is a topic branch prune.
      
      --HG--
      branch : heptapod-stable
      376d59c06826
    • Georges Racinet's avatar
      Passing vcs_type to GitlabShellWorker and fixing repo removal · 1f9e56d13b01
      Georges Racinet authored
      Because sidekiq workers don't seem to take named arguments, we
      pass it as the last required positional one and convert it to
      named argument in the actual call to `Gitlab::Shell`.
      
      Then it was simpler just to let `Gitlab::Shell#rm_namespace`, the
      only method not taking a `vcs_type` named arg to accept it for
      uniformity.
      
      All call sites for GitlabShellWorker are updated. There aren't so
      many, due to the general deprecation of `Gitlab::Shell`, also a
      reason not to worry about new callers appearing in future versions.
      
      Finally, the `remove` method wasn't even implemented in
      `HgGitRepository`.
      
      --HG--
      branch : heptapod
      1f9e56d13b01
  3. 08 Jun, 2020 1 commit
  4. 05 Jun, 2020 1 commit
    • Georges Racinet's avatar
      Imports: adapted for GitLab 12.10 · 38a64ea242b6
      Georges Racinet authored
      The main code path for imports is no longer in `Gitlab::Shell`.
      Instead, that functionality is provided by the `Repository` objects,
      themselves called rather directly from `Projects::ImportService`.
      
      We're adopting our existing code in `HgGitRepository`. Because that
      is a repository creation, it means we have to passover the Mercurial
      specific arguments as well, and hence make the pure Git version from
      `Git::Repository` accept them.
      
      Finally, this allows us to remove our band-aid executable and its
      underlying lib (`gitlab_project_hg.rb`).
      
      Of course `ImportService` is itself always called from a Worker.
      
      --HG--
      branch : heptapod
      38a64ea242b6
  5. 04 Jun, 2020 3 commits
    • Georges Racinet's avatar
      Rebase: accepting new argument · 529df5c54809
      Georges Racinet authored
      At some point we'll have to investigate these push options and
      what Mercurial can do with them. Ignoring them is good enough
      for our immediate purpose: making our existing Heptapod work
      on GitLab 12.10
      
      --HG--
      branch : heptapod
      529df5c54809
    • Georges Racinet's avatar
      Squash: adapted for API break · 318ead425ecd
      Georges Racinet authored
      The `branch` argument does not exist any more. Not so surprising:
      we weren't using it.
      
      --HG--
      branch : heptapod
      318ead425ecd
    • Georges Racinet's avatar
      HgGitRepository: fixed hg username generation · 027c191d4027
      Georges Racinet authored
      This hg username is used notably as commit author.
      
      Between GitLab 12.3 and 12.10, `committer_hash` was simply
      removed. That seems to be because Gitaly accepts separate
      name and email anyway.
      
      There was one calling site not using the `hg_prepare_user_encoding`.
      In that case, just using it is the solution.
      
      --HG--
      branch : heptapod
      027c191d4027
  6. 02 Jun, 2020 1 commit
    • Georges Racinet's avatar
      Adapted rename/transfer for GitLab 12.10 · 95a24b052d2a
      Georges Racinet authored
      We're following the trend of letting the Repository
      object take care of most of the logic, instead of calling
      separate methods for Git and Hg from the services or
      from `Gitlab::Shell`.
      
      For renames, including migration to hashed storage,
      this works because the fixup for the HGRC namespace inclusion
      can find the path to the namespace from the include line, or
      there's nothing to be fixed. This is what the new
      `rpath_to_namespace_if_hgrc_included` method does.
      
      The end result is certainly suboptimal: the repo HGRC will be
      scanned several times, and the transfers now fixup the inclusion,
      only to change it afterwards. But none of these is a real
      concern for now, the important thing is the concentration of
      the hg/git dispatching in less places than before.
      
      --HG--
      branch : heptapod
      95a24b052d2a
  7. 29 May, 2020 1 commit
  8. 28 May, 2020 2 commits
    • Georges Racinet's avatar
      Squash: honouring squash_in_progress? · cfd7b187b509
      Georges Racinet authored
      This works with the same principle as rebase_in_progress?,
      see parent commit.
      
      --HG--
      branch : heptapod
      cfd7b187b509
    • Georges Racinet's avatar
      Rebase: honouring rebase_in_progress? · d7a0e368784b
      Georges Racinet authored
      We name the temporary working directory according to the given
      rebase_id and the question whether it's in progress boils down
      to the workdir being there.
      
      Ideally, some other kind of persistent storage should be used
      immediately, but we can use this as a better approximation for
      the time being. In upstream GitLab, this is handled by Gitaly,
      whose rebase RPC call receives the `rebase_id` and then answers
      the question. Later on HGitaly will do the same.
      
      --HG--
      branch : heptapod
      d7a0e368784b
  9. 27 May, 2020 1 commit
    • Georges Racinet's avatar
      Mercurial rebase for topics · 5f6de060df79
      Georges Racinet authored
      Closes #280
      
      This is enough for merge requests based on topics, which
      currently would display a 'rebase' button if the method is
      'ff'.
      
      For now we add a hard check that the branch being rebased is
      related to a topic, that makes for a simple revset.
      
      As in the case of squashs, the protection against concurrent
      rebases seems at first sight to be unneeded (the rebase itself
      is atomic enough, and a subsequent would fail because it'd create
      instabilities) but because of #277 we have to be cautious
      with concurrency and avoid potential thundering herds of shares.
      This will be done for both squash and rebase in a follow-up that
      would simply test for presence of a share.
      
      --HG--
      branch : heptapod
      5f6de060df79
  10. 28 May, 2020 2 commits
  11. 23 May, 2020 2 commits
    • Georges Racinet's avatar
      Temporary workdirs: switching to the closure style · 8211911f4bca
      Georges Racinet authored
      This is better style and it makes sense for the method that
      create the temporary working directory to take care of its
      cleanup. Overall, we lower the risk to forget an `ensure`
      statement.
      
      We change the name of the method in the same move because
      
      - it's consistent with what the method does
      - it helps us being sure there isn't a leftover call to
        the old `hg_new_workdir` method.
      
      --HG--
      branch : heptapod
      8211911f4bca
    • Georges Racinet's avatar
      Cleaning up temporary workdirs (shares) if creation fails · ac2102a384f3
      Georges Racinet authored
      Severe leaks of temporary workdirs are reported in #277.
      
      This change removes a potential source, not claiming to be a
      complete solution: if the update to revision failed, it would
      leave a directory behind, that maybe could be large already.
      
      That is because the begin … ensure clauses typically happen
      after the return from `hg_new_workdir`
      
      --HG--
      branch : heptapod
      ac2102a384f3
  12. 12 May, 2020 1 commit
    • Georges Racinet's avatar
      Mercurial::Repository.multi_action normalize exception raise · 6fc21d2a7dcb
      Georges Racinet authored
      Higher level methods in GitLab expect Git::Index::IndexError,
      so we have to give them some. For instance, the API will catch
      these and provide appropriate 400 errors from the message.
      
      We decided to settle for the same messages as provided by Gitlab
      (actually by Gitaly through the `user_commit_files` request of
      the OperationService), even though some are a bit ambiguous,
      especially in the case of renames.
      
      This doesn't change the fact that the hg calls could still fail,
      even though many cases are covered by the a priori checks. We'll
      still raise HgError for unexpected failures.
      
      --HG--
      branch : heptapod
      6fc21d2a7dcb
  13. 11 May, 2020 7 commits
    • Georges Racinet's avatar
      Mercurial::Repository.multi_action distinction create/update · 52e78e85ca23
      Georges Racinet authored
      What they are supposed to do is written in comment, and
      actually obtained by looking at the Git implementation.
      
      Notably, subdirectory creation is implicit in create, etc.
      
      --HG--
      branch : heptapod
      52e78e85ca23
    • Georges Racinet's avatar
      Mercurial::Repository.multi_action start_branch_name defaulting · 6a19a5f00cd0
      Georges Racinet authored
      Up to now, our implementation behaved as if it were actually
      required, but some callers (e.g., Wikis::CreateAttachmentService)
      don't pass it, and defaulting to `branch_name` is the right way
      to go.
      
      --HG--
      branch : heptapod
      6a19a5f00cd0
    • Georges Racinet's avatar
      Mercurial::Repository.multi_action create_dir · 701a767a9316
      Georges Racinet authored
      It had previously not been implemented
      
      --HG--
      branch : heptapod
      701a767a9316
    • Georges Racinet's avatar
      Mercurial::Repository.multi_action logging tweaks · e7f932486f0e
      Georges Racinet authored
      Since the content is part of the action Hash, it's important
      to avoid it. Name and path will be enough.
      
      Also, logging `share_path` is useful to debug (before it
      is destroyed, obviously)
      
      --HG--
      branch : heptapod
      e7f932486f0e
    • Georges Racinet's avatar
      Mercurial::Repository multi_action refactor for FS independence · 52136082f11a
      Georges Racinet authored
      The contract for multi_action is that incoming path are specified
      the "web" way (as obvious e.g., in Wiki operations), meaning with
      forward slash as separator.
      
      In the Git case for GitLab 12 they are passed over as-is to Gitaly.
      In our local Hg implementation, we should now convert with `File.join`
      
      The odds of using Heptapod on Windows are low, but it's good hygiene.
      Also, factorizing in the `file_path` and `abspath` variables for
      further developments (separation of `create` and `update`, notably).
      
      --HG--
      branch : heptapod
      52136082f11a
    • Georges Racinet's avatar
      Mercurial::Repository Fixed writing of binary files · ce44e19166a7
      Georges Racinet authored
      This is used, e.g, in Wiki attachments.
      With `File.write` (implicitely text), the binary string
      would give a conversion error to Encoding.default_external
      (usually UTF-8) whereas it goes through in `wb` mode, and I
      checked that actual unicode content is well supported.
      
      --HG--
      branch : heptapod
      ce44e19166a7
    • Georges Racinet's avatar
      Mercurial::Repository using logprefix in multi_action · 1347a8595bdf
      Georges Racinet authored
      The whole point of this prefix is to make logs understandable.
      The mere fact that `Rails.logger` is deprecated in GitLab will
      be adressed another day.
      
      --HG--
      branch : heptapod
      1347a8595bdf
  14. 09 May, 2020 6 commits
  15. 08 May, 2020 1 commit
    • Georges Racinet's avatar
      Gitlab::Mercurial adopted method for repo creation · d9eac4fb23dd
      Georges Racinet authored
      The main advantage is that this looks more uniform from the
      upper layers, at the price of making
      `Git::Repository.create_repository` accept arbitrary
      arguments.
      
      Also `Gitlab::Shell` is slowly being faded away. By
      calling the raw repo's `create_repository` we are
      actually anticipating on later versions.
      
      The saner structure can be also be felt in that we
      don't need to give some methods a classmethod
      version any more.
      
      --HG--
      branch : heptapod
      d9eac4fb23dd
  16. 01 May, 2020 1 commit
  17. 30 Apr, 2020 2 commits
    • Georges Racinet's avatar
      Gitlab::Mercurial moved highest level changeset methods to HgGitRepository · 566aed1f5eae
      Georges Racinet authored
      This is enough to remove the dispatching in the Repository model
      class, and keeps the call flow in a single direction: the parent
      Git::Repositroy never calls a method of HgGitRepository
      
      --HG--
      branch : heptapod
      566aed1f5eae
    • Georges Racinet's avatar
      Gitlab::Mercurial started using a dedicated repo class · 972ed4e01239
      Georges Racinet authored
      We're calling it HgGitRepository to keep open the future
      possibility to have also a HGitalyRepository later on.
      Although it's possible we won't be needing it.
      
      This changeset defines the class and calls it from
      the Repository *model*.
      
      A final remark for people not familiar with Rails: it's
      absolutely necessary to respect the file/class convention.
      Otherwise, everything will look fine for the main app in development
      mode, but the class may not be loaded in some other cases, notably
      from rake in production mode (if mode is even relevant to rake)
      
      --HG--
      branch : heptapod
      972ed4e01239