1. 13 Dec, 2020 9 commits
    • Georges Racinet's avatar
      Heptapod CI: hosting postgres and redis images on registry.h.n · c98aa6a7cef2
      Georges Racinet authored
      We're hitting rate limits now even for these. Of course long term
      it will be nicer to have transparent local caches, but we don't
      have these for Docker Hub yet.
      
      --HG--
      branch : heptapod
      c98aa6a7cef2
    • Georges Racinet's avatar
      Heptapod CI: running the "extended" Rspec scope on release branches · a8a3f5d7e397
      Georges Racinet authored
      Release branches are the current `heptapod-0-16`, `heptapod-0-17` etc.
      They are essentially the result of merging of development branches
      (`heptapod` or `heptapod-stable`) with the latest upstream patch
      release.
      
      The rationale is that the extended RSpec scope is too heavy for
      regular development work, and even moreso in the near future,
      because we are precisely in the process of re-enabling more RSpec tests.
      
      For now, failures are allowed, so that it's non-blocking,
      but we should come up with a policy to prepare the release branches
      in advance.
      
      It's also possible to trigger the extended job by passing the
      `CI_HEPTAPOD_RSPEC_SCOPE` variable directly from the Web UI.
      Name differs from the environment variable that `heptapod_knapsack.sh`
      interprets because check occurs after the variable has been set with
      the `variable` directive (hence if it were the same, the job would
      always be triggered)
      
      We're also taking care that the regular job doesn't start
      when using the variable to trigger the extended job.
      
      --HG--
      branch : heptapod
      a8a3f5d7e397
    • Georges Racinet's avatar
      Heptapod CI: stop using "manual" for RSpec tests · 69d40ed3d6fd
      Georges Racinet authored
      Manually triggering parallel jobs is done for each of the
      spawned one independently. This isn't very useful, it's much
      more efficient to trigger by a whole pipeline on a given
      branch. Later on we may want to gate via an explicit environment
      variable, but this is good enough for now.
      
      --HG--
      branch : heptapod
      69d40ed3d6fd
    • Georges Racinet's avatar
      BareRepositoryImport: fixing for Git projects · 958932318e12
      Georges Racinet authored
      Not sure what this really does, and perhaps someday it'll be
      useful for Mercurial as well. But right now, and unless our
      functional tests complain, the simplest seems to decide that
      this is Git-specific code, and hence expecting a Git world.
      
      (current concern is having spec/lib/gitlab pass)
      
      --HG--
      branch : heptapod
      958932318e12
    • Georges Racinet's avatar
      Checks::ProjectMoved: applying rubocop rule · 5c256351617f
      Georges Racinet authored
      --HG--
      branch : heptapod
      5c256351617f
    • Georges Racinet's avatar
      RSpec tests: fixed spec/lib/gitlab/checks · 95db1a4943c9
      Georges Racinet authored
      At 25s, and given the importance, these should be considered for
      the regular MR subset.
      
      --HG--
      branch : heptapod
      95db1a4943c9
    • Georges Racinet's avatar
      Bitbucket import: removed most of the code · fdb960fc35b5
      Georges Racinet authored
      There aren't Mercurial repositories to import from Bitbucket
      any more: our divergence with upstream on this is now just a
      drag.
      
      Removal is partial in order not to jeopardize the installation
      of 0.18.0rc1 on foss.h.n (second OAuth, worker queue registration
      etc.). In all cases, behaviour on such configurations will be
      tested and adjusted before release.
      
      --HG--
      branch : heptapod
      fdb960fc35b5
    • Georges Racinet's avatar
      RSpec tests: introducing "scopes" in parallel runs · e3b221981d54
      Georges Racinet authored
      The new `HEPTAPOD_RSPEC_SCOPE` will be used to define several
      subsets of tests to run.
      
      The default is what we are currently
      running in CI for everyday development (Merge Requests on
      `heptapod` and `heptapod-stable` branches).
      
      The 'extended' scope will be used in release preparation, and
      will grow a lot from now on.
      
      --HG--
      branch : heptapod
      e3b221981d54
    • Georges Racinet's avatar
      RSpec parallel tests: exclusion tags · 20544f19c65b
      Georges Racinet authored
      These are the same tags used by upstream in the "unit" RSpec job
      (with a much wider pattern, of which ours is a small fraction).
      
      --HG--
      branch : heptapod
      20544f19c65b
  2. 10 Dec, 2020 1 commit
  3. 11 Dec, 2020 2 commits
  4. 10 Dec, 2020 5 commits
  5. 09 Dec, 2020 1 commit
    • Georges Racinet's avatar
      Merge with GitLab v13.5.4 · 3cc386756f43
      Georges Racinet authored
      This result passed all RSpec and functional tests on my
      dev workstation.
      
      Here follow the most notable changes that will need further
      attention.
      
      The upstream backup code has been entirely refactored, with a
      new separation of concerns that has the Repository class take
      care of the actual repo bundle production and restoration.
      This is a breeze for Heptapod and its hierarchy of repository
      classes. Not only is our specific code clearly delimited and
      short (45 lines, including comments), but it is also more
      future proof because we don't rely on spaggheti-like conditionals
      any more. For instance, the day native projects don't include
      a Git repository any more, we can simply provide appropriate
      methods in HGitalyRepository. First shot implementation passed
      the functional tests right away. It will need RSpec tests, though,
      which are by the way now much easier to write as well.
      
      There's been refactoring around Wikis, probably related to the
      introduction of the Group Wikis. It was enough to pass the
      tests to just delegate the two Heptapod-specific methods needed
      for the raw repository instantiation, but this needs more scrutinity,
      and probably Mercurial options for Group Wikis have to be
      considered.
      
      --HG--
      branch : heptapod
      3cc386756f43
  6. 08 Dec, 2020 14 commits
  7. 06 Dec, 2020 3 commits
    • Georges Racinet's avatar
      Heptapod CI: running RSpec tests in parallel · ddefe30b3c6f
      Georges Racinet authored
      Starting with 6 jobs should give us pipelines around 15 minutes,
      but perhaps we'll need to cache download.heptapod.net
      
      Worth of notice is that for the `parallel` directive, GitLab chose
      the exact environment variables expected by knapsack.
      
      Lately we've needed a tremendous
      amount of retries on one example (#384), that explains
      the very high `RETRIES` setting.
      
      --HG--
      branch : heptapod-stable
      ddefe30b3c6f
    • Georges Racinet's avatar
      HgitalyRepository tests: remove useless includes · 9e6d387532db
      Georges Racinet authored
      These were copied over from other tests and are of no use in
      that one (`Term` is about end-user terms acceptance, just
      to underline how far that is from the concerns of this test).
      
      --HG--
      branch : heptapod-stable
      9e6d387532db
    • Georges Racinet's avatar
      RSpec tests: terminate HGitaly with SIGTERM · c527891b9457
      Georges Racinet authored
      Killing the main process had no other impact on the
      worker processes than to orphan them.
      
      Depends on hgitaly#34 for the wished result.
      
      --HG--
      branch : heptapod-stable
      c527891b9457
  8. 08 Dec, 2020 1 commit
    • Georges Racinet's avatar
      RSpec tests: parallel launch with Knapsack · baa204a5c885
      Georges Racinet authored
      Knapsack is the sharding tool also used by upstream GitLab.
      To balance the shards, it uses a time table (the report file).
      Our main difference with upstream is that we have a checked in
      report file instead of maintaining it from the CI in a S3 bucket.
      
      Updates will have to be done manually. There is a companion
      `heptapod-rspec-parallel` script in the HDK (for lack of a better
      place) that can leverage knapsack to run the tests
      across several entire HDK workspaces.
      
      --HG--
      branch : heptapod-stable
      baa204a5c885
  9. 05 Dec, 2020 4 commits
    • Georges Racinet's avatar
      knapsack: allow report generation if not on CI · f724d7805e7b
      Georges Racinet authored
      First experiences for parallelization of RSpec tests will
      be done on personal workstations. The KNAPSACK_GENERATE_REPORT
      variable should be enough to gate it.
      
      --HG--
      branch : heptapod-stable
      f724d7805e7b
    • Georges Racinet's avatar
      RSpec hg_access_protected_branches: removed lengthy setup · 611aad7f35ed
      Georges Racinet authored
      It turns out that these tests don't need the branches to actually
      exist. This gets the time to run this RSpec file down from 10
      minutes to less than 4 on my development workstation.
      
      That is enough to make sharding meaningful.
      
      Of course that raises the question whether they test anything,
      since some expected Forbidden could actually now be raised for
      the wrong reason. There is the case of the
      `safe_commit_for_new_protected_branch?` where one checks that
      protected branch rules are applied even if the commit actually
      creates the protected branch (through wildcard protection,
      that is indeed possible), but this is not currently covered,
      although it should.
      
      In other words actual coverage is no worse, and we should add
      new tests for the cases that aren't strongly covered,
      (perhaps UPSTREAM).
      
      The empty repo is a special case, but the general factory fixture
      takes care of that.
      
      --HG--
      branch : heptapod-stable
      611aad7f35ed
    • Georges Racinet's avatar
      RSpec tests for hg_access: using an empty repo for main tests · fc0da2f09a87
      Georges Racinet authored
      Thanks to this, the running time goes down from about 3mn to
      1mn and 50s on my system.
      
      The LFS tests are probably irrelevant: the day we activate Mercurial's
      version of LFS, we will certainly have to perform other types of checks.
      But then, they will perhaps serve as reminders that there is
      something to adapt.
      
      --HG--
      branch : heptapod-stable
      fc0da2f09a87
    • Georges Racinet's avatar
      RSpec tests for hg_access: split in two · 84b0544435e7
      Georges Racinet authored
      The tests involving protected branches are likely to have different
      setup needs that the other. The bad news is that it's unbalanced:
      tests for protected branches take 10mn on my system, the others
      `hg_access_main_spec.rb` amount for just 3mn.
      
      --HG--
      branch : heptapod-stable
      84b0544435e7