1. 19 Feb, 2021 1 commit
    • Georges Racinet's avatar
      Heptapod: added start/end logging for backups · 1d18e6772e46
      Georges Racinet authored
      On HDK instances, the backup restoration is often really long,
      it's worthwile to know it has started and is still ongoing.
      
      In cases one launches the tests and comes back later, it is also
      useful to notice how long it took (time to purge repositories/ and
      run `rake dev:setup` to start afresh)
      
      Also, an immediate log that the restoration failed does not hurt.
      1d18e6772e46
  2. 18 Feb, 2021 6 commits
  3. 16 Feb, 2021 9 commits
    • Georges Racinet's avatar
      Heptapod: invalidate instance cache according to type and URL · 4104ff18e9be
      Georges Racinet authored
      With the new testing of production servers, the fixed user
      names can differ from an instance (e.g. local, for tests development)
      to another (actual production), leading to errors.
      
      Invalidating based on URL only would not be enough for people that
      actually work on the production mode – by testing it on a local
      instance (typically provided by the HDK).
      
      There already was a invalidation system for the token itself (test
      and recreate if it does not work), but not for the mere existence
      of an user in the registry.
      
      --HG--
      branch : stable
      4104ff18e9be
    • Georges Racinet's avatar
      Testing production servers: activated some runner tests · c7516a71cef9
      Georges Racinet authored
      With the previous change providing insulation by disabling
      shared and group runners on the project, this works right away.
      
      --HG--
      branch : stable
      c7516a71cef9
    • Georges Racinet's avatar
      Projects, Runner tests: disable shared and group runners · 5af24c7192bb
      Georges Racinet authored
      If regular runners are hooked to the instance being tested,
      we must make really sure that our mock runners don't enter a
      race with the regular runners.
      
      Sadly, there's no direct API setting for group runners (UPSTREAM)
      nor is there a way to block surrounding group runners with
      settings in a sub group, so we had to resort to using Selenium
      for this.
      
      It is important to not even attempt to disable group runners this
      way for personal namespaces, because the button is not present,
      leading to an error. It is also faster, because we don't even
      need a webdriver to be initialized in this case.
      
      --HG--
      branch : stable
      5af24c7192bb
    • Georges Racinet's avatar
      Testing production servers: new dedicated SSH key · b89866c6a137
      Georges Racinet authored
      Granted, it was lazy yet somewhat consistent to
      use the same key as for the `root` user of normal tests.
      
      The consequence was a major pain to remove it each time one
      switches from the normal mode to the production server mode
      and back. Of course, that really matters for developers of
      the tests and of the production server mode itself.
      
      --HG--
      branch : stable
      b89866c6a137
    • Georges Racinet's avatar
      requests: fixing the connection pool for HTTPS · 780e14b76d48
      Georges Racinet authored
      The HTTPS pool must be distinct from the HTTP pool,
      presumably because this connection notion already
      includes TLS (by contrast with raw sockets).
      
      Amazingly, heptapod-tests had not been used over
      HTTPS since the first preflight tests of foss.h.n until now.
      
      --HG--
      branch : stable
      780e14b76d48
    • Georges Racinet's avatar
      Testing production servers: nicer error if group owner not specified · 7aeafe8b27d8
      Georges Racinet authored
      This is better than an obscure unexpected `None` down the road.
      Maybe there's even a way to have pytest issue the proper `parser.error`
      call, but this will be good enough for the time being.
      
      --HG--
      branch : stable
      7aeafe8b27d8
    • Georges Racinet's avatar
      Testing production servers: defaulting to topmost group · 3162debaf0f1
      Georges Racinet authored
      This is a bit nicer for users, a reasonable default instead
      of a mandatory group id.
      
      --HG--
      branch : stable
      3162debaf0f1
    • Georges Racinet's avatar
      Testing production servers: group CLI option · 9796ee2d68f1
      Georges Racinet authored
      Up to now, project creations were occurring in the group
      owner personal namespace. With this change, they happen
      in the target group, if specified.
      
      --HG--
      branch : stable
      9796ee2d68f1
    • Georges Racinet's avatar
      Group.api_retrieve(): owner_name optional argument · 54a2aeb10a91
      Georges Racinet authored
      Otherwise, all the API calls (including the initial GET) are
      done with `root`, and can't work in production server mode.
      
      --HG--
      branch : stable
      54a2aeb10a91
  4. 15 Feb, 2021 5 commits
    • Georges Racinet's avatar
      Testing production instances: activated a MR test · dfdf8eae784d
      Georges Racinet authored
      This is non trivial exercise for the background jobs.
      
      --HG--
      branch : stable
      dfdf8eae784d
    • Georges Racinet's avatar
      Testing production servers: working with SSH · c73d68dfb8ca
      Georges Racinet authored
      We've got a first test using SSH to pass.
      This required adaptation of the logic to load SSH keys.
      
      An important new requirement was to remove
      the key from the server at the end of the
      test session (quite contrary to what we
      want to do for speed on development instances), because
      it's just not acceptable for a key that *anyone can use*
      to stay associated to an user on a production server,
      even sandboxed (spam or worse).
      
      A nice side effect is that we will be actually testing the
      actual logic behind key registration (see foss.heptapod.net#97
      for a small incident involving that).
      
      --HG--
      branch : stable
      c73d68dfb8ca
    • Georges Racinet's avatar
      User: non admin SSH key management · f613b1a6371a
      Georges Racinet authored
      For User instances that already have a token, it is now
      possible to manage SSH keys even if the root password is
      unknown to the tests.
      
      --HG--
      branch : stable
      f613b1a6371a
    • Georges Racinet's avatar
      Testing production servers: first test without root · ef0f9ec9f942
      Georges Racinet authored
      We introduce a new "Group owner" CLI option to specify the
      relevant user, with the idea in mind that all test data will
      evntually be created inside this group. For now it is
      in the user personal namespace for simplicity.
      
      The basic idea is to avoid changing the test itself,
      playing on the fixtures. The first test to actually
      work in this mode is the very simple `push_basic` test.
      
      Of course some tests perform more server-side object
      creation, assuming admin powers, and would need deeper
      adaptation, but a good majority of our tests should work
      readily.
      
      To achieve this, we subclass `Heptapod` and replace preparation
      (of the `root` and `test_basic` users) with preparation of the
      given Group owner.
      
      --HG--
      branch : stable
      ef0f9ec9f942
    • Georges Racinet's avatar
      Testing production servers: created option, test selection system · 368dfe7d4fd2
      Georges Racinet authored
      Only those tests that have the `@suitable.prod_server` mark will
      be run in this mode.
      
      Currently validated with the `push_basic` test.
      
      --HG--
      branch : stable
      368dfe7d4fd2
  5. 16 Feb, 2021 1 commit
    • Georges Racinet's avatar
      Heptapod: store user in registry if not creation · 71afa012e004
      Georges Racinet authored
      In the refactor for the users registry, we forgot to store
      the `User` instance for where expected users (only actual example
      is `test_basic`) already present in the database.
      
      Wasn't catched by CI because it runs on a fresh database.
      
      --HG--
      branch : stable
      71afa012e004
  6. 15 Feb, 2021 1 commit
    • Georges Racinet's avatar
      Heptapod: users registry now contains User instances · 6b153b87be71
      Georges Racinet authored
      This solves a discrepancy that has plagued the heptapod-tests
      code base for long.
      
      In the beginning, users where represented as simple dicts, themselves
      kept in a registry: `user` attribute of the `Heptapod` instance.
      Then, more advanced manipulation needs pushed to the creation of
      the `User` class, but the user registry wasn't updated at the time.
      
      The tricky part, explaining why it wasn't made right away is that
      we often start with a partial representation of the user: for instance
      not knowing its numeric id.
      
      Over time some of the test code has been updated to use higher-level
      methods of the `Heptapod` class, such as `get_user_webdriver()`, in
      prevision for the present move, but still, the registry values remained
      at heart just dicts, while lots of needed logic is implemented in
      the `User` class.
      
      Now we'll have `User` instances in the registry. That still leaves us
      with some illogical aspects, like lazy webdriver and token initialization
      indirected through the `Heptapod` class, instead of being transparent
      properties `User`. But from now on, improving on this should be incremental
      and involve identifiers easy to search in the code base.
      
      --HG--
      branch : stable
      6b153b87be71
  7. 13 Feb, 2021 2 commits
  8. 30 Jan, 2021 1 commit
    • Georges Racinet's avatar
      tox: specific testenv for HDK context · 1e31ede8abc0
      Georges Racinet authored
      In the context of the Heptapod Development Kit (HDK), we use a
      wrapper launch script, which gives us the opportunity to inject
      environment variables. Notably we can:
      
      - use the same version as Python Heptapod components (such as
        HGitaly
      - avoid the system defaults. A current use case is that pytest-parallel
        does not support Python 3.9 yet.
      
      With the `HDK_HEPTAPOD_TESTS_BASEPYTHON` environment variable,
      we can define `testenv:hdk` without local modifications to `tox.ini`.
      
      The default `python3` value should help in transition time, even if
      it doesn't seem likely that we'll have wrapper scripts that use the
      `hdk` testenv without providing the environment variable value.
      1e31ede8abc0
  9. 27 Jan, 2021 1 commit
  10. 19 Jan, 2021 1 commit
  11. 10 Jan, 2021 1 commit
  12. 08 Jan, 2021 3 commits
  13. 11 Jan, 2021 1 commit
  14. 20 Dec, 2020 2 commits
    • Georges Racinet's avatar
      python-gitlab: tests for vcs_type and hg commit ids · 24b45f7436f3
      Georges Racinet authored
      The commit ids part is for heptapod#387.
      
      We check in particular that we didn't break python-gitlab's
      view of Git commits: the new `hg_id` and `short_hg_id` entries
      are not even there for them.
      24b45f7436f3
    • Georges Racinet's avatar
      Utilities to run tests with python-gitlab · 2f2562454c1c
      Georges Racinet authored
      The API handle (`gitlab.Gitlab` instance) can be retrieved on
      a `User` instance with `user.python_gitlab_client`).
      
      Since there are several GitLab API client libraries for Python,
      this raises the question whether we should make this
      conditional, and how to make variants if they cannot be installed
      in the same virtualenv (that's probably the case)
      2f2562454c1c
  15. 17 Dec, 2020 1 commit
  16. 16 Dec, 2020 1 commit
  17. 17 Dec, 2020 3 commits