heptapod issueshttps://foss.heptapod.net/heptapod/heptapod/-/issues2024-01-16T15:40:03Zhttps://foss.heptapod.net/heptapod/heptapod/-/issues/1319Mercurial 6.62024-01-16T15:40:03ZGeorges RacinetMercurial 6.6<!---
Use this template for Mercurial and Evolve version bumps.
Please indicate the target versions explicitely.
The task list can also be adapted for other Python dependencies. Differences
would mostly be about the checks to perform.
...<!---
Use this template for Mercurial and Evolve version bumps.
Please indicate the target versions explicitely.
The task list can also be adapted for other Python dependencies. Differences
would mostly be about the checks to perform.
--->
Task list
- [x] bump in default branch of heptapod/ci-images/mercurial>.
Subsequent checks:
+ hg-evolve version
+ versions in stable branches
+ results of downstream (py-heptapod etc) pipeline triggered by publication of the change
- [x] main repo: bump Mercurial and hg-evolve in heptapod/heptapod, with [skip ci], as a draft in `heptapod` branch.
- [x] rebuild heptapod/ci-images/heptapod-base> (it uses head of `heptapod` branch of main repo).
Subsequent checks:
+ hg-evolve version
+ heptapod/ci-images/heptapod-gitaly rebuilt triggered by heptapod/ci-images/heptapod-base pipeline
- [x] trigger CI on main repo: remove `[skip ci]`, assign a topic, create a MR
Subsequent checks:
+ mercurial and evolve versions (look for hpd-versions in any of the RSpec jobs logs)
- [x] MR merged
- [ ] merge in release branch of main repo, run pipeline, including manual `tarball` job
- [x] make intermediate Omnibus release (using manual tarball and SHA256 visible in main repo `tarball` job)
- [x] publish everything if ok up to functional testsHeptapod 0.41https://foss.heptapod.net/heptapod/heptapod/-/issues/1188GitLab v16.32024-01-07T13:52:30ZGeorges RacinetGitLab v16.3EOL on 2023-11-22, could be the final target for Heptapod 0.40EOL on 2023-11-22, could be the final target for Heptapod 0.40Heptapod 0.41https://foss.heptapod.net/heptapod/heptapod/-/issues/1187GitLab v16.22023-11-21T14:54:26ZGeorges RacinetGitLab v16.2EOL on 2023-10-22, could be the final target for Heptapod 0.40EOL on 2023-10-22, could be the final target for Heptapod 0.40Heptapod 0.40https://foss.heptapod.net/heptapod/heptapod/-/issues/1186GitLab v16.12023-10-17T05:00:05ZGeorges RacinetGitLab v16.1Heptapod 0.40https://foss.heptapod.net/heptapod/heptapod/-/issues/1185GitLab v16.02023-10-09T11:37:35ZGeorges RacinetGitLab v16.0Heptapod 0.40https://foss.heptapod.net/heptapod/heptapod/-/issues/867Release branches in binary builds and functional tests2023-09-13T18:33:42ZGeorges RacinetRelease branches in binary builds and functional testsOmnibus Heptapod (omnibus-heptapod>) is currently our only binary build downstream of the Rails app (heptapod/heptapod>), but that may change in the future.
In Omnibus Heptapod We've long had only branches matching the Rails app develop...Omnibus Heptapod (omnibus-heptapod>) is currently our only binary build downstream of the Rails app (heptapod/heptapod>), but that may change in the future.
In Omnibus Heptapod We've long had only branches matching the Rails app development branches: `heptapod`, `heptapod-stable` etc. This forces the CI/CD scripts in the Rails app to infer the development branch name, which cannot be done reliably from commit information in a release branch (e.g., `heptapod-0-38`), hence has to be done using another meta-information: the `HEPTAPOD_SERIES` environment variable, which is set in the main [CI/CD configuration](heptapod-ci.yml) and is a release critical hazard when juggling with branches (stable branches shift).
The reason for this, despite upstream maintaining release branches in Omnibus GitLab, is that we very seldom had to introduce changes in Omnibus Heptapod after a change of upstream GitLab minor version, and even less to make changes between patch versions (upstream or not), hence it was too heavy to maintain release branches in Omnibus Heptapod. However the price to pay is:
- complexity in Rails release scripts and downstream pipeline triggers.
- systematic conflicts when jumping upstream versions in Omnibus Heptapod (usually just upstream version files, such as `GITLAB_SHELL_VERSION`, changelogs, but sometimes worse than that).
- complexity in Omnibus CI/CD configuration (triggering again)
- now that we merge earlier (branching point at rc42 in Omnibus, too), we'll be at greater risk of incompatibilities between upstream versions than when we just merged a final support version.
Note: pipeline triggering is a relatively recent addition, especially from Rails to Omnibus
From Heptapod 0.38 onwards, we'll try the following approach:
- Release branches in Omnibus Heptapod: `heptapod-0-38` etc.
- Heptapod tests will keep its simpler branch names (`default`, `stable`) etc, but we'll introduce release branches in there, too, even if for an empty diff most of the times.
Note: the `HEPTAPOD_SERIES` environment variable will still be needed in the Rails CI/CD configuration, as it governs the branches in CI images. This could be alleviated with out of repository information (static payload in some web site), but that is bad for build reproducibility).Heptapod 0.38https://foss.heptapod.net/heptapod/heptapod/-/issues/821Migrations: squash all Heptapod-specific migrations as in v15.112024-01-21T21:28:17ZGeorges RacinetMigrations: squash all Heptapod-specific migrations as in v15.11Lots of our migrations are now older than the `init_schema` migration. This is very problematic for testing, the latter being non reversible.
We need to squash into `init_schema`, like upstream is doing for instance in 8f0f88b68227Lots of our migrations are now older than the `init_schema` migration. This is very problematic for testing, the latter being non reversible.
We need to squash into `init_schema`, like upstream is doing for instance in 8f0f88b68227Heptapod 0.41.2https://foss.heptapod.net/heptapod/heptapod/-/issues/735Mercurial 6.42023-03-31T23:08:46ZGeorges RacinetMercurial 6.4In turn this requires two compatibilities we had previously postponed:
- hg-evolve 11 (6.3 is the latest supported in 10.5.3)
- hg-git 1.0.2 (not supported in 0.10.x)In turn this requires two compatibilities we had previously postponed:
- hg-evolve 11 (6.3 is the latest supported in 10.5.3)
- hg-git 1.0.2 (not supported in 0.10.x)Heptapod 0.36https://foss.heptapod.net/heptapod/heptapod/-/issues/709Mercurial 6.32022-11-22T10:19:39ZGeorges RacinetMercurial 6.3Time to update!
As noted in hgitaly#117 there is even in 6.3.1 a fix for a performance issue triggered by Heptapod.
Task list
- [x] bump in default branch of heptapod/ci-images/mercurial.
Subsequent checks:
+ hg-evolve version
+...Time to update!
As noted in hgitaly#117 there is even in 6.3.1 a fix for a performance issue triggered by Heptapod.
Task list
- [x] bump in default branch of heptapod/ci-images/mercurial.
Subsequent checks:
+ hg-evolve version
+ versions in stable branches
+ results of downstream (py-heptapod etc) pipeline triggered by publication of the change
- [x] main repo: bump Mercurial and hg-evolve in heptapod/heptapod, with [skip ci], as a draft in `heptapod` branch.
- [x] rebuild heptapod/ci-images/heptapod-base (it uses head of `heptapod` branch of main repo).
Subsequent checks:
+ hg-evolve version
+ heptapod/ci-images/heptapod-gitaly rebuilt triggered by heptapod/ci-images/heptapod-base pipeline
- [x] trigger CI on main repo: remove `[skip ci]`, assign a topic, create a MR
Subsequent checks:
+ mercurial and evolve versions (look for hpd-versions in any of the RSpec jobs logs)
- [x] MR merged
- [x] merge in release branch of main repo, run pipeline, including manual `tarball` job
- [x] make intermediate Omnibus release (using manual tarball and SHA256 visible in main repo `tarball` job)
- [x] publish everything if ok up to functional testsHeptapod 0.34.0https://foss.heptapod.net/heptapod/heptapod/-/issues/699Binary builds and functional tests for upstream merges2022-09-25T14:28:27ZGeorges RacinetBinary builds and functional tests for upstream mergesBecause we're more and more relying on tools to automate the process of merging GitLab upstream changes into Heptapod,
we need a mechanism to validate the resulting Merge Requests more fully, with a Docker image build and functional test...Because we're more and more relying on tools to automate the process of merging GitLab upstream changes into Heptapod,
we need a mechanism to validate the resulting Merge Requests more fully, with a Docker image build and functional tests.
Steps
- [x] trigger the tarball job of Heptapod Rails for MRs with the `Upstream merge` label
- [x] in the tarball job, provide the resulting version as artifacts
- [x] trigger a downstream Omnibus pipeline, [passing down the versions artifacts](https://docs.gitlab.com/ee/ci/pipelines/downstream_pipelines.html#pass-artifacts-to-a-downstream-pipeline)
- [x] in Omnibus, read versions from the artifacts
- [x] make the Omnibus pipeline depend on the results of its downstream Heptapod Tests pipeline
We need the artifacts because the versions information is more complicated that just the content of CI/CD variables.https://foss.heptapod.net/heptapod/heptapod/-/issues/689Direct conversion from GitLab upstream2024-01-20T10:49:11ZGeorges RacinetDirect conversion from GitLab upstreamThe Heptapod Mercurial repositories for the Rails application and Omnibus have always been converted from the upstream Open Source version, a.k.a Community Edition, see theit [marketing guidelines](https://about.gitlab.com/handbook/marke...The Heptapod Mercurial repositories for the Rails application and Omnibus have always been converted from the upstream Open Source version, a.k.a Community Edition, see theit [marketing guidelines](https://about.gitlab.com/handbook/marketing/strategic-marketing/tiers/) for terminology.
Originally, the Community Edition Git repository was upstream of the Enterprise Edition Git repository, but that was changed around version 12.3, with the CE repository [relocated](https://gitlab.com/gitlab-org/gitlab-foss) and now fed with periodic extracts from the [EE repository](https://gitlab.com/gitlab-org/gitlab).
Heptapod kept on converting from the CE repository, with the ill effects described in #594 (bearable) and worse, a garbled ancestry of release branches: instead of having a release (x-y-stable) branch starting over every month from the trunk, they are artifically stacked on top of each other. This causes artificial conflicts when trying to merge upstream changes in Heptapod in advance of the actual releases, and prevents us from following upstream continuously.
We should therefore convert/import directly from the true upstream, implementing ourselves the removal of the EE code, that we obviously haven't rights to redistribute.
Also, rebasing Heptapod on (a conversion of) the upstream trunk would pave the road for semi-automatic update from upstream (automatic merge performed by CI if tests pass).
Task list:
- [x] rebase of the Heptapod development branch onto a changeset equivalent to the conversion of the gca of upstream EE main branch and the 14-10-stable branch (called below the *14.10 branching point*. The corresponding Git commit does not really exist in `gitlab-foss`. We don't want to convert again ancestors of the branching point, because
1. we already have the code, and don't want to create more than 200 000 changesets for no purpose
2. in versions prior to 12.3, there is no reason for the EE code to be in the same separate directories as it is currently. Cutting at 12.3 would be no simpler than at 14.10.
- [x] patched hg-git able to produce the same results as the GitLab exporter of EE to CE
- [x] direct import of the Git EE commits that are not descendents of the branching point but are ancestors of the upstream trunk (branches started before the branching point and merged after, dubbed *anterior merged ramifications*). This is necessary to prevent hg-git importing the whole repository. A fixed list is good enough for the time being. As we are about 2 months after the 14.10 EOL, chances are that no new anterior merged ramifications will appear (contributors being probably asked to rebase on a current commit).
- [x] automated validation: a script to check the release tags we get at the end of the process should have the exact same content as in the `gitlab-foss` upstream repository (we cannot compare intermediate commits). It would be safer to leave the converted changesets in the draft phase until there is a tagged release to compare, but that has strong implications on the development workflow (non-publishing MRs, which are not currently available in Heptapod).
- [x] at least a way to guard against importing ancestors of the branching point (will happen if a new anterior merged ramification appears)
- [x] continuous operation, using the patched hg-git and the automated validation above.
Continuous merge of upstream in Heptapod would be another issue.Heptapod 1.0Georges RacinetGeorges Racinethttps://foss.heptapod.net/heptapod/heptapod/-/issues/666PostgreSQL 132023-10-09T11:38:02ZGeorges RacinetPostgreSQL 13PostgreSQL 13 support appeared in GitLab 14.1.0 (CI jobs and Omnibus binary build, under the `postgresql_new`) moniker.
- [x] switch heptapod/ci-images/heptapod-base> to upstream's `postgresql-13` base image
- [x] change base CI job fro...PostgreSQL 13 support appeared in GitLab 14.1.0 (CI jobs and Omnibus binary build, under the `postgresql_new`) moniker.
- [x] switch heptapod/ci-images/heptapod-base> to upstream's `postgresql-13` base image
- [x] change base CI job from `.pg12` to `.pg13` in `heptapod-ci.yml`
- [x] check that other CI images don't rely on PostgreSQL
- [x] make the standard in Omnibus (will be in advance compared to GitLab, even after 14.10)
- [x] upgrade foss.heptapod.net in some RC deployment
- [x] upgrade heptapod.host in some final version deploymentHeptapod 0.40https://foss.heptapod.net/heptapod/heptapod/-/issues/664GitLab 14.92022-05-31T19:51:24ZGeorges RacinetGitLab 14.9Will be the upstream GitLab version for %"Heptapod 0.31.0"Will be the upstream GitLab version for %"Heptapod 0.31.0"Heptapod 0.31.0https://foss.heptapod.net/heptapod/heptapod/-/issues/651GitLab 14.102022-06-24T09:40:22ZGeorges RacinetGitLab 14.10Prerequisites:
- [x] heptapod#534 (hgitaly-backup)
(created a bit early to keep track of dependencies)Prerequisites:
- [x] heptapod#534 (hgitaly-backup)
(created a bit early to keep track of dependencies)Heptapod 0.32.0https://foss.heptapod.net/heptapod/heptapod/-/issues/633Bump Mercurial to 6.1 for important performance improvement2022-03-03T10:15:01ZGeorges RacinetBump Mercurial to 6.1 for important performance improvementBumping Mercurial is a recurring part of keeping Heptapod up to date, but in this case, we'll do it earlier than usual, for the massive performance improvement of heptapod#632.
- Mercurial 6.1rc0 (with smooth upgrade to 6.1 if possible)...Bumping Mercurial is a recurring part of keeping Heptapod up to date, but in this case, we'll do it earlier than usual, for the massive performance improvement of heptapod#632.
- Mercurial 6.1rc0 (with smooth upgrade to 6.1 if possible)
- hg-evolve 10.5.0
Task list
- [x] bump in default branch of heptapod/ci-images/mercurial.
Subsequent checks:
+ hg-evolve version
+ versions in stable branches
+ results of downstream (py-heptapod etc) pipeline triggered by publication of the change
- [x] main repo: bump Mercurial and hg-evolve in heptapod/heptapod, with [skip ci], as a draft in `heptapod` branch.
- [x] rebuild heptapod/ci-images/heptapod-base (it uses head of `heptapod` branch of main repo).
Subsequent checks:
+ hg-evolve version
+ heptapod/ci-images/heptapod-gitaly rebuilt triggered by heptapod/ci-images/heptapod-base pipeline
- [x] trigger CI on main repo: remove [skip ci] and repush on heptapod/heptapod, launch pipeline manually if needed.
Subsequent checks:
+ mercurial and evolve versions (look for hpd-versions in any of the RSpec jobs logs)
- [x] merge in release branch of main repo, run pipeline, including manual `tarball` job
- [x] make intermediate Omnibus release (using manual tarball and SHA256 visible in main repo `tarball` job)
- [x] publish everything if ok up to functional testsHeptapod 0.30.0https://foss.heptapod.net/heptapod/heptapod/-/issues/615GitLab 14.72022-03-15T11:03:32ZGeorges RacinetGitLab 14.7Not clear if it can be done before EOL of 14.5 on Feb 22nd (Heptapod 0.28)Not clear if it can be done before EOL of 14.5 on Feb 22nd (Heptapod 0.28)Heptapod 0.30.0https://foss.heptapod.net/heptapod/heptapod/-/issues/426GitLab 13.92021-02-22T12:52:44ZGeorges RacinetGitLab 13.9GitLab 13.9.0 is now only 4 days ahead.
We'll try and base Heptapod 0.20.0rc1 on it, so that we can have a week of RC time and the final version shortly after GitLab 13.9.1 (always a week after the .0)GitLab 13.9.0 is now only 4 days ahead.
We'll try and base Heptapod 0.20.0rc1 on it, so that we can have a week of RC time and the final version shortly after GitLab 13.9.1 (always a week after the .0)Heptapod 0.20.0https://foss.heptapod.net/heptapod/heptapod/-/issues/377GitLab 13.62021-01-17T18:55:35ZGeorges RacinetGitLab 13.6Released on 2020-11-22, EOL on 2021-02-22.
Not strictly necessary for Heptapod 0.18 but it'd be nice to regain one more month of slack.Released on 2020-11-22, EOL on 2021-02-22.
Not strictly necessary for Heptapod 0.18 but it'd be nice to regain one more month of slack.Heptapod 0.18.1https://foss.heptapod.net/heptapod/heptapod/-/issues/364Inner GitLab hooks for native Mercurial repositories2020-11-20T00:23:57ZGeorges RacinetInner GitLab hooks for native Mercurial repositoriesFor the initial exposition of native (HGitaly) Mercurial repositories in Heptapod, we will need the inner hooks sent from the Mercurial processes to contain Mercurial Node IDs (SHAs) instead of their Git counterpart.
Since we are a long...For the initial exposition of native (HGitaly) Mercurial repositories in Heptapod, we will need the inner hooks sent from the Mercurial processes to contain Mercurial Node IDs (SHAs) instead of their Git counterpart.
Since we are a long way to replace the existing `hg-git` backed repositories, we'll need it to be conditional. Overall this will all components, except HGitaly itself:
- [x] py-heptapod: introduce a new `heptapod.native` configuration item, defaulting to `False`. If `True`, send Mercurial SHAs in the hooks. In later milestones, the same configuration item will be used to bypass conversion to Git entirely
- [x] py-heptapod: introduce a special HTTP header (e.g., `X-HEPTAPOD-HG-NATIVE` and have the WSGI application interpret it to set the value of `heptapod.native`
- [x] py-heptapod: also react to the `HEPTAPOD_HG_NATIVE` environment variable to set the value of `heptapod.native`
- [ ] Rails: make sure the internal API endpoint used by Shell provides the `vcs_type` information
- [ ] Rails: provide a way to include `vcs_type` information in responses to Workhorse (probably will be a special header, could even be the same one as understood by `py-heptapod`)
- [x] Shell: read `vcs_type` from the internal API payload, and use it to ~~pass `--config heptapod.native` on the command line~~ set `HEPTAPOD_HG_NATIVE` environment variable
- [x] Workhorse: pass the appropriate `X-HEPTAPOD-HG-NATIVE` header to the Mercurial HTTP backend
- [ ] Rails: pass `--config heptapod.native` on the command line or set `HEPTAPOD_HG_NATIVE` environment variable for all direct `hg` calls.HGitaly1: Hg SHAsGeorges RacinetGeorges Racinethttps://foss.heptapod.net/heptapod/heptapod/-/issues/363GitLab 13.52020-12-11T15:02:03ZGeorges RacinetGitLab 13.5Georges RacinetGeorges Racinet