diff --git a/.gitlab/merge_request_templates/Component upgrade.md b/.gitlab/merge_request_templates/Component upgrade.md new file mode 100644 index 0000000000000000000000000000000000000000..b8731859bee4533290a0a3b102d1ac81ebb9f68f_LmdpdGxhYi9tZXJnZV9yZXF1ZXN0X3RlbXBsYXRlcy9Db21wb25lbnQgdXBncmFkZS5tZA== --- /dev/null +++ b/.gitlab/merge_request_templates/Component upgrade.md @@ -0,0 +1,54 @@ +<!-- After merging changes to this template, update the `Default description template for merge requests` --> +<!-- found under Settings - General Merge Requests --> +## What does this MR do? + +<!-- Briefly describe what this MR is about. --> + +%{first_multiline_commit} + +## Related issues + +<!-- Link related issues below. Insert the issue link or reference after the word "Closes" if merging this should automatically close it. --> + +## Checklist + +See [Definition of done](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/CONTRIBUTING.md#definition-of-done). + +For anything in this list which will not be completed, please provide a reason in the MR discussion. + +### Required + +- [ ] MR title and description are up to date, accurate, and descriptive. +- [ ] MR targets the appropriate branch. +- [ ] Component test plan copied into MR description. +- [ ] Latest merged results pipeline is green. +- [ ] When ready for review, MR is labeled "~workflow::ready for review" per the [Distribution MR workflow](https://about.gitlab.com/handbook/engineering/development/enablement/systems/distribution/merge_requests.html). + +#### For GitLab team members + +If you don't have permissions to trigger pipelines for this MR, the reviewer +should trigger these jobs for you during the review process. + +- [ ] The manual `Trigger:ee-package` jobs have a green pipeline running against latest commit. To debug QA failures, refer to [Investigate QA failures](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/enablement-saas-platforms-qe-team/distribution/#investigate-qa-failures). +- [ ] If `config/software` or `config/patches` directories are changed, `build-package-on-all-os` job within the `Trigger:ee-package` downstream pipeline succeeded. +- [ ] If you are changing anything SSL related, the `Trigger:package:fips` manual job within the `Trigger:ee-package` downstream pipeline succeeded. +- [ ] If CI/CD configuration is changed, the branch is pushed to [`dev.gitlab.org`](https://dev.gitlab.org/gitlab/omnibus-gitlab) to confirm regular branch builds aren't broken. + +### Expected (please provide an explanation if not completing) + +- [ ] Test plan indicating conditions for success has been posted and passes. +- [ ] Documentation created or updated. +- [ ] Tests added. +- [ ] Integration tests added to [GitLab QA](https://gitlab.com/gitlab-org/gitlab-qa). +- [ ] Equivalent MR/issue for the [GitLab Chart](https://gitlab.com/gitlab-org/charts/gitlab) opened. +- [ ] Potential values for new configuration settings validated. Formats such as integer `10`, duration `10s`, URI `scheme://user:passwd@host:port` may require quotation or other special handling when rendered in a template and written to a configuration file. + +## Test plan + +<!-- +* Create a test plan file if it does already exist. Refer to +[Upgrading software components](../upgrading-software-component#test-plans) for +details. Consider modifying existing an existing plan to meet new requirements. + +* Copy the contents of the test plan here. +--> diff --git a/doc/development/contribute-to-omnibus-gitlab.md b/doc/development/contribute-to-omnibus-gitlab.md index 5ad1bf2879953bdb11a69005f8ca45ea32a2e6b9_ZG9jL2RldmVsb3BtZW50L2NvbnRyaWJ1dGUtdG8tb21uaWJ1cy1naXRsYWIubWQ=..b8731859bee4533290a0a3b102d1ac81ebb9f68f_ZG9jL2RldmVsb3BtZW50L2NvbnRyaWJ1dGUtdG8tb21uaWJ1cy1naXRsYWIubWQ= 100644 --- a/doc/development/contribute-to-omnibus-gitlab.md +++ b/doc/development/contribute-to-omnibus-gitlab.md @@ -16,6 +16,7 @@ ## Common maintenance tasks +- [Upgrading software components](upgrading-software-components.md) - [Patching upstream software](creating-patches.md) - [Managing PostgreSQL versions](managing-postgresql-versions.md) - [Upgrading the bundled Chef version](upgrading-chef.md) diff --git a/doc/development/upgrading-software-components.md b/doc/development/upgrading-software-components.md new file mode 100644 index 0000000000000000000000000000000000000000..b8731859bee4533290a0a3b102d1ac81ebb9f68f_ZG9jL2RldmVsb3BtZW50L3VwZ3JhZGluZy1zb2Z0d2FyZS1jb21wb25lbnRzLm1k --- /dev/null +++ b/doc/development/upgrading-software-components.md @@ -0,0 +1,143 @@ +--- +stage: Systems +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Upgrading software components + +The Linux package is created from a series of software components, some of which +are developed by GitLab and others which are sourced from free and open source +projects. Software components can be updated individually as new features, bug +fixes, and security vulnerabilities become available. + +Software component upgrades can be risky, especially when non-backwards +compatible changes are made. Considering [Semantic versioning](https://semver.org/)], examining changelogs, and examining release +notes can give a sense of the amount of risk involved in an upgrade. In all +cases, upgrades should be thoroughly tested before merging. + +The [CNG](https://gitlab.com/gitlab-org/build/CNG) project uses some of these +same software components. Components common to both projects should be updated +in both. + +## Types of software components + +There are two types of software components used in the Linux package: + +- External software components +- Internal software components + +### External software components + +External software components source is downloaded directly from an external site +or copied from the +[`omnibus-mirror`](https://gitlab.com/gitlab-org/build/omnibus-mirror) repository. +A component can be provided using a `git clone`, extracting from a source +tarball, performing a `gem install` (for Ruby modules), or performing a `pip +install` (for Python modules). + +### Internal software components + +Internal software component are developed by GitLab and external contributors. +Source for internal software components is downloaded from a project's GitLab +repository. Versions used in a build are set by the Git reference contained in +the project's corresponding `*VERSION` files. These versions can be overridden +by environment variables. For more information, see +[Use specific branches or versions of a GitLab component](../build/team_member_docs.md#use-specific-branches-or-versions-of-a-gitlab-component). + +Updates to internal software components are done by merge requests in the +corresponding repository. + +## Internal software component update workflow + +A typical workflow for updating an internal software component. + +### Create a fork/branch + +External contributors should create a fork of the +[`gitlab-org/omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab) repository. + +Create a new branch from the target branch (usually `master` of the +[`gitlab-org/omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab) repository. + +### Modify `config/software/<component.rb>` + +1. Find the corresponding configure file for the component that you want to + update in the `config/software` directory. For example + `config/software/prometheus.rb` is used for the Prometheus component. + +1. Change the version to the version you want to update to. If applicable, also +change the corresponding `sha256` to the value of the corresponding version +source tarball. + +### Add or modify any required patches + +The new component version may require: + +- Adding new patches. +- Removing existing patches. +- Changing existing patches. + +All patch files go in `config/patches/<component name>`. They are then +referenced in the corresponding `config/software/<component name>.rb` file. +Examples can be found at: + +- [unzip component](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/config/software/unzip.rb) +- [unzip patches](https://gitlab.com/gitlab-org/omnibus-gitlab/-/tree/master/config/patches/unzip). + +### Push branch + +Push the branch to the to the upstream repository. + +### Create merge request (MR) + +Create a merge request using your development branch and the target branch. + +### Build + +Build the Linux package either: + +- Using the CI/CD system. +- Building locally using [Build local Linux package](../build/build_package.md). + +You must build on all architectures using the CI/CD system before a merge +request for an updated software component can be accepted. + +### Test + +Install the resulting Linux package and test the component changes. + +## Testing software component updates + +### Minimum test requirements + +At a minimum, the following tests should be performed when updating a software +component. + +- Perform a successful GitLab Enterprise Edition (EE) build on all supported platforms. +- Run `qa-test` CI/CD test job for both GitLab Enterprise Edition and GitLab Community Edition. +- Install and verify that component version has been upgraded. +- Verify basic functionality of the software component. + +Additional testing is almost always required and varies depending on the +software component. + +### Test plans + +Here are test plans for individual components. They are meant to be copied into +the merge request where their execution can be recorded. + +Not every component is listed here. Please consider creating a merge request to +add one for a component upgrade that you are working on. Use +`test-plans/upgrade-component-testplan-template.md` as a starting point. + +These test plans are not exhaustive. The might need to be supplemented depending +on the degree of change made to the component. Record these additions in the +merge request and consider adding them here. Use the following filename pattern +when creating the test plan file: + +```plaintext +upgrade-<component-name>-testplan.md +``` + +And add a link in `test-plans/index.md`. diff --git a/test-plans/index.md b/test-plans/index.md new file mode 100644 index 0000000000000000000000000000000000000000..b8731859bee4533290a0a3b102d1ac81ebb9f68f_dGVzdC1wbGFucy9pbmRleC5tZA== --- /dev/null +++ b/test-plans/index.md @@ -0,0 +1,9 @@ +# Omnibus GitLab test plans + +## Updating components + +<!-- Keep this list sorted alphabetically. --> +- [golang](upgrade-golang-testplan.md) +- [libre2](upgrade-libre2-testplan.md) +- [Mattermost](upgrade-mattermost-testplan.md) +- [redis](upgrade-redis-testplan.md) diff --git a/test-plans/upgrade-component-testplan-template.md b/test-plans/upgrade-component-testplan-template.md new file mode 100644 index 0000000000000000000000000000000000000000..b8731859bee4533290a0a3b102d1ac81ebb9f68f_dGVzdC1wbGFucy91cGdyYWRlLWNvbXBvbmVudC10ZXN0cGxhbi10ZW1wbGF0ZS5tZA== --- /dev/null +++ b/test-plans/upgrade-component-testplan-template.md @@ -0,0 +1,15 @@ +# `<component name>` component upgrade test plan + +<!-- Copy and paste the following into your MR description. --> +## Test plan + +<!-- - +Add the component test-plan here. Use "[ ]" for each step/task to be completed. + +At a minimum, the following test should be run: + +- [ ] Performed a successful GitLab Enterprise Edition (EE) build on all supported platforms. +- [ ] Ran `qa-test` CI/CD test job for both GitLab Enterprise Edition and GitLab Community Edition. +- [ ] Installed and verified that the component version has been upgraded. +- [ ] Verified basic functionality of the software component. +--> diff --git a/test-plans/upgrade-golang-testplan.md b/test-plans/upgrade-golang-testplan.md new file mode 100644 index 0000000000000000000000000000000000000000..b8731859bee4533290a0a3b102d1ac81ebb9f68f_dGVzdC1wbGFucy91cGdyYWRlLWdvbGFuZy10ZXN0cGxhbi5tZA== --- /dev/null +++ b/test-plans/upgrade-golang-testplan.md @@ -0,0 +1,58 @@ +# `golang` component upgrade test plan + +<!-- Copy and paste the following into your MR description. --> +## Test plan + +- [ ] QA tests passed for FIPS and non FIPS builds +- [ ] Confirmed build was done with go 1.19 `strings /opt/gitlab/embedded/bin/gitaly | grep 'go1\.' | tail -1` +- [ ] Confirmed Omnibus-built services that are owned by distribution are working + - [ ] prometheus - (is scraping metrics) + + ```shell + curl 'localhost:9090/api/v1/query?query=up' + ``` + + - [ ] pgbouncer exporter - (metrics endpoint returns data) + + 1. Configured pgbouncer using https://docs.gitlab.com/ee/administration/postgresql/pgbouncer.html + 1. Run: + + ```shell + curl http://localhost:9188/metrics + ``` + + - [ ] redis-exporter - (metrics endpoint returns data) + + ```shell + curl http://localhost:9121/metrics + ``` + + - [ ] postgres-exporter - (metrics endpoint returns data) + + ```shell + curl http://localhost:9187/metrics + ``` + + - [ ] node-exporter - (metrics endpoint returns data) + + ```shell + curl http://localhost:9100/metrics + ``` + + - [ ] alertmanager - (test trigger an alert) + + 1. Set `prometheus['listen_address'] = '0.0.0.0:9090'` in `/etc/gitlab/gitlab.rb` and run `sudo gitlab-ctl reconfigure`. + 1. Shut down `gitaly` service: + + ```shell + gitlab-ctl stop gitaly + ``` + + 1. Wait 5 minutes and check prometheus console `http://<gitlab host>:9090/alerts?search=` for service down alert. + 1. Start `gitaly` service: + + ```shell + gitlab-ctl start gitaly + ``` + + 1. Wait 5 minutes and check prometheus console `http://<gitlab host>:9090/alerts?search=` for service back up. diff --git a/test-plans/upgrade-libre2-testplan.md b/test-plans/upgrade-libre2-testplan.md new file mode 100644 index 0000000000000000000000000000000000000000..b8731859bee4533290a0a3b102d1ac81ebb9f68f_dGVzdC1wbGFucy91cGdyYWRlLWxpYnJlMi10ZXN0cGxhbi5tZA== --- /dev/null +++ b/test-plans/upgrade-libre2-testplan.md @@ -0,0 +1,42 @@ +# libre2 component upgrade test plan + +## Test plan + +- [ ] Green pipeline on gitlab.com including `Trigger:ce-package` and `Trigger:ee-package`. These pipelines should have ran their respective `build-package-on-all-os` pipelines. +- [ ] Verified build options for CC and CXX are correct. + -[ ] CentOS 7 + -[ ] Modern OS, e.g Ubuntu jammy. +- [ ] Installed package or container. Verified no installation issues. + -[ ] CentOS 7 + -[ ] Modern OS +- Ran ldd on re2 gem and verified correct `libre2`` library is used. + -[ ] CentOS 7 + -[ ] Modern OS +- [ ] Ran `re2` Spec test to verify correct ruby integration: + + ```shell + docker run -it registry.gitlab.com/gitlab-org/gitlab-omnibus-builder/debian_10:4.10.0 bash + cd /tmp + git clone -b 2023-03-01 https://github.com/google/re2 google-re2 + git clone -b v1.6.0 https://github.com/mudge/re2.git + cd google-re2 + make install + cd .. + cd re2 + bundle install + bundle exec rake compile + bundle exec rspec + ``` + +- [ ] Ran gitlab-rails console smoke test + - [ ] CentOS 7 + - [ ] Modern OS + + ```ruby + irb(main):005:0> regex = RE2('\(iP.+; CPU .*OS (\d+)[_\d]*.*\) AppleWebKit\/') + => #<RE2::Regexp /\(iP.+; CPU .*OS (\d+)[_\d]*.*\) AppleWebKit\//> + irb(main):006:0> regex.match?('foo') + => false + irb(main):017:0> regex.match?("Mozilla/5.0 (iPhone; CPU iPhone OS 12_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1") + => true + ``` diff --git a/test-plans/upgrade-mattermost-testplan.md b/test-plans/upgrade-mattermost-testplan.md new file mode 100644 index 0000000000000000000000000000000000000000..b8731859bee4533290a0a3b102d1ac81ebb9f68f_dGVzdC1wbGFucy91cGdyYWRlLW1hdHRlcm1vc3QtdGVzdHBsYW4ubWQ= --- /dev/null +++ b/test-plans/upgrade-mattermost-testplan.md @@ -0,0 +1,116 @@ +# Mattermost component upgrade test plan + +<!-- Copy and paste the following into your MR description. --> +## Test plan + +### Build tests + +- [ ] Built on all supported platforms +- [ ] Ran `Trigger:ee-package` and then `qa-test` CI jobs on `gitlab.com`. + +### Fresh installation tests + +Installed a Linux package created in the build system on a fresh OS installation +and ran the following actions, checks, and tests: + +- [ ] Installed Linux package with the new Mattermost version: + + - [ ] Verified that package installation logs/output shows no errors. + - [ ] Verified Mattermost version by running `/opt/gitlab/embedded/bin/mattermost version`. + +- [ ] Edited `/etc/gitlab/gitlab.rb` and set: + + - `external_url` + - `mattermost_external_url` + + Both URLs should point to the same system so that GitLab and Mattermost are co-located. Example: + + ```yaml + external_url 'gitlab.example.com' + mattermost_external_url 'mattermost.example.com' + ``` + +- [ ] Ran `gitlab-ctl reconfigure`. +- [ ] Connected to `gitlab.example.com`. +- [ ] Navigated to `Admin>Settings>Network>Outbound requests` and added `mattermost.example.com` to `Local IP addresses and domain names that hooks and services can accesss` and clicked `Save changes`. +- [ ] Navigated to `mattermost.example.com`. +- [ ] Verified that single-sign on using GitLab credentials was working: + + 1. Clicked on `or create and account with` GitLab. + 1. When presented with the `Authorize GitLab Mattermost to use your account?` page, clicked on `Authorize`. + You should have landed on the `Select teams` page. + +- [ ] Verified that when creating a group in GitLab, checking the box for **Create a Mattermost team for this group** also created a team in Mattermost and the GitLab user is a member of that team. +- [ ] Created a test project within the group created in the previous step and initialize with a `README`. + +- [ ] Verified Mattermost slash command operation: + - [ ] Enabled slash commands using [GitLab documentation](https://docs.gitlab.com/ee/user/project/integrations/mattermost_slash_commands.html#configure-automatically). + - [ ] Tested slash commands by creating a new issue from the Mattermost instance. After following the prompt to re-authorize, the issue should have been successfully created in GitLab. + +- [ ] Verified GitLab issue notification. + + - [ ] Configured incoming web hooks in Mattermost using the [GitLab Documentation](https://docs.gitlab.com/ee/user/project/integrations/mattermost.html). Note that you have to configure both Mattermost and GitLab. + - [ ] Using the created web hook, followed the documentation for adding [notification support](https://docs.gitlab.com/ee/user/project/integrations/mattermost.html#configure-mattermost-to-receive-gitlab-notifications). + - [ ] Created an issue in the test project. Verified that the notification for the issue appeared in Mattermost for the GitLab user. + +### Upgrade installation tests + +Install a Linux package from the previous, latest minor number release on a +fresh OS installation. Run the following actions, checks, and tests: + +- [ ] Installed Linux package from the previous, latest minor number release: + + - [ ] Verified that package installation logs/output showed no errors. + - [ ] Verified Mattermost version by running `/opt/gitlab/embedded/bin/mattermost version`. + +- [ ] Edited `/etc/gitlab/gitlab.rb` and set: + + - `external_url` + - `mattermost_external_url` + + Both URLs should point to the same system so that GitLab and Mattermost are co-located. Example: + + ```yaml + external_url 'gitlab.example.com' + mattermost_external_url 'mattermost.example.com' + ``` + +- [ ] Ran `gitlab-ctl reconfigure`. +- [ ] Connected to `gitlab.example.com`. +- [ ] Navigated to `Admin>Settings>Network>Outbound requests` and added `mattermost.example.com` to `Local IP addresses and domain names that hooks and services can accesss` and clicked `Save changes`. +- [ ] Navigated to `mattermost.example.com`. +- [ ] Verified that single-sign on using GitLab credentials is working: + + 1. Clicked on `or create and account with` GitLab. + 1. When presented with the `Authorize GitLab Mattermost to use your account?` page, clicked on `Authorize`. + 1. You should have landed on the `Select teams` page. + +- [ ] Verified that when creating a group in GitLab, checking the box for **Create a Mattermost team for this group** also created a team in Mattermost and the GitLab user is a member of that team. +- [ ] Created a test project within the group created in the previous step and initialized with a `README`. + +- [ ] Verified Mattermost slash command operation: + + - [ ] Enabled slash commands using [GitLab documentation](https://docs.gitlab.com/ee/user/project/integrations/mattermost_slash_commands.html#configure-automatically). + - [ ] Tested slash commands by creating a new issue from the Mattermost instance. After following the prompt to re-authorize, the issue should have been successfully created in GitLab. + +- [ ] Verified GitLab issue notification. + + - [ ] Configured incoming web hooks in Mattermost using the [GitLab Documentation](https://docs.gitlab.com/ee/user/project/integrations/mattermost.html). Note that you have to configure both Mattermost and GitLab. + - [ ] Using the created web hook, followed the documentation for adding [notification support](https://docs.gitlab.com/ee/user/project/integrations/mattermost.html#configure-mattermost-to-receive-gitlab-notifications). + - [ ] Created an issue in the test project. Verified that the notification for the issue appears in Mattermost for the GitLab user. + +Upgrade GitLab with a Linux package created with the new Mattermost version. Run the +following actions, checks, and tests: + +- [ ] Upgraded to package with new Mattermost version: + + - [ ] Verified that package installation logs/output shows no errors. + - [ ] Verified Mattermost version by running `/opt/gitlab/embedded/bin/mattermost version`. + +- [ ] Verified Mattermost slash command operation: + + - [ ] Tested slash commands by creating a new issue from the Mattermost instance. After following the prompt to re-authorize, the issue should have been successfully created in GitLab. + +- [ ] Verified GitLab issue notification. + + - [ ] Created an issue in the test project. Verified that the notification for the issue appeared in Mattermost for the GitLab user. diff --git a/test-plans/upgrade-redis-testplan.md b/test-plans/upgrade-redis-testplan.md new file mode 100644 index 0000000000000000000000000000000000000000..b8731859bee4533290a0a3b102d1ac81ebb9f68f_dGVzdC1wbGFucy91cGdyYWRlLXJlZGlzLXRlc3RwbGFuLm1k --- /dev/null +++ b/test-plans/upgrade-redis-testplan.md @@ -0,0 +1,56 @@ +# redis component upgrade test plan + +<!-- Copy and paste the following into your MR description. --> +## Test plan + +- [ ] Performed a successful GitLab Enterprise Edition (EE) build on all supported platforms (include `build-package-on-all-os` job). +- [ ] Ran `qa-test` CI/CD test job for both GitLab Enterprise Edition and GitLab Community Edition. +- [ ] Redis indicator test cases were not failing. + - [realtime components via assignee test](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347941) + - [project template import](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347932) +- [ ] Performed fresh install and verified operation: + - [ ] Installed on single node + - [ ] Checked installed version: + + ```shell + /opt/gitlab/embedded/bin/redis-server --version + ``` + + - [ ] Sign-in + - [ ] Imported a project (confirms Sidekiq works) + - [ ] Checked `gitlab-kas` log for Redis errors + - [ ] Checked `redis-exporter` logs for errors + - [ ] Confirmed `gitlab-redis-cli` command connects to Redis + - [ ] Installed on HA Sentinels (see https://docs.gitlab.com/ee/administration/redis/replication_and_failover.html) + - [ ] Signed-in + - [ ] Imported a project (confirms Sidekiq works) + - [ ] Checked `gitlab-kas` log for Redis errors + - [ ] Performed failover + +- [ ] Updates + - [ ] Updated on single node: + - [ ] Verified that you get the message: + + ```plaintext + Running version of Redis different than installed version. Restart redis" + ``` + + - [ ] Sign-in + - [ ] Imported a project (confirms Sidekiq works) + - [ ] Checked `gitlab-kas log` for Redis errors + - [ ] Checked `redis-exporter logs` for errors + - [ ] Confirmed `gitlab-redis-cli` command connects to Redis + + - [ ] Updated HA Sentinel/Redis nodes (see https://docs.gitlab.com/ee/update/zero_downtime.html#redis-ha-using-sentinel) + - [ ] Verified that you get the message: + + ```plaintext + Running version of Redis different than installed version. Restart redis" + ``` + + - [ ] Signed-in + - [ ] Imported a project (confirms Sidekiq works) + - [ ] Checked `gitlab-kas` log for Redis errors + - [ ] Checked `redis-exporter` logs for errors + - [ ] Confirmed `gitlab-redis-cli` command connects to Redis on a Redis node + - [ ] Performed failover