Skip to content
Snippets Groups Projects
Commit b8731859 authored by Andrew Patterson's avatar Andrew Patterson
Browse files

Added updating software components instructions

Added documentation on how to upgrade and test software components.
parent 5ad1bf28
No related branches found
No related tags found
2 merge requests!109heptapod#1351: making Heptapod 0.41 the new oldstable,!105heptapod#1283: making 0.41 the new stable
<!-- After merging changes to this template, update the `Default description template for merge requests` -->
<!-- found under Settings - General Merge Requests -->
## What does this MR do?
<!-- Briefly describe what this MR is about. -->
%{first_multiline_commit}
## Related issues
<!-- Link related issues below. Insert the issue link or reference after the word "Closes" if merging this should automatically close it. -->
## Checklist
See [Definition of done](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/CONTRIBUTING.md#definition-of-done).
For anything in this list which will not be completed, please provide a reason in the MR discussion.
### Required
- [ ] MR title and description are up to date, accurate, and descriptive.
- [ ] MR targets the appropriate branch.
- [ ] Component test plan copied into MR description.
- [ ] Latest merged results pipeline is green.
- [ ] When ready for review, MR is labeled "~workflow::ready for review" per the [Distribution MR workflow](https://about.gitlab.com/handbook/engineering/development/enablement/systems/distribution/merge_requests.html).
#### For GitLab team members
If you don't have permissions to trigger pipelines for this MR, the reviewer
should trigger these jobs for you during the review process.
- [ ] The manual `Trigger:ee-package` jobs have a green pipeline running against latest commit. To debug QA failures, refer to [Investigate QA failures](https://about.gitlab.com/handbook/engineering/quality/quality-engineering/enablement-saas-platforms-qe-team/distribution/#investigate-qa-failures).
- [ ] If `config/software` or `config/patches` directories are changed, `build-package-on-all-os` job within the `Trigger:ee-package` downstream pipeline succeeded.
- [ ] If you are changing anything SSL related, the `Trigger:package:fips` manual job within the `Trigger:ee-package` downstream pipeline succeeded.
- [ ] If CI/CD configuration is changed, the branch is pushed to [`dev.gitlab.org`](https://dev.gitlab.org/gitlab/omnibus-gitlab) to confirm regular branch builds aren't broken.
### Expected (please provide an explanation if not completing)
- [ ] Test plan indicating conditions for success has been posted and passes.
- [ ] Documentation created or updated.
- [ ] Tests added.
- [ ] Integration tests added to [GitLab QA](https://gitlab.com/gitlab-org/gitlab-qa).
- [ ] Equivalent MR/issue for the [GitLab Chart](https://gitlab.com/gitlab-org/charts/gitlab) opened.
- [ ] Potential values for new configuration settings validated. Formats such as integer `10`, duration `10s`, URI `scheme://user:passwd@host:port` may require quotation or other special handling when rendered in a template and written to a configuration file.
## Test plan
<!--
* Create a test plan file if it does already exist. Refer to
[Upgrading software components](../upgrading-software-component#test-plans) for
details. Consider modifying existing an existing plan to meet new requirements.
* Copy the contents of the test plan here.
-->
......@@ -16,6 +16,7 @@
## Common maintenance tasks
- [Upgrading software components](upgrading-software-components.md)
- [Patching upstream software](creating-patches.md)
- [Managing PostgreSQL versions](managing-postgresql-versions.md)
- [Upgrading the bundled Chef version](upgrading-chef.md)
......
---
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments
---
# Upgrading software components
The Linux package is created from a series of software components, some of which
are developed by GitLab and others which are sourced from free and open source
projects. Software components can be updated individually as new features, bug
fixes, and security vulnerabilities become available.
Software component upgrades can be risky, especially when non-backwards
compatible changes are made. Considering [Semantic versioning](https://semver.org/)], examining changelogs, and examining release
notes can give a sense of the amount of risk involved in an upgrade. In all
cases, upgrades should be thoroughly tested before merging.
The [CNG](https://gitlab.com/gitlab-org/build/CNG) project uses some of these
same software components. Components common to both projects should be updated
in both.
## Types of software components
There are two types of software components used in the Linux package:
- External software components
- Internal software components
### External software components
External software components source is downloaded directly from an external site
or copied from the
[`omnibus-mirror`](https://gitlab.com/gitlab-org/build/omnibus-mirror) repository.
A component can be provided using a `git clone`, extracting from a source
tarball, performing a `gem install` (for Ruby modules), or performing a `pip
install` (for Python modules).
### Internal software components
Internal software component are developed by GitLab and external contributors.
Source for internal software components is downloaded from a project's GitLab
repository. Versions used in a build are set by the Git reference contained in
the project's corresponding `*VERSION` files. These versions can be overridden
by environment variables. For more information, see
[Use specific branches or versions of a GitLab component](../build/team_member_docs.md#use-specific-branches-or-versions-of-a-gitlab-component).
Updates to internal software components are done by merge requests in the
corresponding repository.
## Internal software component update workflow
A typical workflow for updating an internal software component.
### Create a fork/branch
External contributors should create a fork of the
[`gitlab-org/omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab) repository.
Create a new branch from the target branch (usually `master` of the
[`gitlab-org/omnibus-gitlab`](https://gitlab.com/gitlab-org/omnibus-gitlab) repository.
### Modify `config/software/<component.rb>`
1. Find the corresponding configure file for the component that you want to
update in the `config/software` directory. For example
`config/software/prometheus.rb` is used for the Prometheus component.
1. Change the version to the version you want to update to. If applicable, also
change the corresponding `sha256` to the value of the corresponding version
source tarball.
### Add or modify any required patches
The new component version may require:
- Adding new patches.
- Removing existing patches.
- Changing existing patches.
All patch files go in `config/patches/<component name>`. They are then
referenced in the corresponding `config/software/<component name>.rb` file.
Examples can be found at:
- [unzip component](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/config/software/unzip.rb)
- [unzip patches](https://gitlab.com/gitlab-org/omnibus-gitlab/-/tree/master/config/patches/unzip).
### Push branch
Push the branch to the to the upstream repository.
### Create merge request (MR)
Create a merge request using your development branch and the target branch.
### Build
Build the Linux package either:
- Using the CI/CD system.
- Building locally using [Build local Linux package](../build/build_package.md).
You must build on all architectures using the CI/CD system before a merge
request for an updated software component can be accepted.
### Test
Install the resulting Linux package and test the component changes.
## Testing software component updates
### Minimum test requirements
At a minimum, the following tests should be performed when updating a software
component.
- Perform a successful GitLab Enterprise Edition (EE) build on all supported platforms.
- Run `qa-test` CI/CD test job for both GitLab Enterprise Edition and GitLab Community Edition.
- Install and verify that component version has been upgraded.
- Verify basic functionality of the software component.
Additional testing is almost always required and varies depending on the
software component.
### Test plans
Here are test plans for individual components. They are meant to be copied into
the merge request where their execution can be recorded.
Not every component is listed here. Please consider creating a merge request to
add one for a component upgrade that you are working on. Use
`test-plans/upgrade-component-testplan-template.md` as a starting point.
These test plans are not exhaustive. The might need to be supplemented depending
on the degree of change made to the component. Record these additions in the
merge request and consider adding them here. Use the following filename pattern
when creating the test plan file:
```plaintext
upgrade-<component-name>-testplan.md
```
And add a link in `test-plans/index.md`.
# Omnibus GitLab test plans
## Updating components
<!-- Keep this list sorted alphabetically. -->
- [golang](upgrade-golang-testplan.md)
- [libre2](upgrade-libre2-testplan.md)
- [Mattermost](upgrade-mattermost-testplan.md)
- [redis](upgrade-redis-testplan.md)
# `<component name>` component upgrade test plan
<!-- Copy and paste the following into your MR description. -->
## Test plan
<!-- -
Add the component test-plan here. Use "[ ]" for each step/task to be completed.
At a minimum, the following test should be run:
- [ ] Performed a successful GitLab Enterprise Edition (EE) build on all supported platforms.
- [ ] Ran `qa-test` CI/CD test job for both GitLab Enterprise Edition and GitLab Community Edition.
- [ ] Installed and verified that the component version has been upgraded.
- [ ] Verified basic functionality of the software component.
-->
# `golang` component upgrade test plan
<!-- Copy and paste the following into your MR description. -->
## Test plan
- [ ] QA tests passed for FIPS and non FIPS builds
- [ ] Confirmed build was done with go 1.19 `strings /opt/gitlab/embedded/bin/gitaly | grep 'go1\.' | tail -1`
- [ ] Confirmed Omnibus-built services that are owned by distribution are working
- [ ] prometheus - (is scraping metrics)
```shell
curl 'localhost:9090/api/v1/query?query=up'
```
- [ ] pgbouncer exporter - (metrics endpoint returns data)
1. Configured pgbouncer using https://docs.gitlab.com/ee/administration/postgresql/pgbouncer.html
1. Run:
```shell
curl http://localhost:9188/metrics
```
- [ ] redis-exporter - (metrics endpoint returns data)
```shell
curl http://localhost:9121/metrics
```
- [ ] postgres-exporter - (metrics endpoint returns data)
```shell
curl http://localhost:9187/metrics
```
- [ ] node-exporter - (metrics endpoint returns data)
```shell
curl http://localhost:9100/metrics
```
- [ ] alertmanager - (test trigger an alert)
1. Set `prometheus['listen_address'] = '0.0.0.0:9090'` in `/etc/gitlab/gitlab.rb` and run `sudo gitlab-ctl reconfigure`.
1. Shut down `gitaly` service:
```shell
gitlab-ctl stop gitaly
```
1. Wait 5 minutes and check prometheus console `http://<gitlab host>:9090/alerts?search=` for service down alert.
1. Start `gitaly` service:
```shell
gitlab-ctl start gitaly
```
1. Wait 5 minutes and check prometheus console `http://<gitlab host>:9090/alerts?search=` for service back up.
# libre2 component upgrade test plan
## Test plan
- [ ] Green pipeline on gitlab.com including `Trigger:ce-package` and `Trigger:ee-package`. These pipelines should have ran their respective `build-package-on-all-os` pipelines.
- [ ] Verified build options for CC and CXX are correct.
-[ ] CentOS 7
-[ ] Modern OS, e.g Ubuntu jammy.
- [ ] Installed package or container. Verified no installation issues.
-[ ] CentOS 7
-[ ] Modern OS
- Ran ldd on re2 gem and verified correct `libre2`` library is used.
-[ ] CentOS 7
-[ ] Modern OS
- [ ] Ran `re2` Spec test to verify correct ruby integration:
```shell
docker run -it registry.gitlab.com/gitlab-org/gitlab-omnibus-builder/debian_10:4.10.0 bash
cd /tmp
git clone -b 2023-03-01 https://github.com/google/re2 google-re2
git clone -b v1.6.0 https://github.com/mudge/re2.git
cd google-re2
make install
cd ..
cd re2
bundle install
bundle exec rake compile
bundle exec rspec
```
- [ ] Ran gitlab-rails console smoke test
- [ ] CentOS 7
- [ ] Modern OS
```ruby
irb(main):005:0> regex = RE2('\(iP.+; CPU .*OS (\d+)[_\d]*.*\) AppleWebKit\/')
=> #<RE2::Regexp /\(iP.+; CPU .*OS (\d+)[_\d]*.*\) AppleWebKit\//>
irb(main):006:0> regex.match?('foo')
=> false
irb(main):017:0> regex.match?("Mozilla/5.0 (iPhone; CPU iPhone OS 12_0 like Mac OS X) AppleWebKit/605.1.15 (KHTML, like Gecko) Version/12.0 Mobile/15E148 Safari/604.1")
=> true
```
# Mattermost component upgrade test plan
<!-- Copy and paste the following into your MR description. -->
## Test plan
### Build tests
- [ ] Built on all supported platforms
- [ ] Ran `Trigger:ee-package` and then `qa-test` CI jobs on `gitlab.com`.
### Fresh installation tests
Installed a Linux package created in the build system on a fresh OS installation
and ran the following actions, checks, and tests:
- [ ] Installed Linux package with the new Mattermost version:
- [ ] Verified that package installation logs/output shows no errors.
- [ ] Verified Mattermost version by running `/opt/gitlab/embedded/bin/mattermost version`.
- [ ] Edited `/etc/gitlab/gitlab.rb` and set:
- `external_url`
- `mattermost_external_url`
Both URLs should point to the same system so that GitLab and Mattermost are co-located. Example:
```yaml
external_url 'gitlab.example.com'
mattermost_external_url 'mattermost.example.com'
```
- [ ] Ran `gitlab-ctl reconfigure`.
- [ ] Connected to `gitlab.example.com`.
- [ ] Navigated to `Admin>Settings>Network>Outbound requests` and added `mattermost.example.com` to `Local IP addresses and domain names that hooks and services can accesss` and clicked `Save changes`.
- [ ] Navigated to `mattermost.example.com`.
- [ ] Verified that single-sign on using GitLab credentials was working:
1. Clicked on `or create and account with` GitLab.
1. When presented with the `Authorize GitLab Mattermost to use your account?` page, clicked on `Authorize`.
You should have landed on the `Select teams` page.
- [ ] Verified that when creating a group in GitLab, checking the box for **Create a Mattermost team for this group** also created a team in Mattermost and the GitLab user is a member of that team.
- [ ] Created a test project within the group created in the previous step and initialize with a `README`.
- [ ] Verified Mattermost slash command operation:
- [ ] Enabled slash commands using [GitLab documentation](https://docs.gitlab.com/ee/user/project/integrations/mattermost_slash_commands.html#configure-automatically).
- [ ] Tested slash commands by creating a new issue from the Mattermost instance. After following the prompt to re-authorize, the issue should have been successfully created in GitLab.
- [ ] Verified GitLab issue notification.
- [ ] Configured incoming web hooks in Mattermost using the [GitLab Documentation](https://docs.gitlab.com/ee/user/project/integrations/mattermost.html). Note that you have to configure both Mattermost and GitLab.
- [ ] Using the created web hook, followed the documentation for adding [notification support](https://docs.gitlab.com/ee/user/project/integrations/mattermost.html#configure-mattermost-to-receive-gitlab-notifications).
- [ ] Created an issue in the test project. Verified that the notification for the issue appeared in Mattermost for the GitLab user.
### Upgrade installation tests
Install a Linux package from the previous, latest minor number release on a
fresh OS installation. Run the following actions, checks, and tests:
- [ ] Installed Linux package from the previous, latest minor number release:
- [ ] Verified that package installation logs/output showed no errors.
- [ ] Verified Mattermost version by running `/opt/gitlab/embedded/bin/mattermost version`.
- [ ] Edited `/etc/gitlab/gitlab.rb` and set:
- `external_url`
- `mattermost_external_url`
Both URLs should point to the same system so that GitLab and Mattermost are co-located. Example:
```yaml
external_url 'gitlab.example.com'
mattermost_external_url 'mattermost.example.com'
```
- [ ] Ran `gitlab-ctl reconfigure`.
- [ ] Connected to `gitlab.example.com`.
- [ ] Navigated to `Admin>Settings>Network>Outbound requests` and added `mattermost.example.com` to `Local IP addresses and domain names that hooks and services can accesss` and clicked `Save changes`.
- [ ] Navigated to `mattermost.example.com`.
- [ ] Verified that single-sign on using GitLab credentials is working:
1. Clicked on `or create and account with` GitLab.
1. When presented with the `Authorize GitLab Mattermost to use your account?` page, clicked on `Authorize`.
1. You should have landed on the `Select teams` page.
- [ ] Verified that when creating a group in GitLab, checking the box for **Create a Mattermost team for this group** also created a team in Mattermost and the GitLab user is a member of that team.
- [ ] Created a test project within the group created in the previous step and initialized with a `README`.
- [ ] Verified Mattermost slash command operation:
- [ ] Enabled slash commands using [GitLab documentation](https://docs.gitlab.com/ee/user/project/integrations/mattermost_slash_commands.html#configure-automatically).
- [ ] Tested slash commands by creating a new issue from the Mattermost instance. After following the prompt to re-authorize, the issue should have been successfully created in GitLab.
- [ ] Verified GitLab issue notification.
- [ ] Configured incoming web hooks in Mattermost using the [GitLab Documentation](https://docs.gitlab.com/ee/user/project/integrations/mattermost.html). Note that you have to configure both Mattermost and GitLab.
- [ ] Using the created web hook, followed the documentation for adding [notification support](https://docs.gitlab.com/ee/user/project/integrations/mattermost.html#configure-mattermost-to-receive-gitlab-notifications).
- [ ] Created an issue in the test project. Verified that the notification for the issue appears in Mattermost for the GitLab user.
Upgrade GitLab with a Linux package created with the new Mattermost version. Run the
following actions, checks, and tests:
- [ ] Upgraded to package with new Mattermost version:
- [ ] Verified that package installation logs/output shows no errors.
- [ ] Verified Mattermost version by running `/opt/gitlab/embedded/bin/mattermost version`.
- [ ] Verified Mattermost slash command operation:
- [ ] Tested slash commands by creating a new issue from the Mattermost instance. After following the prompt to re-authorize, the issue should have been successfully created in GitLab.
- [ ] Verified GitLab issue notification.
- [ ] Created an issue in the test project. Verified that the notification for the issue appeared in Mattermost for the GitLab user.
# redis component upgrade test plan
<!-- Copy and paste the following into your MR description. -->
## Test plan
- [ ] Performed a successful GitLab Enterprise Edition (EE) build on all supported platforms (include `build-package-on-all-os` job).
- [ ] Ran `qa-test` CI/CD test job for both GitLab Enterprise Edition and GitLab Community Edition.
- [ ] Redis indicator test cases were not failing.
- [realtime components via assignee test](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347941)
- [project template import](https://gitlab.com/gitlab-org/gitlab/-/quality/test_cases/347932)
- [ ] Performed fresh install and verified operation:
- [ ] Installed on single node
- [ ] Checked installed version:
```shell
/opt/gitlab/embedded/bin/redis-server --version
```
- [ ] Sign-in
- [ ] Imported a project (confirms Sidekiq works)
- [ ] Checked `gitlab-kas` log for Redis errors
- [ ] Checked `redis-exporter` logs for errors
- [ ] Confirmed `gitlab-redis-cli` command connects to Redis
- [ ] Installed on HA Sentinels (see https://docs.gitlab.com/ee/administration/redis/replication_and_failover.html)
- [ ] Signed-in
- [ ] Imported a project (confirms Sidekiq works)
- [ ] Checked `gitlab-kas` log for Redis errors
- [ ] Performed failover
- [ ] Updates
- [ ] Updated on single node:
- [ ] Verified that you get the message:
```plaintext
Running version of Redis different than installed version. Restart redis"
```
- [ ] Sign-in
- [ ] Imported a project (confirms Sidekiq works)
- [ ] Checked `gitlab-kas log` for Redis errors
- [ ] Checked `redis-exporter logs` for errors
- [ ] Confirmed `gitlab-redis-cli` command connects to Redis
- [ ] Updated HA Sentinel/Redis nodes (see https://docs.gitlab.com/ee/update/zero_downtime.html#redis-ha-using-sentinel)
- [ ] Verified that you get the message:
```plaintext
Running version of Redis different than installed version. Restart redis"
```
- [ ] Signed-in
- [ ] Imported a project (confirms Sidekiq works)
- [ ] Checked `gitlab-kas` log for Redis errors
- [ ] Checked `redis-exporter` logs for errors
- [ ] Confirmed `gitlab-redis-cli` command connects to Redis on a Redis node
- [ ] Performed failover
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment