diff --git a/doc/architecture/index.md b/doc/architecture/index.md index ce6eaaa2c355fd5e6da5d132e5113fd57b5ae46c_ZG9jL2FyY2hpdGVjdHVyZS9pbmRleC5tZA==..6130ec3b4aaade0697d8d38fdbd63ca671de7cee_ZG9jL2FyY2hpdGVjdHVyZS9pbmRleC5tZA== 100644 --- a/doc/architecture/index.md +++ b/doc/architecture/index.md @@ -235,3 +235,19 @@ ### What happens during `gitlab-ctl reconfigure` One of the commonly used commands while managing a GitLab instance is `gitlab-ctl reconfigure`. This command, in short, parses the config file and runs the recipes with the values supplied from it. The recipes to be run are defined in a file called `dna.json` present in the `embedded` folder inside the installation directory (This file is generated by a software dependency named `gitlab-cookbooks` that's defined in the software definitions). In the case of GitLab CE, the cookbook named `gitlab` will be selected as the master recipe, which in turn invokes all other necessary recipes, including runit. In short, reconfigure is a chef-client run that configures different files and services with the values provided in the configuration template. + +## Multiple databases + +Previously, the GitLab Rails application was the sole client connected to the +Omnibus GitLab database. Over time, this has changed: + +- Praefect and Container Registry use their own databases. +- The Rails application now uses a [decomposed database](https://gitlab.com/groups/gitlab-org/-/epics/5883). + +Because additional databases might be necessary: + +- The [multi-database blueprint](multiple_database_support/index.md) explains + how to add database support to Omnibus GitLab for new components and features. +- The [accompanying development document](../development/database_support.md) + details the implementation model and provides examples of adding database + support. diff --git a/doc/architecture/multiple_database_support/index.md b/doc/architecture/multiple_database_support/index.md new file mode 100644 index 0000000000000000000000000000000000000000..6130ec3b4aaade0697d8d38fdbd63ca671de7cee_ZG9jL2FyY2hpdGVjdHVyZS9tdWx0aXBsZV9kYXRhYmFzZV9zdXBwb3J0L2luZGV4Lm1k --- /dev/null +++ b/doc/architecture/multiple_database_support/index.md @@ -0,0 +1,152 @@ +--- +status: proposed +creation-date: "2023-10-02" +authors: [ "@pursultani" ] +approvers: [ "@product-manager", "@engineering-manager" ] +owning-stage: "~devops::systems" +participating-stages: [] +--- + +# Multiple databases support + +## Summary + +This document explains how to support a component with one or more databases. It +describes different levels of support and offers an implementation model for +each level to overcome the several challenges of the [recommended deployment models](https://docs.gitlab.com/ee/administration/reference_architectures/). + +The [architecture page](../index.md#multiple-databases) provides some +background on this subject. + +A [development document](../../development/database_support.md) accompanies this +blueprint. It details the implementation model and provides a few examples. + +## Goals + +- Offer [higher levels of support](#levels-of-support) for current and new + components with database requirements. +- Implementation refactors maintain the current configuration options + already present in `gitlab.rb`. +- Minimize breaking changes and refactors in database code with a consistent, + testable, and extensible implementation model. +- Migrate code to the newer implementation method. + +## Proposal + +### Terminology + +|Term|Definition| +|-|-| +|Database|A _logical_ database that a component, such as Rails application, uses. For example, `gitlabhq_production`. A component can have more than one database.| +|Database server| A _standalone process_ or a _cluster_ that provides PostgreSQL database service. Not to be confused with database objects or data.| +|Database objects| Anything that is created with Data Definition Language (DDL), such as `DATABASE`, `SCHEMA`, `ROLE`, or `FUNCTION`. It may include reference data or indices as well. These are partially created by Omnibus GitLab and the rest are created by application-specific _database migrations_.| +|Standalone database server| A single PostgreSQL database server. It can be accessed through a PgBouncer instance.| +|Database server cluster|Encompasses multiple PostgreSQL database servers, managed by Patroni services, backed by a Consul cluster, accessible by using one or more PgBouncer instances, and may include an HAProxy (in TCP mode) as a frontend.| + +### Levels of support + +There are different levels of database support for Omnibus GitLab components. +Higher levels indicate more integration into Omnibus GitLab. + +#### Level 1 + +Configure the component with user-provided parameters from `gitlab.rb` to work +with the database server. For example, `database.yml` is rendered with database +server connection details of the Rails application or database parameters of +Container Registry are passed to its `config.yml`. + +#### Level 2 + +Create database objects and run migrations of the component. Full support at +this level requires Omnibus GitLab to not only create the required database +objects, such as `DATABASE` and `ROLE`, but also to run the application +migration to for the component. + +#### Level 3 + +Static configuration of PgBouncer. At this level, Omnibus GitLab can create a +_dedicated PgBouncer user_ for the component and configure it with user-provided +(from `gitlab.rb`) or application-mandated connection settings. + +This is not specific to clustered database server setups but it is a requirement +for it. There are scenarios where PgBouncer is configured with a standalone +database server. However, all clustered database server setups depend on +PgBouncer configuration. + +#### Level 4 + +Configuration of database server cluster in high-availability (HA) mode. At this +level, Omnibus GitLab supports various deployment models, ranging from _one +cluster for all databases_ to _one cluster per database_. + +Therefore the HA configuration of logical databases must be independent of the +deployment model. + +Consul [services](https://developer.hashicorp.com/consul/docs/srvices/configuration/services-configuration-reference) +can have multiple health-checks and [watches](https://developer.hashicorp.com/consul/docs/dynamic-app-config/watches#service). +At this level, Omnibus GitLab defines _a Consul service per database cluster_ +and _a service watch per logical database_. + +Omnibus GitLab configures [Patroni to register a Consul service](https://patroni.readthedocs.io/en/latest/yaml_configuration.html#consul). +The name of the service is the scope parameter as its tag is the role of the +node which can be one of `master`, `primary`, `replica`, or `standby-leader`. It +uses this service name, which is the same as the scope of Patroni cluster, to +address a database cluster and associate it to any logical database that the +cluster serves. + +This is done with Consul watches that track Patroni services. They find cluster +leaders and notify PgBouncer with the details of both the database cluster and +the logical database. + +#### Level 5 + +Automated or assisted transition from previous deployment models. Not all +components require this level of support but, in some cases, where a recommended +yet deprecated database configuration is in use, Omnibus GitLab may provide +specialized tools or procedures to allow transitioning to the new database +model. In most cases, this is not supported unless specified. + +### Design overview + +Each component manages every aspect of its own database requirements, _except +its database users_. It means that component-specific implementation of database +operations are done in the specific cookbooks of each component. For example, +Rails or Registry database requirements are exclusively addressed in `gitlab` +and `registry` cookbooks and not in `postgresql`, `pgbouncer`, or `patroni` +cookbooks. + +The database users are excluded because `SUPERUSER` or users with `CREATEROLE` +privilege can create PostgreSQL users. Due to security considerations we do not +grant this privilege to the users that are connected over TCP connection. So +components that may connect to a remote database do not have the permission to +create their users. + +Hence each component creates its own database objects, _except its database user_. +`postgresql` and `patroni` cookbooks create the database users but each component +creates the rest of its database objects. The database users must have `CREATEDB` +privilege to allow components create their own `DATABASE` and trusted `EXTENSION`. + +To impose a structure and fix some of the shortcomings of this approach, such as +locality and limited reusability, we use [Chef resource model](https://docs.chef.io/resources/) +and leverage [custom resources](https://docs.chef.io/custom_resources/) for +database configuration and operations, including: + +- Manage lifecycle of component-specific database objects +- Run application-specific database migrations +- Set up PgBouncer to serve the application +- Set up Consul watches to track Patroni clusters + +Cross-cutting concerns such as [central on/off switch for auto-migration](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7716), +logging control, and [pre-flight checks](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/5428) +are addressed with [helper classes](https://docs.chef.io/helpers/) that are +available to all components. The `package` cookbook is a suitable place for +these helpers. + +Helper classes also provide a place to translate the existing user configuration +model (in `gitlab.rb`) to the new model needed for management of +multiple databases. + +### Implementation details + +[Development document](../../development/database_support.md) provides +implementation details and concrete examples for the proposed design. diff --git a/doc/development/database_support.md b/doc/development/database_support.md new file mode 100644 index 0000000000000000000000000000000000000000..6130ec3b4aaade0697d8d38fdbd63ca671de7cee_ZG9jL2RldmVsb3BtZW50L2RhdGFiYXNlX3N1cHBvcnQubWQ= --- /dev/null +++ b/doc/development/database_support.md @@ -0,0 +1,456 @@ +# Database support + +This document provides details and examples on how to implement database support +for an Omnibus GitLab component. The [architecture blueprint](../architecture/multiple_database_support/index.md) +provides the design and definitions. + +1. [Level 1](#level-1) +1. [Level 2](#level-2) + 1. [Examples](#examples) + 1. [Example 1: Registry database objects](#example-1-registry-database-objects) + 1. [Example 2: Registry database migrations](#example-2-registry-database-migrations) + 1. [Example 3: Use database objects and migrations of Registry](#example-3-use-database-objects-and-migrations-of-registry) + 1. [Example 4: Parametrized database objects resource for Rails](#example-4-parameterized-database-objects-resource-for-rails) +1. [Level 3](#level-3) +1. [Level 4](#level-4) +1. [Considerations](#considerations) +1. [Bridge the gap](#bridge-the-gap) + 1. [Reorganize the existing database operations](#reorganize-the-existing-database-operations) + 1. [Support dedicated PgBouncer user for databases](#support-dedicated-pgbouncer-user-for-databases) + 1. [Delay the population of PgBouncer database configuration](#delay-the-population-of-pgbouncer-database-configuration) + 1. [Configurable Consul watch for databases](#configurable-consul-watch-for-databases) + 1. [Helper class for general database migration requirements](#helper-class-for-general-database-migration-requirements) + +## Level 1 + + 1. Add the new database-related configuration attributes to `gitlab.rb`. Do + not forget to update `gitlab.rb.template`. + 1. Update the Chef recipe to consume the configuration attributes. At this + level, the requirement is to pass down the attributes to the component, + generally its through configuration files or command-line arguments. + +For example in `registry` cookbook: + +- `registry['database']` attribute is added to `gitlab.rb` (see [`attributes/default.rb`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/565f7a73f721fa40efc936dfd735b849986ce0ac/files/gitlab-cookbooks/registry/attributes/default.rb#L39)). + +- The configuration template uses the attribute to configure registry (see [`templates/default/registry-config.yml.erb`](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/565f7a73f721fa40efc936dfd735b849986ce0ac/files/gitlab-cookbooks/registry/templates/default/registry-config.yml.erb#L47)). + +## Level 2 + +1. Add dependency to `postgresql` and `pgbouncer` cookbooks. Use `depends` in + `metadata.rb`. This ensures that requirements are met and their Chef custom + resources are available to the cookbook. + +1. Create a `database_objects` custom resource in `resources/` directory of the + cookbook with the default `nothing` action (a no-op action) and a `create` + action. The `create` action can leverage the existing `postgresql` custom + resources to setup the required database objects for the component. + + See: + - `postgresql_user` + - `postgresql_database` + - `postgresql_schema` + - `postgresql_extension` + - `postgresql_query` + + A `database_objects` resource of must create _all_ of the required database + objects for a component. It must not assume that another cookbook creates + some of the objects that it needs. + +1. Create a `database_migrations` custom resource in `resources/` directory of + the cookbook with the default `nothing` action (a no-op action) and a `run` + action. The `run` action executes the commands + for database migrations of the component. + + When the migration runs, it can safely assume that all of the required + database objects are available. Therefore this resource depends on successful + `create` action of `database_objects` resource. + +1. In the `default` recipe of the cookbook, use a `database_objects` resource + that notifies a `database_migrations` resources to `run`. The migrations + should be able to run `immediately` after the preparation of database objects + but a component may choose not to use the immediate trigger. + +### Examples + +All of the following code blocks are provided as examples. You may need +to make adjustments to ensure that they meet your requirements. + +#### Example 1: Registry database objects + +The following example shows a `database_objects` resource in the `registry` +cookbook defined in `registry/resources/database_objects.rb`. + +Notice how it uses custom resources from the `postgresql` cookbook to create +the required database objects. + +```ruby +# registry/resources/database_objects.rb + +unified_mode true + +property :pg_helper, [GeoPgHelper, PgHelper], required: true, sensitive: true + +default_action :nothing + +action :nothing do +end + +action :create do + host = node['registry']['database']['host'] + port = node['registry']['database']['port'] + database_name = node['registry']['database']['database_name'] + username = node['registry']['database']['username'] + password = node['registry']['database']['password'] + + postgresql_user username do + password "md5#{password}" unless password.nil? + + action :create + end + + postgresql_database database_name do + database_socket host + database_port port + owner username + helper new_resource.pg_helper + + action :create + end + + postgresql_extension 'btree_gist' do + database database_name + + action :enable + end +end +``` + +#### Example 2: Registry database migrations + +The following example shows a `database_migrations` resource in the `registry` +cookbook defined in `registry/resources/database_objects.rb`. +Notice how the resource accepts additional parameters. Parameters help +support different migration scenarios, for example separation of pre-deployment +or post-deployment migrations. It also uses [`MigrationHelper`](#helper-class-for-general-database-migration-requirements) +to decide whether to run a migration or not. + +```ruby +# registry/resources/database_migrations.rb + +unified_mode true + +property :name, name_property: true +property :direction, Symbol, default: :up +property :dry_run, [true, false], default: false +property :force, [true, false], default: false +property :limit, [Integer, nil], default: nil +property :skip_post_deployment, [true, false], default: false + +default_action :nothing + +action :nothing do +end + +action :run do + # MigrationHelper is not implemented. It contains general-purpose helper + # methods for managing migrations, for example if a specific component + # migrations can run or not. + # + # See: "Helper class for general database migration requirements" + migration_helper = MigrationHelper.new(node) + + account_helper = AccountHelper.new(node) + logfiles_helper = LogfilesHelper.new(node) + logging_settings = logfiles_helper.logging_settings('registry') + + bash_hide_env "migrate registry database: #{new_resource.name}" do + code <<-EOH + set -e + + LOG_FILE="#{logging_settings[:log_directory]}/db-migrations-$(date +%Y-%m-%d-%H-%M-%S).log" + + umask 077 + /opt/gitlab/embedded/bin/registry \ + #{new_resource.direction} \ + #{"--dry-run" if new_resource.dry_run} \ + #{"--limit #{new_resource.limit}" unless new_resource.limit.nil?} \ + ... \ + #{working_dir}/config.yml \ + 2>& 1 | tee ${LOG_FILE} + + STATUS=${PIPESTATUS[0]} + + chown #{account_helper.gitlab_user}:#{account_helper.gitlab_group} ${LOG_FILE} + + exit $STATUS + EOH + + not_if { migration_helper.run_migration?('registry') } + end +end +``` + +#### Example 3: Use database objects and migrations of Registry + +The resources defined in the previous examples are used in +`registry/recipes/enable.rb` recipe. + +See how `only_if` and `not_if` guards are used to decide when to create the +database objects or run the migrations. Also, pay attention to the way that +`notifies` is used to show the dependency of the migrations on the successful +creation of database objects. + +```ruby +# registry/recipes/enable.rb + +# ... + +pg_helper = PgHelper.new(node) + +registry_database_objects 'default' do + pg_helper pg_helper + action :create + only_if { node['registry']['database']['enable'] } + not_if { pg_helper.replica? } + notifies :create, 'registry_database_migrations[up]', :immediately if pg_helper.is_ready? +end + +registry_database_migrations 'up' do + direction :up + only_if { node['registry']['database']['enable'] } + not_if { pg_helper.replica? } +end + +# ... +``` + +#### Example 4: Parameterized database objects resource for Rails + +The following example shows how a single implementation of the database objects +for Rails application can satisfy the requirements of the decomposed database model. + +In this example the _logical_ database is passed as the _resource name_ and is +used to lookup settings of each database from the configuration. The settings +are passed to `postgresql` custom resources. This is particularly useful when +the majority of implementation of can be reused to replace two or more resources. + +```ruby +# gitlab/resources/database_objects.rb + +unified_mode true + +property :pg_helper, [GeoPgHelper, PgHelper], required: true, sensitive: true + +default_action :nothing + +action :nothing do +end + +action :create do + global_database_settings = { + # ... + port: node['postgresql']['port'] + host: node['gitlab']['gitlab_rails']['db_host'] + # ... + } + database_settings = node['gitlab']['gitlab_rails']['databases'][new_resource.resource_name] + + database_settings = global_database_settings.merge(database_settings) unless database_settings.nil? + + username = database_settings[:username] + password = database_settings[:password] + database_name = database_settings[:database_name] + host = database_settings[:host] + port = database_settings[:port] + + postgresql_user username do + password "md5#{password}" unless password.nil? + + action :create + end + + postgresql_database database_name do + database_socket host + database_port port + owner username + helper new_resource.pg_helper + + action :create + end + + postgresql_extension 'btree_gist' do + database database_name + + action :enable + end +end +``` + +And this is how it is used in `gitlab/recipes/default.rb`: + +```ruby +gitlabe_database_objects 'main' do + pg_helper pg_helper + + action :create + only_if { node['gitlab']['gitlab_rails']['databases']['main']['enable'] ... } +end + +gitlabe_database_objects 'ci' do + pg_helper pg_helper + + action :create + only_if { node['gitlab']['gitlab_rails']['databases']['ci']['enable'] ... } +end +``` + +## Level 3 + +1. Add a new attribute for PgBouncer user. Make sure that this attribute is + mapped to the existing `pgbouncer['databases']` setting and can consume it. + This attribute is used to create a dedicated PgBouncer user for the component + as opposed to reusing the existing Rails user, the same as what Praefect + currently does. + + NOTE **Note:** + It is very important that we do not introduce any breaking changes to + `gitlab.rb`. The current user settings must work without any change. + +1. Use `pgbouncer_user` custom resource from `pgbouncer` cookbook to create the + dedicated PgBouncer user for the component. Use the attribute that is + described in the previous step. + +## Level 4 + +1. Add a new attribute for the component to specify the name of the Consul + service of the database cluster. This is either the name of the scope of the + Patroni cluster (when automatic service registry for Patroni, i.e. `patroni['register_service']`, + is enabled) or the name of the Consul service that is configured manually + without Omnibus GitLab. + +1. Use `database_watch` custom resource<sup>[[Needs Implementation](#configurable-consul-watch-for-databases)]</sup> + to define a new Consul watch for the database cluster service. It notifies + PgBouncer to update the logical database endpoint when the leader of the + cluster changes. Pass the name of the Consul service, logical database, and + any other PgBouncer options as parameters to the watch. + +_All `database_watch` resources must be placed in the `consul` cookbook_. As +opposed to the previous levels, this is the only place where database-related +resources are concentrated in one cookbook, `consul`, and not managed in +the same cookbooks as their associated components. + +The reason for this exception is that the watches run on the PgBouncer nodes, +where `pgbouncer_role` is used. All components, except PgBouncer and Consul, +are disabled. Note that this is in line with existing user configuration since +it is [the recommended configuration for PgBouncer node](https://docs.gitlab.com/ee/administration/postgresql/replication_and_failover.html#configure-pgbouncer-nodes). +We don't want to introduce any breaking changes into `gitlab.rb`. + +## Considerations + +- _No other resource should be involved with database setup_. + +- All custom resources _must be idempotent_. For example they must not fail + when an object already exist even though they are created or ran in another + cookbook. Instead they must be able to update the current state of the + database objects, configuration, or migrations based on the new user inputs. + +- In HA mode, given that multiple physical nodes are involved, Omnibus GitLab + may encounter certain limitations to provide full automation of the + configuration. This is an acceptable limitation. + +## Bridge the gap + +Currently not all of the custom resources or helper classes are available. Even +if they are, they may require some adjustments to meet the specified requirements. +Here are some examples. + +### Reorganize the existing database operations + +This model requires some adjustments in `postgresql`, `patroni`, and `gitlab` +cookbooks. For example `database_objects` that is defined in `gitlab` cookbook +must be used in the same cookbook and its usage must be removed from `postgresql` +and `patroni` cookbooks. + +The database service cookbooks (`postgresql` and `patroni`) should not deal with +database objects and migrations and must delegate them to the application +cookbooks (e.g. `gitlab`, `registry`, and `praefect`). However, to support this, +custom resources of `postgresql` cookbook must be able to work on any node. +Currently they assume they run on the PostgreSQL node and use the UNIX socket to +connect to the database server. This assumption forces to place all database +operations in one cookbook. + +The same is true for `pgbouncer` cookbook. Currently the only PgBouncer user is +created in the `users` recipe of this cookbook. This can change as well to allow +each component cookbook to create its own PgBouncer users. + +### Support dedicated PgBouncer user for databases + +The current `pgbouncer` cookbook [_mostly_ supports multiple databases](https://docs.gitlab.com/ee/administration/gitaly/praefect.html#configure-a-new-pgbouncer-database-with-pool_mode--session). + +The `pgbouncer` cookbook only creates PgBouncer users for the main Rails database. This +is why non-Rails applications connect with the same PgBouncer user created for Rails. + +We can currently set up PgBouncer support for decomposed Rails databases sharing +the same user. But for Praefect or Registry, we need additional work to create +dedicated PgBouncer users. + +NOTE **Note:** +A shared user does not mean connection settings for each database must +be the same. It only means that multiple databases use the same user for +PgBouncer connection. + +### Delay the population of PgBouncer database configuration + +The implementation of `gitlab-ctl pgb-notify` supports multiple +databases. It is generic enough that, as long as the PgBouncer users are created, +it can update `databases.ini` from the `databases.json` file. + +However, when PgBouncer users are pulled into individual cookbooks, the initial +`databases.ini` that is created or updated in `gitlab-ctl reconfigure` may not +be valid because it references PgBouncer users that are not created yet. +We should be able to fix this by delaying the action on Chef resource that calls +`gitlab-ctl pgb-notify`. + +### Configurable Consul watch for databases + +Consul cluster can be shared between multiple Patroni clusters (using different +scopes, such as `patroni['scope']`), but updating PgBouncer configuration is still +problematic because the Consul watch scripts are not fully configurable. + +The current implementation has several limitations: + +1. Omnibus GitLab uses `postgresql` service that is [explicitly defined](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-cookbooks/consul/recipes/enable_service_postgresql.rb) + in `consul` cookbook. This service, that is currently being used to notify + PgBouncer, is a leftover of the transition from RepMgr to Patroni. It must + be replaced with the Consul service that [Patroni registers](https://patroni.readthedocs.io/en/latest/yaml_configuration.html#consul). + When `patroni['register_service']` is enabled Patroni registers a Consul + service with `patroni['scope']` parameter and the tag `master`, `primary`, + `replica`, or `standby-leader` depending on the node's role. + +1. The current [failover script](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/master/files/gitlab-cookbooks/consul/templates/default/watcher_scripts/failover_pgbouncer.erb) + is associated to a Consul watch for `postgresql` service and is not capable + of handling multiple databases because [database name can not be changed](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/a9c14f6cfcc9fe0d9e98da7d04f43c5772d5f768/files/gitlab-cookbooks/consul/libraries/watch_helper.rb#L50). + +We need to extend the current Omnibus GitLab capability to use Consul watches to +track Patroni services, find cluster leaders, and notify PgBouncer with a +parameterized failover script. + +In order to do this we implement `database_watch` custom resource in `consul` +cookbook. It defines a database-specific Consul watch for database a cluster +service and passes the required information to a parameterized failover script +to notify PgBouncer. The key attributes of this resource are: + + 1. The service name, that specifies which database cluster must be watched. + It could be the scope of the Patroni cluster when `patroni['register_service']` + is enabled or a Consul service name when it is manually configured. + + 1. The database name that specifies which logical databases should be + reconfigured when the database cluster leader changes. + +### Helper class for general database migration requirements + +`MigrationHelper`<sup>[Needs implementation]</sup> implements general +requirements of database migrations, including the central switch for enabling +or disabling auto-migrations. It can also provides the mapping between the +existing and new configuration attributes.