- Dec 31, 2024
-
-
John Skarbek authored
This reverts merge request !7962
-
James Liu authored
For single node instances, recommend `gitaly['configuration']['storage']` setting. For instances where Gitaly is external, recommend `gitlab_rails['repositories_storages']` for the Rails node Changelog: deprecated
-
- Dec 18, 2024
-
-
Lin Jen-Shin authored
Co-authored-by:
Andrew Patterson <apatterson@gitlab.com>
-
- Sep 17, 2024
-
-
Balasankar 'Balu' C authored
Rather than re-use the attribute from GitLab Shell, Gitaly stores the token it uses to authenticate with GitLab under a unique attribute. Closes https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/8688 Changelog: changed Signed-off-by:
Balasankar "Balu" C <balasankar@gitlab.com>
-
- Jun 04, 2024
-
-
Will Chandler authored
57396c7ab (Prevent Gitaly storages from using the same path, 2024-05-09) added a check to validate that only one storage is associated with a given filesystem path. We have received a customer report[0] that the error message used is confusing. Update the message to make it clear what the problem is. [0] https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/8565
-
- Jun 24, 2024
-
-
Ian Baum authored
* Creates the gitlab-backup user and groups if they dont' exist * Adds the user to other groups * For use by gitlab-backup-cli for read/write permission to data for backups and restores Changelog: added
-
- May 10, 2024
-
-
Balasankar 'Balu' C authored
- If a node has legacy `git-data` storages configured but no longer runs the gitaly service, skip the unique storage path validation check. Signed-off-by:
Balasankar 'Balu' C <balasankar@gitlab.com>
-
- May 09, 2024
-
-
Will Chandler authored
Gitaly is making a breaking change with v17.0 to prevent multiple storages from sharing the same local path. This is being done as part of the work to add a write-ahead log to Gitaly, see https://gitlab.com/gitlab-org/gitaly/-/issues/5598 for further details. Validate that Gitaly's config does not have more than one storage using the same path, dereferencing any symlinks. Changelog: changed
-
- Feb 29, 2024
-
-
Ahmad Sherif authored
Currently, `gitlab_rails` service needs to be enabled in order for `gitaly` service to function properly. Specifically, `gitlab_rails` creates `git_data_dirs` (e.g. `/var/opt/gitlab/git-data/repositories`) which is needed for Gitaly to start-up. If only the `gitaly` service is enabled, then Gitaly won't start due to config validation error (i.e. storage directories have to be present). This MR makes the creation of these directories the responsibility of the `gitaly` cookbook. The `gitlab` cookbook still creates the directories as backward-compatibility measure, as I'm not sure if these directories are needed outside of Gitaly. This is needed as part of https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/24530. Changelog: fixed
-
- Feb 27, 2024
-
-
Ahmad Sherif authored
Administrators may now set `gitlab` configuration values directly rather than rely on automatic calculations from the rails service. Related https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/24530 Changelog: added
-
- Dec 05, 2023
-
-
Will Chandler authored
Gitaly has added several subcommands over the last few years. Execute Gitaly using the `serve` subcommand, rather then without a subcommand, which implicitly calls `serve`. Changelog: changed
-
Will Chandler authored
Gitaly is in the process of implementing a write-ahead log (WAL) for git operations, which will provide significant benefits for resiliance and scalability. This model requires that we have only one Gitaly process running at a time. However, our current upgrade method keeps the old and new Gitaly processes running in parallel for an period of time defined by the administrator. The `gitaly-wrapper` binary is used by Omnibus to provide a stable pid to `runit` and sets the `GITALY_UPGRADES_ENABLED` environment variable that causes Gitaly to perform upgrades with concurrent processes. To prepare for the deployment of Gitaly's WAL architecture, add a new `gitaly['use_wrapper']` option which allows admins to execute Gitaly without the `gitaly-wrapper` binary so that there is never more than one Gitaly process running at once. This is enabled by default, admins must opt into the new behavior. Without the wrapper, sending SIGHUP to Gitaly will cause momentary service interruptions for git clone operations, ~300ms in our benchmarks. Read-only web requests will retry automatically for outages lasting up to 750ms and will handle the upgrade gracefully. We retain the wrapper for Praefect as this will remain compatible with running concurrent processes. Changelog: added
-
- May 17, 2023
-
-
João Alexandre Cunha authored
The old configuration structure was deprecated and expected to be removed in 16.0. User should refer to this link for the config migration: https://docs.gitlab.com/ee/update/#gitaly-omnibus-gitlab-configuration-structure-change Changelog: removed
-
- May 05, 2023
-
-
Toon Claes authored
Gitaly removed the gitaly-ruby configuration because it's no longer using Ruby. So we can remove code that composes this configuration. Issue: https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7711 Changelog: deprecated
-
- May 04, 2023
-
-
Jason Young authored
Changelog: added Signed-off-by:
Jason Young <jyoung@gitlab.com>
-
- Apr 18, 2023
-
-
Balasankar 'Balu' C authored
So that non-actionable deprecation notice about gitaly['storage'] is not shown to users. Closes https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7743 Changelog: fixed Signed-off-by:
Balasankar "Balu" C <balasankar@gitlab.com>
-
- Apr 03, 2023
-
-
Jason Young authored
Replaces logrotate_helper Changelog: changed Signed-off-by:
Jason Young <jyoung@gitlab.com>
-
- Mar 17, 2023
-
-
Balasankar 'Balu' C authored
Related https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6873 Related https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/7455 Changelog: deprecated Signed-off-by:
Balasankar "Balu" C <balasankar@gitlab.com>
-
- Feb 22, 2023
-
-
Sami Hiltunen authored
The configuration formats of Gitaly and Omnibus deviate from each other creating a few problems: 1. Configuration keys for the same option differ between the context of Omnibus and Gitaly creating confusion for the user. 2. Each new key added requires duplicate work to map the different keys into Omnibus. This change introduces gitaly['configuration'] matching configuration in Omnibus with Gitaly's own format. The previous configuration keys are remapped into this new format to assist administrators during the transition period. The configuration values are now serialized directly into TOML instead of using a template. Removes unneeded values allowing Gitaly's own source to be the single source of truth for configuration. Updates tests because the new serialization method does not output empty config sections. Guards are included to prevent errors with latency configurations that must be arrays where the previous configuration expected a string until such time as Gitaly's own validation subroutines are implemented. Changelog: changed
-
- Feb 09, 2023
-
-
Steve Azzopardi authored
What --- Mark the gitaly configuration as `sensitive` Why --- Stop showing the diff in the `chef-client` run logs since the Gitaly configuration has tokens inside of it, and logging these tokens can be easily leaked if we copy/paste the logs or if we store these logs externally. Reference: https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/17171 Changelog: changed Signed-off-by:
Steve Azzopardi <sazzopardi@gitlab.com>
-
- Jan 25, 2023
-
-
Sami Hiltunen authored
-
- Jan 17, 2023
-
-
Katrin Leinweber authored
Was 4bf3bf85 maybe too specific? Folks might introduce left-leading and right-trailing whitespace. Since we have a chance here to work around such typos, should we? If not, we'd pass incorrect values down to for example https://github.com/git/git/blob/v2.38.1/config.c#L1433-L1437 Which triggers quite a clear and useful message, though. Co-authored-by:
Ben Prescott <bprescott@gitlab.com>
-
- Jan 16, 2023
-
-
Patrick Steinhardt authored
The `Gitlab['omnibus-gitconfig']['system']` attribute used to contain Git configuration that was subsequently used by Gitaly. This was done by writing its config entries into the global `gitconfig` file so that they were picked up by all Git commands. This has proven to be insufficient in the context of Gitaly, which needs a lot more fine-grained control over the exact Git configuration it uses when spawning Git commands. One point in case was the deprecation of the old `core.fsyncObjetFiles` configuration, which if present caused us to print user-visible warnings. We have since moved all default Git configuration into Gitaly itself, which thus does not need any external Git configuration by default anymore. Furthermore, we have implemented a mechanism in Gitaly to just plain ignore the global `gitconfig` file -- instead, non-default Git configuration needs to be written into Gitaly's `config.toml`. In Omnibus though we still carry legacy defaults that were used for the Git configuration back when Gitaly didn't yet knew to set up its own defaults. As we know to strip those defaults when writing Gitaly's `config.toml` they don't apply at all in that context, but we still write them into the global `gitconfig` file. Consequentially, these defaults apply to all Git commands not spawned by Gitaly itself. Arguably though this is not required: - `pack.threads=1` controls how many packs should be used when writing Git packfiles. This was added to limit the amount of resources a single Git process could use in the context of Gitaly. - `receive.fsckObjects=true` asks Git to perform consistency checks for objects when serving a push. We should never accept pushes into Git repositories outside of Gitaly anyway. - `receive.advertisePushOptions=true` asks Git to advertise the availability of push options to clients when serving a push. Again, we should never accept pushes into Git repositories outside of Gitaly. - `transfer.hideRefs` hides various internal namespaces when serving fetches and pushes. Same as above, we should not accept pushes into Git repositories outside of Gitaly. - `repack.writeBitmaps=true` asks Git to write bitmaps via git-repack(1). This would even be wrong if git-repack(1) was to be executed in a repository that is connected to an object pool. - `fetch.writeCommitGraph=true` asks Git to write commit-graphs when fetching into a repository. This data structure is used to speed up various operations. There is no reason to enable this globally. - `core.alternateRefsCommand="exit 0 #"` will disable listing alternate references when a repository is connected to another repository. There is no reason to disable this globally. - `core.fsyncObjectFiles=true` is the only one that is debatable as it enables calling fsync(3P) when writing objects into a Git repository. This option has been deprecated in Git v2.36.0 though and causes deprecation warnings to be printed on every invocation of Git. So if at all, we would want to set `core.fsync` nowadays to enable this code again. On the other hand though there should be no manual commands that write Git objects into repositories controlled by Gitaly in the first place as that is not supported. Drop all the default values. All of these config entries are managed by Gitaly already, have no obvious purpose when running Git commands globally and add complexity and confusion around what the actual configuration is now. Changelog: removed
-
- Jan 11, 2023
-
-
Patrick Steinhardt authored
This reverts 8f3d825f6 (Merge branch 'pks-omnibus-gitconfig-drop-default-values' into 'master', 2023-01-11), which has dropped the gitconfig default values. This has caused a pipeline failure when packaging: ================================================================================ Error executing action `create` on resource 'template[/opt/gitlab/embedded/etc/gitconfig]' ================================================================================ Chef::Mixin::Template::TemplateError ------------------------------------ undefined method `each' for nil:NilClass Resource Declaration: --------------------- # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb 70: template "#{install_dir}/embedded/etc/gitconfig" do 71: source "gitconfig-system.erb" 72: mode 0755 73: variables gitconfig: node['gitlab']['omnibus-gitconfig']['system'] 74: end 75: Changelog: fixed
-
Patrick Steinhardt authored
The `Gitlab['omnibus-gitconfig']['system']` attribute used to contain Git configuration that was subsequently used by Gitaly. This was done by writing its config entries into the global `gitconfig` file so that they were picked up by all Git commands. This has proven to be insufficient in the context of Gitaly, which needs a lot more fine-grained control over the exact Git configuration it uses when spawning Git commands. One point in case was the deprecation of the old `core.fsyncObjetFiles` configuration, which if present caused us to print user-visible warnings. We have since moved all default Git configuration into Gitaly itself, which thus does not need any external Git configuration by default anymore. Furthermore, we have implemented a mechanism in Gitaly to just plain ignore the global `gitconfig` file -- instead, non-default Git configuration needs to be written into Gitaly's `config.toml`. In Omnibus though we still carry legacy defaults that were used for the Git configuration back when Gitaly didn't yet knew to set up its own defaults. As we know to strip those defaults when writing Gitaly's `config.toml` they don't apply at all in that context, but we still write them into the global `gitconfig` file. Consequentially, these defaults apply to all Git commands not spawned by Gitaly itself. Arguably though this is not required: - `pack.threads=1` controls how many packs should be used when writing Git packfiles. This was added to limit the amount of resources a single Git process could use in the context of Gitaly. - `receive.fsckObjects=true` asks Git to perform consistency checks for objects when serving a push. We should never accept pushes into Git repositories outside of Gitaly anyway. - `receive.advertisePushOptions=true` asks Git to advertise the availability of push options to clients when serving a push. Again, we should never accept pushes into Git repositories outside of Gitaly. - `transfer.hideRefs` hides various internal namespaces when serving fetches and pushes. Same as above, we should not accept pushes into Git repositories outside of Gitaly. - `repack.writeBitmaps=true` asks Git to write bitmaps via git-repack(1). This would even be wrong if git-repack(1) was to be executed in a repository that is connected to an object pool. - `fetch.writeCommitGraph=true` asks Git to write commit-graphs when fetching into a repository. This data structure is used to speed up various operations. There is no reason to enable this globally. - `core.alternateRefsCommand="exit 0 #"` will disable listing alternate references when a repository is connected to another repository. There is no reason to disable this globally. - `core.fsyncObjectFiles=true` is the only one that is debatable as it enables calling fsync(3P) when writing objects into a Git repository. This option has been deprecated in Git v2.36.0 though and causes deprecation warnings to be printed on every invocation of Git. So if at all, we would want to set `core.fsync` nowadays to enable this code again. On the other hand though there should be no manual commands that write Git objects into repositories controlled by Gitaly in the first place as that is not supported. Drop all the default values. All of these config entries are managed by Gitaly already, have no obvious purpose when running Git commands globally and add complexity and confusion around what the actual configuration is now. Changelog: removed
-
- Nov 30, 2022
-
-
Patrick Steinhardt authored
Right now, it's only possible for admins to specify the Gitaly gitconfig as a single key and value pair. While this works just fine in theory when you only consider this in the context of the `[[git.config]]` stanzas. We'll also migrate the global gitconfig from its current location in `omnibus_gitconfig['system']` to a new key though, and it would be great if we could use the same configuration format for both. But we'd have to continue to do laborious string manipulation if we wanted to use the key-value format for that new configuration as we don't write `[[git.config]]` stanzas there, but gitconfig files. Instead, introduce the ability to explicitly specify the Git config as a touple of sections (`http`), subsections (`"http://example.com"`) and keys (`insteadOf`) that together would form the complete key `http."http://example.com".insteadOf`. Like this, we have sufficient information to write both file formats without having to do any string conversations at all. In order to not break backwards compatibility with the just-introduced `gitaly['gitconfig']`, we retain the ability to just specify a key. Changelog: fixed
-
- Oct 17, 2022
-
-
Balasankar 'Balu' C authored
This reverts merge request !6401
-
- Sep 29, 2022
-
-
Takuya Noguchi authored
As the patch to allow using libedit was removed in Python 3.10.0, the patch is now removed. https://github.com/python/cpython/pull/24189 Signed-off-by:
Takuya Noguchi <takninnovationresearch@gmail.com> Changelog: other
-
- Aug 23, 2022
-
-
Will Chandler authored
Gitaly recently added the option to GPG sign all commits created by GitLab, such as changes made via the WebIDE, merge commits, etc. This commit adds a new `gitaly['gpg_signing_key_path']` option to make the new setting accessible in Omnibus. Changelog: added
-
- Aug 25, 2022
-
-
Stan Hu authored
Some users have configured `omnibus_gitconfig['system']` improperly, causing this obscure error: ``` NoMethodError: undefined method `lstrip' for nil:NilClass /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitaly/libraries/gitaly.rb:93:in `block (2 levels) in parse_gitconfig' /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitaly/libraries/gitaly.rb:91:in `map' /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitaly/libraries/gitaly.rb:91:in `block in parse_gitconfig' ``` This commit ensures each config line has a proper `key=value` form and displays a clearer message. For example: ``` RuntimeError ------------ Invalid entry detected in omnibus_gitconfig['system']: 'fsckObjects' should be in the form key=value ``` Relates to https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6986 Changelog: changed
-
- Aug 05, 2022
-
-
Patrick Steinhardt authored
In 4bf3bf85 (gitaly: Reintroduce migration to [[git.config]] stanzas, 2022-07-13), we introduced a toggle telling Gitaly to ignore any gitconfig files on the local disk. Git configuration should only have one source of truth, and with this toggle enabled Gitaly only uses its own default values and overrides must be provided by the administrator through the new `[[git.config]]` stanzas. Because the initial attempt had to be reverted, this feature was rolled out to our production fleet gated behind an opt-in flag. We observed no change in behavior during testing, so this toggle may now be removed and local gitconfig files will be permanently ignored. Removes the flag and gitaly now ignores the gitconfig unconditionally. Closes https://gitlab.com/gitlab-org/gitaly/-/issues/4230 Changelog: changed
-
- Jul 19, 2022
-
-
Stan Hu authored
If for some reason an EE installation with Consul is downgraded to a CE installation, `gitlab-ctl reconfigure` will fail with an obscure message: ``` NoMethodError occurred in delayed notification: undefined method `run_action' for "execute[reload consul]":String ``` In a CE build, the Consul recipes are not loaded so `reload consul` is not available. To avoid this error, we can use the `Services.enabled?` helper, which will verify that Consul is in the list of available services in the first place. Relates to https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6934 Changelog: fixed
-
- Jul 15, 2022
-
-
Patrick Steinhardt authored
The `self_signed_cert` setting that had been part of the gitlab-shell configuration has been deprecated in v14.8 and removed in v15.0 without any replacement. Administrators must instead make sure that they install any self-signed certificates into `/etc/gitlab/trusted-certs`. Follow through with the deprecation and remove the setting where it's still passed down. This most importantly also includes Gitaly and Praefect, which reuse the configuration injected via gitlab-shell. Given that we have already gone through the deprecation phase of this toggle it's not necessary to go through the same deprecation cycle again for these projects. Changelog: deprecated
-
- Jul 13, 2022
-
-
Patrick Steinhardt authored
With commit 2727686d2 (gitaly: Migrate to inject Git configuration via `config.toml`, 2022-06-20), we have migrated the configuration for Git commands spawned by Gitaly from the system-level gitconfig file into Gitaly's `config.toml` via `[[git.config]]` stanzas. This change caused an incident because it change inadvertently fixed a configuration error: because we had globally been setting `core.alternateRefsCommand=exit 0#` in the gitconfig, Gitaly mistakenly ignored all alternate references in object pools. This is wrong though when calculating the repository size, where we need to consider alternate references in order to correctly calculate the size of objects that are only contained in the repository itself. But while the migration to `[[git.config]]` stanzas fixed this bug, it surfaced that the current implementation of how this is done in Gitaly is extremely inperformant in large repositories with many references. And this inefficiency ultimately end to starving the node of resources, which led to the revert of this change via a42c5611b (Merge branch 'revert-fe00a13e' into 'master', 2022-06-20). Gitaly has since fixed this performance edge case so that we shouldn't hit the same bug this time. So let's reintroduce the migration to the new way of configuring Git so that we can also fix the bug about repo size calculations. Note that this commit is a simple revert. This time around we'll also introduce a new flag to opt-in to this migration so that we can first excessively test it in staging without also impacting production. To keep this as a clean revert though this change will be done in the subsequent commit. Changelog: added
-
- Jun 20, 2022
-
-
Stan Hu authored
This reverts merge request !6128
-
Patrick Steinhardt authored
Currently, there is a single `gitlab['omnibus-gitconfig']` setting that controls the Git configuration across different components of the stack by writing either a system-level or global-level gitconfig file. This makes it hard to see which parts of the gitconfig are used by what part of the stack. For Gitaly the story is a bit complicated: the gitconfig files are in fact read, but almost all of its values are ignored because Gitaly uses its own set of defaults which override them, which is really confusing for administrators. Gitaly has instead introduced a new configuration `[[git.config]]` that needs to be added to its `config.toml` file, and this configuration does indeed override Gitaly's default configuration. We do not currently expose this setting in Omnibus though. To lessen the confusion what's read and what isn't, Gitaly will start to ignore all gitconfig files with v16.0. Prepare for this deprecation by migrating from writing gitconfig files to instead writing `[[git.config]]` sections. Given that other parts of the stack may still be reading the old gitconfig files we cannot just stop writing gitconfig files. Instead, this commit carves out Gitaly's own gitconfig so that it can be configured independently of the rest of the stack. Due to backwards-compatibility reasons we need to migrate old settings that had been set in `omnibus_gitconfig['system']` automatically to the new `gitaly['gitconfig']` configuration until v16.0 has been released. This migration is further complicated by a eprecation in Git v2.36.0 itself though: if the old `core.fsyncObjectFiles` configuration is set, then Git will print a user-visible warning every time it's executed. Unfortunately though, administrators were required to set this config in the past, and we thus advertised this configuration entry as part of the default configuration. So we must ensure that this old default key is not set in the new configuration: in fact, we don't want any of the previously set defaults to end up in the new configuration so that Gitaly is the single source of truth with regards to what the defaults are. So in the general case, `gitaly['gitconfig']` should be empty. To work around this issue we thus automatically migrate configuration in `omnibus_gitconfig['system']`, but strip out any of the previous default values. The end result is thus a configuration that only contains what the administrator has manually overridden. Changelog: changed
-
- May 16, 2022
-
-
Patrick Steinhardt authored
Gitaly is still using Rugged in its Ruby sidecar to provide a very limited set of RPCs that have not yet been ported to Go. While Rugged in the past required the gitconfig file to function correctly, nowadays it only requires the `core.fsyncObjectFiles` config entry. This entry must be set to `true` or otherwise it is trivial to end up with repository corruption. This means that the gitconfig must be in a specific well-known state for Rugged to operate correctly. But currently there is still the option to set Rugged's gitconfig search path, which must in fact point to a path that contains a gitconfig with the mentioned config entry. This is quite fragile without any benefit for the administrator at all. To fix this, Gitaly has introduced a fallback mode via [1] that causes it to write a Rugged-specific gitconfig file in case the Rugged search path is not set up. This allows us to now phase out support for setting this configuration altogether: it's not required, may cause repository corruption if misconfigured, and ultimately only makes the architecture harder to reason about given that we currently share this gitconfig file between multiple components. Remove the ability to set Rugged's gitconfig search path without any replacement. [1]: https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4560 Changelog: removed
-
- May 24, 2022
-
-
John Cai authored
In 15.0, we are updating the way that cgroups are configured. https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4483 has the Gitaly changes. This commit modifies the way the gitaly config toml generates the cgroups config according to the new format. Changelog: deprecated
-
- Apr 20, 2022
-
-
Robert Marshall authored
- Remove the custom_hooks_dir from the gitlab_shell configuration tree Closes https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6393 Changelog: removed Signed-off-by:
Robert Marshall <rmarshall@gitlab.com>
-
- Apr 27, 2022
-
-
Patrick Steinhardt authored
Support for configuring the internal socket directory has been deprecated in favor of the new runtime directory introduced by Gitaly in v14.10. Remove the infrastructure to configure the internal socket directory. Changelog: removed
-