Skip to content
Snippets Groups Projects
  1. Dec 31, 2024
  2. Dec 18, 2024
  3. Sep 17, 2024
  4. Jun 04, 2024
  5. Jun 24, 2024
    • Ian Baum's avatar
      Add gitlab-backup user · ab269a7991a2
      Ian Baum authored
      * Creates the gitlab-backup user and groups if they dont' exist
      * Adds the user to other groups
      * For use by gitlab-backup-cli for read/write permission to data for
        backups and restores
      
      Changelog: added
      ab269a7991a2
  6. May 10, 2024
  7. May 09, 2024
  8. Feb 29, 2024
    • Ahmad Sherif's avatar
      Create git_data_dirs even if gitlab_rails is disabled · ea069d9fe8c9
      Ahmad Sherif authored
      Currently, `gitlab_rails` service needs to be enabled in order for
      `gitaly` service to function properly. Specifically, `gitlab_rails`
      creates `git_data_dirs` (e.g. `/var/opt/gitlab/git-data/repositories`)
      which is needed for Gitaly to start-up. If only the `gitaly` service is
      enabled, then Gitaly won't start due to config validation error (i.e.
      storage directories have to be present).
      
      This MR makes the creation of these directories the responsibility of
      the `gitaly` cookbook. The `gitlab` cookbook still creates the
      directories as backward-compatibility measure, as I'm not sure if these
      directories are needed outside of Gitaly.
      
      This is needed as part of
      https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/issues/24530.
      
      Changelog: fixed
      ea069d9fe8c9
  9. Feb 27, 2024
  10. Dec 05, 2023
    • Will Chandler's avatar
      Run gitaly with 'serve' subcommand · c012732c96af
      Will Chandler authored
      Gitaly has added several subcommands over the last few years.
      Execute Gitaly using the `serve` subcommand, rather then without a
      subcommand, which implicitly calls `serve`.
      
      Changelog: changed
      c012732c96af
    • Will Chandler's avatar
      Add 'use_wrapper' setting to Gitaly · 77c9b1eb7559
      Will Chandler authored
      Gitaly is in the process of implementing a write-ahead log (WAL) for git
      operations, which will provide significant benefits for resiliance and
      scalability.
      
      This model requires that we have only one Gitaly process running at a
      time. However, our current upgrade method keeps the old and new Gitaly
      processes running in parallel for an period of time defined by the
      administrator.
      
      The `gitaly-wrapper` binary is used by Omnibus to provide a stable pid
      to `runit` and sets the `GITALY_UPGRADES_ENABLED` environment variable
      that causes Gitaly to perform upgrades with concurrent processes.
      
      To prepare for the deployment of Gitaly's WAL architecture, add a new
      `gitaly['use_wrapper']` option which allows admins to execute Gitaly
      without the `gitaly-wrapper` binary so that there is never more than one
      Gitaly process running at once. This is enabled by default, admins must
      opt into the new behavior.
      
      Without the wrapper, sending SIGHUP to Gitaly will cause momentary
      service interruptions for git clone operations, ~300ms in our
      benchmarks. Read-only web requests will retry automatically for outages
      lasting up to 750ms and will handle the upgrade gracefully.
      
      We retain the wrapper for Praefect as this will remain compatible with
      running concurrent processes.
      
      Changelog: added
      77c9b1eb7559
  11. May 17, 2023
  12. May 05, 2023
  13. May 04, 2023
  14. Apr 18, 2023
  15. Apr 03, 2023
  16. Mar 17, 2023
  17. Feb 22, 2023
    • Sami Hiltunen's avatar
      Match Gitaly configuration format in Omnibus · a0f05d22ef78
      Sami Hiltunen authored
      The configuration formats of Gitaly and Omnibus deviate from each
      other creating a few problems:
      
      1. Configuration keys for the same option differ between the context of
         Omnibus and Gitaly creating confusion for the user.
      
      2. Each new key added requires duplicate work to map the different keys
         into Omnibus.
      
      This change introduces gitaly['configuration'] matching configuration
      in Omnibus with Gitaly's own format. The previous configuration keys
      are remapped into this new format to assist administrators during the
      transition period.
      
      The configuration values are now serialized directly into
      TOML instead of using a template.
      
      Removes unneeded values allowing Gitaly's own source to be the single
      source of truth for configuration.
      
      Updates tests because the new serialization method does not output empty
      config sections.
      
      Guards are included to prevent errors with latency configurations that
      must be arrays where the previous configuration expected a string until
      such time as Gitaly's own validation subroutines are implemented.
      
      Changelog: changed
      a0f05d22ef78
  18. Feb 09, 2023
  19. Jan 25, 2023
  20. Jan 17, 2023
  21. Jan 16, 2023
    • Patrick Steinhardt's avatar
      Drop default values for `omnibus-gitconfig` · 2894e127b281
      Patrick Steinhardt authored
      The `Gitlab['omnibus-gitconfig']['system']` attribute used to contain
      Git configuration that was subsequently used by Gitaly. This was done by
      writing its config entries into the global `gitconfig` file so that they
      were picked up by all Git commands.
      
      This has proven to be insufficient in the context of Gitaly, which needs
      a lot more fine-grained control over the exact Git configuration it uses
      when spawning Git commands. One point in case was the deprecation of the
      old `core.fsyncObjetFiles` configuration, which if present caused us to
      print user-visible warnings.
      
      We have since moved all default Git configuration into Gitaly itself,
      which thus does not need any external Git configuration by default
      anymore. Furthermore, we have implemented a mechanism in Gitaly to just
      plain ignore the global `gitconfig` file -- instead, non-default Git
      configuration needs to be written into Gitaly's `config.toml`.
      
      In Omnibus though we still carry legacy defaults that were used for the
      Git configuration back when Gitaly didn't yet knew to set up its own
      defaults. As we know to strip those defaults when writing Gitaly's
      `config.toml` they don't apply at all in that context, but we still
      write them into the global `gitconfig` file. Consequentially, these
      defaults apply to all Git commands not spawned by Gitaly itself.
      Arguably though this is not required:
      
          - `pack.threads=1` controls how many packs should be used when
            writing Git packfiles. This was added to limit the amount of
            resources a single Git process could use in the context of Gitaly.
      
          - `receive.fsckObjects=true` asks Git to perform consistency checks
            for objects when serving a push. We should never accept pushes
            into Git repositories outside of Gitaly anyway.
      
          - `receive.advertisePushOptions=true` asks Git to advertise the
            availability of push options to clients when serving a push.
            Again, we should never accept pushes into Git repositories outside
            of Gitaly.
      
          - `transfer.hideRefs` hides various internal namespaces when serving
            fetches and pushes. Same as above, we should not accept pushes
            into Git repositories outside of Gitaly.
      
          - `repack.writeBitmaps=true` asks Git to write bitmaps via
            git-repack(1). This would even be wrong if git-repack(1) was to be
            executed in a repository that is connected to an object pool.
      
          - `fetch.writeCommitGraph=true` asks Git to write commit-graphs when
            fetching into a repository. This data structure is used to speed
            up various operations. There is no reason to enable this globally.
      
          - `core.alternateRefsCommand="exit 0 #"` will disable listing
            alternate references when a repository is connected to another
            repository. There is no reason to disable this globally.
      
          - `core.fsyncObjectFiles=true` is the only one that is debatable as
            it enables calling fsync(3P) when writing objects into a Git
            repository. This option has been deprecated in Git v2.36.0 though
            and causes deprecation warnings to be printed on every invocation
            of Git. So if at all, we would want to set `core.fsync` nowadays
            to enable this code again. On the other hand though there should
            be no manual commands that write Git objects into repositories
            controlled by Gitaly in the first place as that is not supported.
      
      Drop all the default values. All of these config entries are managed by
      Gitaly already, have no obvious purpose when running Git commands
      globally and add complexity and confusion around what the actual
      configuration is now.
      
      Changelog: removed
      2894e127b281
  22. Jan 11, 2023
    • Patrick Steinhardt's avatar
      Revert dropping of Ominbus gitconfig default values · 31d4c4ed5f2d
      Patrick Steinhardt authored
      This reverts 8f3d825f6 (Merge branch
      'pks-omnibus-gitconfig-drop-default-values' into 'master', 2023-01-11),
      which has dropped the gitconfig default values. This has caused a
      pipeline failure when packaging:
      
          ================================================================================
          Error executing action `create` on resource 'template[/opt/gitlab/embedded/etc/gitconfig]'
          ================================================================================
      
          Chef::Mixin::Template::TemplateError
          ------------------------------------
          undefined method `each' for nil:NilClass
      
          Resource Declaration:
          ---------------------
          # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb
      
           70: template "#{install_dir}/embedded/etc/gitconfig" do
           71:   source "gitconfig-system.erb"
           72:   mode 0755
           73:   variables gitconfig: node['gitlab']['omnibus-gitconfig']['system']
           74: end
           75:
      
      Changelog: fixed
      31d4c4ed5f2d
    • Patrick Steinhardt's avatar
      Drop default values for `omnibus-gitconfig` · e7a29dbc8231
      Patrick Steinhardt authored
      The `Gitlab['omnibus-gitconfig']['system']` attribute used to contain
      Git configuration that was subsequently used by Gitaly. This was done by
      writing its config entries into the global `gitconfig` file so that they
      were picked up by all Git commands.
      
      This has proven to be insufficient in the context of Gitaly, which needs
      a lot more fine-grained control over the exact Git configuration it uses
      when spawning Git commands. One point in case was the deprecation of the
      old `core.fsyncObjetFiles` configuration, which if present caused us to
      print user-visible warnings.
      
      We have since moved all default Git configuration into Gitaly itself,
      which thus does not need any external Git configuration by default
      anymore. Furthermore, we have implemented a mechanism in Gitaly to just
      plain ignore the global `gitconfig` file -- instead, non-default Git
      configuration needs to be written into Gitaly's `config.toml`.
      
      In Omnibus though we still carry legacy defaults that were used for the
      Git configuration back when Gitaly didn't yet knew to set up its own
      defaults. As we know to strip those defaults when writing Gitaly's
      `config.toml` they don't apply at all in that context, but we still
      write them into the global `gitconfig` file. Consequentially, these
      defaults apply to all Git commands not spawned by Gitaly itself.
      Arguably though this is not required:
      
          - `pack.threads=1` controls how many packs should be used when
            writing Git packfiles. This was added to limit the amount of
            resources a single Git process could use in the context of Gitaly.
      
          - `receive.fsckObjects=true` asks Git to perform consistency checks
            for objects when serving a push. We should never accept pushes
            into Git repositories outside of Gitaly anyway.
      
          - `receive.advertisePushOptions=true` asks Git to advertise the
            availability of push options to clients when serving a push.
            Again, we should never accept pushes into Git repositories outside
            of Gitaly.
      
          - `transfer.hideRefs` hides various internal namespaces when serving
            fetches and pushes. Same as above, we should not accept pushes
            into Git repositories outside of Gitaly.
      
          - `repack.writeBitmaps=true` asks Git to write bitmaps via
            git-repack(1). This would even be wrong if git-repack(1) was to be
            executed in a repository that is connected to an object pool.
      
          - `fetch.writeCommitGraph=true` asks Git to write commit-graphs when
            fetching into a repository. This data structure is used to speed
            up various operations. There is no reason to enable this globally.
      
          - `core.alternateRefsCommand="exit 0 #"` will disable listing
            alternate references when a repository is connected to another
            repository. There is no reason to disable this globally.
      
          - `core.fsyncObjectFiles=true` is the only one that is debatable as
            it enables calling fsync(3P) when writing objects into a Git
            repository. This option has been deprecated in Git v2.36.0 though
            and causes deprecation warnings to be printed on every invocation
            of Git. So if at all, we would want to set `core.fsync` nowadays
            to enable this code again. On the other hand though there should
            be no manual commands that write Git objects into repositories
            controlled by Gitaly in the first place as that is not supported.
      
      Drop all the default values. All of these config entries are managed by
      Gitaly already, have no obvious purpose when running Git commands
      globally and add complexity and confusion around what the actual
      configuration is now.
      
      Changelog: removed
      e7a29dbc8231
  23. Nov 30, 2022
    • Patrick Steinhardt's avatar
      gitaly: Allow passing gitconfig as separate section, subsection and key · 658300b371a8
      Patrick Steinhardt authored
      Right now, it's only possible for admins to specify the Gitaly gitconfig
      as a single key and value pair. While this works just fine in theory
      when you only consider this in the context of the `[[git.config]]`
      stanzas. We'll also migrate the global gitconfig from its current
      location in `omnibus_gitconfig['system']` to a new key though, and it
      would be great if we could use the same configuration format for both.
      But we'd have to continue to do laborious string manipulation if we
      wanted to use the key-value format for that new configuration as we
      don't write `[[git.config]]` stanzas there, but gitconfig files.
      
      Instead, introduce the ability to explicitly specify the Git config as a
      touple of sections (`http`), subsections (`"http://example.com"`) and
      keys (`insteadOf`) that together would form the complete key
      `http."http://example.com".insteadOf`. Like this, we have sufficient
      information to write both file formats without having to do any string
      conversations at all.
      
      In order to not break backwards compatibility with the just-introduced
      `gitaly['gitconfig']`, we retain the ability to just specify a key.
      
      Changelog: fixed
      658300b371a8
  24. Oct 17, 2022
  25. Sep 29, 2022
  26. Aug 23, 2022
    • Will Chandler's avatar
      Add support for Gitaly GPG signing · 5745b64829e3
      Will Chandler authored
      Gitaly recently added the option to GPG sign all commits created by
      GitLab, such as changes made via the WebIDE, merge commits, etc. This
      commit adds a new `gitaly['gpg_signing_key_path']` option to make the
      new setting accessible in Omnibus.
      
      Changelog: added
      5745b64829e3
  27. Aug 25, 2022
    • Stan Hu's avatar
      Improve error message when omnibus_gitconfig is not set properly · c2e48fd37d37
      Stan Hu authored
      Some users have configured `omnibus_gitconfig['system']` improperly,
      causing this obscure error:
      
      ```
      NoMethodError: undefined method `lstrip' for nil:NilClass
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitaly/libraries/gitaly.rb:93:in `block (2 levels) in parse_gitconfig'
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitaly/libraries/gitaly.rb:91:in `map'
      /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitaly/libraries/gitaly.rb:91:in `block in parse_gitconfig'
      ```
      
      This commit ensures each config line has a proper `key=value` form and
      displays a clearer message. For example:
      
      ```
      RuntimeError
      ------------
      Invalid entry detected in omnibus_gitconfig['system']: 'fsckObjects' should be in the form key=value
      ```
      
      Relates to https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6986
      
      Changelog: changed
      c2e48fd37d37
  28. Aug 05, 2022
    • Patrick Steinhardt's avatar
      gitaly: Unconditionally ignore the gitconfig · 098f8e5e2cbd
      Patrick Steinhardt authored
      In 4bf3bf85 (gitaly: Reintroduce migration to [[git.config]] stanzas,
      2022-07-13), we introduced a toggle telling Gitaly to ignore any
      gitconfig files on the local disk. Git configuration should only have
      one source of truth, and with this toggle enabled Gitaly only uses its
      own default values and overrides must be provided by the administrator
      through the new `[[git.config]]` stanzas.
      
      Because the initial attempt had to be reverted, this feature was rolled
      out to our production fleet gated behind an opt-in flag. We observed no
      change in behavior during testing, so this toggle may now be removed and
      local gitconfig files will be permanently ignored.
      
      Removes the flag and gitaly now ignores the gitconfig unconditionally.
      
      Closes https://gitlab.com/gitlab-org/gitaly/-/issues/4230
      
      Changelog: changed
      098f8e5e2cbd
  29. Jul 19, 2022
    • Stan Hu's avatar
      Reload Consul only if the service is enabled · 2717b10deb5d
      Stan Hu authored
      If for some reason an EE installation with Consul is downgraded to a
      CE installation, `gitlab-ctl reconfigure` will fail with an obscure
      message:
      
      ```
      NoMethodError occurred in delayed notification: undefined method
      `run_action' for "execute[reload consul]":String
      ```
      
      In a CE build, the Consul recipes are not loaded so `reload consul` is
      not available.
      
      To avoid this error, we can use the `Services.enabled?` helper, which
      will verify that Consul is in the list of available services in the
      first place.
      
      Relates to https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/6934
      
      Changelog: fixed
      2717b10deb5d
  30. Jul 15, 2022
    • Patrick Steinhardt's avatar
      global: Remove deprecated `self_signed_cert` setting · f2d02c8ba242
      Patrick Steinhardt authored
      The `self_signed_cert` setting that had been part of the gitlab-shell
      configuration has been deprecated in v14.8 and removed in v15.0 without
      any replacement. Administrators must instead make sure that they install
      any self-signed certificates into `/etc/gitlab/trusted-certs`.
      
      Follow through with the deprecation and remove the setting where it's
      still passed down. This most importantly also includes Gitaly and
      Praefect, which reuse the configuration injected via gitlab-shell. Given
      that we have already gone through the deprecation phase of this toggle
      it's not necessary to go through the same deprecation cycle again for
      these projects.
      
      Changelog: deprecated
      f2d02c8ba242
  31. Jul 13, 2022
    • Patrick Steinhardt's avatar
      gitaly: Reintroduce migration to `[[git.config]]` stanzas · e2ace401be94
      Patrick Steinhardt authored
      With commit 2727686d2 (gitaly: Migrate to inject Git configuration via
      `config.toml`, 2022-06-20), we have migrated the configuration for Git
      commands spawned by Gitaly from the system-level gitconfig file into
      Gitaly's `config.toml` via `[[git.config]]` stanzas. This change caused
      an incident because it change inadvertently fixed a configuration error:
      because we had globally been setting `core.alternateRefsCommand=exit 0#`
      in the gitconfig, Gitaly mistakenly ignored all alternate references in
      object pools. This is wrong though when calculating the repository size,
      where we need to consider alternate references in order to correctly
      calculate the size of objects that are only contained in the repository
      itself.
      
      But while the migration to `[[git.config]]` stanzas fixed this bug, it
      surfaced that the current implementation of how this is done in Gitaly
      is extremely inperformant in large repositories with many references.
      And this inefficiency ultimately end to starving the node of resources,
      which led to the revert of this change via a42c5611b (Merge branch
      'revert-fe00a13e' into 'master', 2022-06-20).
      
      Gitaly has since fixed this performance edge case so that we shouldn't
      hit the same bug this time. So let's reintroduce the migration to the
      new way of configuring Git so that we can also fix the bug about repo
      size calculations.
      
      Note that this commit is a simple revert. This time around we'll also
      introduce a new flag to opt-in to this migration so that we can first
      excessively test it in staging without also impacting production. To
      keep this as a clean revert though this change will be done in the
      subsequent commit.
      
      Changelog: added
      e2ace401be94
  32. Jun 20, 2022
    • Stan Hu's avatar
      Revert "Merge branch 'pks-gitaly-stop-writing-gitconfig' into 'master'" · fa42a9ef3258
      Stan Hu authored
      This reverts merge request !6128
      fa42a9ef3258
    • Patrick Steinhardt's avatar
      gitaly: Migrate to inject Git configuration via `config.toml` · 38119289d47a
      Patrick Steinhardt authored
      Currently, there is a single `gitlab['omnibus-gitconfig']` setting that
      controls the Git configuration across different components of the stack
      by writing either a system-level or global-level gitconfig file. This
      makes it hard to see which parts of the gitconfig are used by what part
      of the stack.
      
      For Gitaly the story is a bit complicated: the gitconfig files are in
      fact read, but almost all of its values are ignored because Gitaly uses
      its own set of defaults which override them, which is really confusing
      for administrators. Gitaly has instead introduced a new configuration
      `[[git.config]]` that needs to be added to its `config.toml` file, and
      this configuration does indeed override Gitaly's default configuration.
      We do not currently expose this setting in Omnibus though. To lessen the
      confusion what's read and what isn't, Gitaly will start to ignore all
      gitconfig files with v16.0.
      
      Prepare for this deprecation by migrating from writing gitconfig files
      to instead writing `[[git.config]]` sections. Given that other parts of
      the stack may still be reading the old gitconfig files we cannot just
      stop writing gitconfig files. Instead, this commit carves out Gitaly's
      own gitconfig so that it can be configured independently of the rest of
      the stack.
      
      Due to backwards-compatibility reasons we need to migrate old settings
      that had been set in `omnibus_gitconfig['system']` automatically to the
      new `gitaly['gitconfig']` configuration until v16.0 has been released.
      
      This migration is further complicated by a eprecation in Git v2.36.0
      itself though: if the old `core.fsyncObjectFiles` configuration is set,
      then Git will print a user-visible warning every time it's executed.
      Unfortunately though, administrators were required to set this config in
      the past, and we thus advertised this configuration entry as part of the
      default configuration. So we must ensure that this old default key is
      not set in the new configuration: in fact, we don't want any of the
      previously set defaults to end up in the new configuration so that
      Gitaly is the single source of truth with regards to what the defaults
      are. So in the general case, `gitaly['gitconfig']` should be empty.
      
      To work around this issue we thus automatically migrate configuration in
      `omnibus_gitconfig['system']`, but strip out any of the previous default
      values. The end result is thus a configuration that only contains what
      the administrator has manually overridden.
      
      Changelog: changed
      38119289d47a
  33. May 16, 2022
    • Patrick Steinhardt's avatar
      gitaly: Remove configuration for Rugged's gitconfig search path · 517a05bf8e59
      Patrick Steinhardt authored
      Gitaly is still using Rugged in its Ruby sidecar to provide a very
      limited set of RPCs that have not yet been ported to Go. While Rugged in
      the past required the gitconfig file to function correctly, nowadays it
      only requires the `core.fsyncObjectFiles` config entry. This entry must
      be set to `true` or otherwise it is trivial to end up with repository
      corruption.
      
      This means that the gitconfig must be in a specific well-known state for
      Rugged to operate correctly. But currently there is still the option to
      set Rugged's gitconfig search path, which must in fact point to a path
      that contains a gitconfig with the mentioned config entry. This is quite
      fragile without any benefit for the administrator at all.
      
      To fix this, Gitaly has introduced a fallback mode via [1] that causes
      it to write a Rugged-specific gitconfig file in case the Rugged search
      path is not set up. This allows us to now phase out support for setting
      this configuration altogether: it's not required, may cause repository
      corruption if misconfigured, and ultimately only makes the architecture
      harder to reason about given that we currently share this gitconfig file
      between multiple components.
      
      Remove the ability to set Rugged's gitconfig search path without any
      replacement.
      
      [1]: https://gitlab.com/gitlab-org/gitaly/-/merge_requests/4560
      
      Changelog: removed
      517a05bf8e59
  34. May 24, 2022
  35. Apr 20, 2022
  36. Apr 27, 2022
Loading