Skip to content
Snippets Groups Projects
troubleshooting.md 46.04 KiB
stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments

Troubleshooting Omnibus GitLab installation issues

DETAILS: Tier: Free, Premium, Ultimate Offering: Self-managed

Use this page to learn about common issues users can encounter when installing Omnibus GitLab packages.

Hash Sum mismatch when downloading packages

apt-get install outputs something like:

E: Failed to fetch https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu/pool/trusty/main/g/gitlab-ce/gitlab-ce_8.1.0-ce.0_amd64.deb  Hash Sum mismatch

Run the following to fix this:

sudo rm -rf /var/lib/apt/lists/partial/*
sudo apt-get update
sudo apt-get clean

See Joe Damato's from Packagecloud comment and his blog article for more context.

Another workaround is to download the package manually by selecting the correct package from the CE packages or EE packages repository:

curl -LJO "https://packages.gitlab.com/gitlab/gitlab-ce/packages/ubuntu/trusty/gitlab-ce_8.1.0-ce.0_amd64.deb/download"
dpkg -i gitlab-ce_8.1.0-ce.0_amd64.deb

Installation on openSUSE and SLES platforms warns about unknown key signature

Omnibus GitLab packages are signed with GPG keys in addition to the package repositories providing signed metadata. This ensures authenticity and integrity of the packages that are distributed to the users. However, the package manager used in openSUSE and SLES operating systems may sometime raise false warnings with these signatures, similar to

File 'repomd.xml' from repository 'gitlab_gitlab-ce' is signed with an unknown key '14219A96E15E78F4'. Continue? [yes/no] (no):
File 'repomd.xml' from repository 'gitlab_gitlab-ce' is signed with an unknown key '14219A96E15E78F4'. Continue? [yes/no] (no): yes

This is a known bug with zypper where zypper ignores the gpgkey keyword in the repository configuration file. With later versions of Packagecloud, there may be improvements regarding this, but currently users have to manually agree to package installation.

So, in openSUSE or SLES systems, if such a warning is displayed, it is safe to continue installation.

apt/yum complains about GPG signatures

You already have GitLab repositories configured, and ran apt-get update, apt-get install or yum install, and saw errors like the following:

The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY 3F01618A51312F3F

or

https://packages.gitlab.com/gitlab/gitlab-ee/el/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml signature could not be verified for gitlab-ee

This is because on April 2020, GitLab changed the GPG keys used to sign metadata of the apt and yum repositories available through the Packagecloud instance. If you see this error, it generally means you do not have the public keys currently used to sign repository metadata in your keyring. To fix this error, follow the steps to fetch the new key.

Reconfigure shows an error: NoMethodError - undefined method '[]=' for nil:NilClass

You ran sudo gitlab-ctl reconfigure or package upgrade triggered the reconfigure which produced error similar to:

 ================================================================================
 Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb
 ================================================================================

NoMethodError
-------------
undefined method '[]=' for nil:NilClass

Cookbook Trace:
---------------
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/config.rb:21:in 'from_file'
  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:26:in 'from_file'

Relevant File Content:

This error is thrown when /etc/gitlab/gitlab.rb configuration file contains configuration that is invalid or unsupported. Double check that there are no typos or that the configuration file does not contain obsolete configuration.

You can check the latest available configuration by using sudo gitlab-ctl diff-config or check the latest gitlab.rb.template.

GitLab is unreachable in my browser

Try specifying an external_url in /etc/gitlab/gitlab.rb. Also check your firewall settings; port 80 (HTTP) or 443 (HTTPS) might be closed on your GitLab server.

Note that specifying the external_url for GitLab, or any other bundled service (Registry and Mattermost) doesn't follow the key=value format that other parts of gitlab.rb follow. Make sure that you have them set in the following format:

external_url "https://gitlab.example.com"
registry_external_url "https://registry.example.com"
mattermost_external_url "https://mattermost.example.com"

NOTE: Don't add the equal sign (=) between external_url and the value.

Emails are not being delivered

To test email delivery you can create a new GitLab account for an email that is not used in your GitLab instance yet.

If necessary, you can modify the 'From' field of the emails sent by GitLab with the following setting in /etc/gitlab/gitlab.rb:

gitlab_rails['gitlab_email_from'] = 'gitlab@example.com'

Run sudo gitlab-ctl reconfigure for the change to take effect.

TCP ports for GitLab services are already taken

By default, Puma listens at TCP address 127.0.0.1:8080. NGINX listens on port 80 (HTTP) and/or 443 (HTTPS) on all interfaces.

The ports for Redis, PostgreSQL and Puma can be overridden in /etc/gitlab/gitlab.rb as follows:

redis['port'] = 1234
postgresql['port'] = 2345
puma['port'] = 3456

For NGINX port changes please see Setting the NGINX listen port.

Git user does not have SSH access

SELinux-enabled systems

On SELinux-enabled systems the Git user's .ssh directory or its contents can get their security context messed up. You can fix this by running sudo gitlab-ctl reconfigure, which sets the gitlab_shell_t security context on /var/opt/gitlab/.ssh.

To improve this behavior, we set the context permanently using semanage. The runtime dependency policycoreutils-python has been added to the RPM package for RHEL based operating systems in order to ensure the semanage command is available.

Diagnose and resolve SELinux issues

Omnibus GitLab detects default path changes in /etc/gitlab/gitlab.rb and should apply the correct file contexts.

NOTE: From GitLab 16.10 forward, administrators can try gitlab-ctl apply-sepolicy to automatically fix SELinux issues. Consult gitlab-ctl apply-sepolicy --help for runtime options.

For installations using custom data path configuration, the administrator may have to manually resolve SELinux issues.

Data paths may be altered via gitlab.rb, however, a common scenario forces the use of symlink paths. Administrators should be cautious, because symlink paths are not supported for all scenarios, such as Gitaly data paths.

For example, if /data/gitlab replaced /var/opt/gitlab as the base data directory, the following fixes the security context:

sudo semanage fcontext -a -t gitlab_shell_t /data/gitlab/.ssh/
sudo semanage fcontext -a -t gitlab_shell_t /data/gitlab/.ssh/authorized_keys
sudo restorecon -Rv /data/gitlab/
sudo semanage fcontext -a -t gitlab_shell_t /data/gitlab/gitlab-shell/config.yml
sudo restorecon -Rv /data/gitlab/gitlab-shell/
sudo semanage fcontext -a -t gitlab_shell_t /data/gitlab/gitlab-rails/etc/gitlab_shell_secret
sudo restorecon -Rv /data/gitlab/gitlab-rails/
sudo semanage fcontext --list | grep /data/gitlab/

After the policies are applied, you can verify the SSH access is working by getting the welcome message:

ssh -T git@gitlab-hostname

All systems

The Git user is created, by default, with a locked password, shown by '!' in /etc/shadow. Unless "UsePam yes" is enabled, the OpenSSH daemon prevents the Git user from authenticating even with SSH keys. An alternative secure solution is to unlock the password by replacing '!' with '*' in /etc/shadow. The Git user is still unable to change the password because it runs in a restricted shell and the passwd command for non-superusers requires entering the current password prior to a new password. The user cannot enter a password that matches '*', which means the account continues to not have a password.

Keep in mind that the Git user must have access to the system so please review your security settings at /etc/security/access.conf and make sure the Git user is not blocked.

PostgreSQL error FATAL: could not create shared memory segment: Cannot allocate memory

The packaged PostgreSQL instance tries to allocate 25% of total memory as shared memory. On some Linux (virtual) servers, there is less shared memory available, which prevents PostgreSQL from starting. In /var/log/gitlab/postgresql/current:

  1885  2014-08-08_16:28:43.71000 FATAL:  could not create shared memory segment: Cannot allocate memory
  1886  2014-08-08_16:28:43.71002 DETAIL:  Failed system call was shmget(key=5432001, size=1126563840, 03600).
  1887  2014-08-08_16:28:43.71003 HINT:  This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory or swap space, or exceeded your kernel's SHMALL parameter.  You can either reduce the request size or reconfigure the kernel with larger SHMALL.  To reduce the request size (currently 1126563840 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections.
  1888  2014-08-08_16:28:43.71004       The PostgreSQL documentation contains more information about shared memory configuration.

You can manually lower the amount of shared memory PostgreSQL tries to allocate in /etc/gitlab/gitlab.rb:

postgresql['shared_buffers'] = "100MB"

Run sudo gitlab-ctl reconfigure for the change to take effect.

PostgreSQL error FATAL: could not open shared memory segment "/PostgreSQL.XXXXXXXXXX": Permission denied

By default, PostgreSQL tries to detect the shared memory type to use. If you don't have shared memory enabled, you might see this error in /var/log/gitlab/postgresql/current. To fix this, you can disable PostgreSQL's shared memory detection. Set the following value in /etc/gitlab/gitlab.rb:

postgresql['dynamic_shared_memory_type'] = 'none'

Run sudo gitlab-ctl reconfigure for the change to take effect.

PostgreSQL error FATAL: remaining connection slots are reserved for non-replication superuser connections

PostgreSQL has a setting for the maximum number of the concurrent connections to the database server. If you see this error, it means that your GitLab instance is trying to exceed this limit on the number of concurrent connections.

To fix this problem, you have two options:

  • Either increase the max connections value:

    1. Edit /etc/gitlab/gitlab.rb:

      postgresql['max_connections'] = 600
    2. Reconfigure GitLab:

      sudo gitlab-ctl reconfigure
  • Or, you can consider using PgBouncer which is a connection pooler for PostgreSQL.

Reconfigure complains about the GLIBC version

$ gitlab-ctl reconfigure

/opt/gitlab/embedded/bin/ruby: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /opt/gitlab/embedded/lib/libruby.so.2.1)
/opt/gitlab/embedded/bin/ruby: /lib64/libc.so.6: version `GLIBC_2.17' not found (required by /opt/gitlab/embedded/lib/libruby.so.2.1)

This can happen if the omnibus package you installed was built for a different OS release than the one on your server. Double-check that you downloaded and installed the correct Omnibus GitLab package for your operating system.

Reconfigure fails to create the Git user

This can happen if you run sudo gitlab-ctl reconfigure as the Git user. Switch to another user.

More importantly: do not give sudo rights to the Git user or to any of the other users used by Omnibus GitLab. Bestowing unnecessary privileges on a system user weakens the security of your system.

Failed to modify kernel parameters with sysctl

If sysctl cannot modify the kernel parameters you could possibly get an error with the following stack trace:

 * execute[sysctl] action run
================================================================================
Error executing action `run` on resource 'execute[sysctl]'
================================================================================


Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '255'
---- Begin output of /sbin/sysctl -p /etc/sysctl.conf ----

This is unlikely to happen with non virtualized machines but on a VPS with virtualization like openVZ, container might not have the required module enabled or container doesn't have access to kernel parameters.

Try enabling the module on which sysctl errored out.

There is a reported workaround described in this issue which requires editing the GitLab' internal recipe by supplying the switch which ignores failures. Ignoring errors can have unexpected side effects on the performance of your GitLab server, so it isn't recommended to do so.

Another variation of this error reports the file system is read-only and shows following stack trace:

 * execute[load sysctl conf] action run
    [execute] sysctl: setting key "kernel.shmall": Read-only file system
              sysctl: setting key "kernel.shmmax": Read-only file system

    ================================================================================
    Error executing action `run` on resource 'execute[load sysctl conf]'
    ================================================================================

    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    Expected process to exit with [0], but received '255'
    ---- Begin output of cat /etc/sysctl.conf /etc/sysctl.d/*.conf  | sysctl -e -p - ----
    STDOUT:
    STDERR: sysctl: setting key "kernel.shmall": Read-only file system
    sysctl: setting key "kernel.shmmax": Read-only file system
    ---- End output of cat /etc/sysctl.conf /etc/sysctl.d/*.conf  | sysctl -e -p - ----
    Ran cat /etc/sysctl.conf /etc/sysctl.d/*.conf  | sysctl -e -p - returned 255

This error is also reported to occur in virtual machines only, and the recommended workaround is to set the values in the host. The values needed for GitLab can be found inside the file /opt/gitlab/embedded/etc/90-omnibus-gitlab.conf in the virtual machine. After setting these values in /etc/sysctl.conf file in the host OS, run cat /etc/sysctl.conf /etc/sysctl.d/*.conf | sysctl -e -p - on the host. Then try running gitlab-ctl reconfigure inside the virtual machine. It should detect that the kernel is already running with the necessary settings, and not raise any errors.

You may have to repeat this process for other lines. For example, reconfigure fails three times, after having added something like this to /etc/sysctl.conf:

kernel.shmall = 4194304
kernel.sem = 250 32000 32 262
net.core.somaxconn = 2048
kernel.shmmax = 17179869184

You may find it easier to look at the line in the Chef output than to find the file (since the file is different for each error). See the last line of this snippet.

* file[create /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmall.conf kernel.shmall] action create
  - create new file /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmall.conf
  - update content in file /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmall.conf from none to 6d765d
  --- /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmall.conf 2017-11-28 19:09:46.864364952 +0000
  +++ /opt/gitlab/embedded/etc/.chef-90-omnibus-gitlab-kernel.shmall.conf kernel.shmall20171128-13622-sduqoj 2017-11-28 19:09:46.864364952 +0000
  @@ -1 +1,2 @@
  +kernel.shmall = 4194304

I am unable to install Omnibus GitLab without root access

Occasionally people ask if they can install GitLab without root access. This is problematic for several reasons.

Installing the .deb or .rpm

To our knowledge there is no clean way to install Debian or RPM packages as a non-privileged user. You cannot install Omnibus GitLab RPM's because the Omnibus build process does not create source RPM's.

Hassle-free hosting on port 80 and 443

The most common way to deploy GitLab is to have a web server (NGINX/Apache) running on the same server as GitLab, with the web server listening on a privileged (below-1024) TCP port. In Omnibus GitLab we provide this convenience by bundling an automatically configured NGINX service that needs to run its master process as root to open ports 80 and 443.

If this is problematic, administrators installing GitLab can disable the bundled NGINX service, but this puts the burden on them to keep the NGINX configuration in tune with GitLab during application updates.

Isolation between Omnibus services

Bundled services in Omnibus GitLab (GitLab itself, NGINX, PostgreSQL, Redis, Mattermost) are isolated from each other using Unix user accounts. Creating and managing these user accounts requires root access. By default, Omnibus GitLab will create the required Unix accounts during gitlab-ctl reconfigure but that behavior can be disabled.

In principle Omnibus GitLab could do with only 2 user accounts (one for GitLab and one for Mattermost) if we give each application its own runit (runsvdir), PostgreSQL and Redis process. But this would be a major change in the gitlab-ctl reconfigure Chef code and it would probably create major upgrade pain for all existing Omnibus GitLab installations. (We would probably have to rearrange the directory structure under /var/opt/gitlab.)

Tweaking the operating system for better performance

During gitlab-ctl reconfigure we set and install several sysctl tweaks to improve PostgreSQL performance and increase connection limits. This can only be done with root access.

gitlab-rake assets:precompile fails with 'Permission denied'

Some users report that running gitlab-rake assets:precompile does not work with the omnibus packages. The short answer to this is: do not run that command, it is only for GitLab installations from source.

The GitLab web interface uses CSS and JavaScript files, called 'assets' in Ruby on Rails-speak. In the upstream GitLab repository these files are stored in a developer-friendly way: easy to read and edit. When you are a normal user of GitLab, you do not want these files to be in the developer friendly format however because that makes GitLab slow. This is why part of the GitLab setup process is to convert the assets from a developer-friendly format to an end-user friendly (compact, fast) format; that is what the rake assets:precompile script is for.

When you install GitLab from source (which was the only way to do it before we had omnibus packages) you need to convert the assets on your GitLab server every time you update GitLab. People used to overlook this step and there are still posts, comments and mails out there on the internet where users recommend each other to run rake assets:precompile (which has now been renamed gitlab:assets:compile). With the omnibus packages things are different: when we build the package we compile the assets for you. When you install GitLab with an omnibus package, the converted assets are already there! That is why you do not need to run rake assets:precompile when you install GitLab from a package.

When gitlab-rake assets:precompile fails with a permission error it fails for a good reason from a security standpoint: the fact that the assets cannot easily be rewritten makes it harder for an attacker to use your GitLab server to serve evil JavaScript code to the visitors of your GitLab server.

If you want to run GitLab with custom JavaScript or CSS code you are probably better off running GitLab from source, or building your own packages.

If you really know what you are doing, you can execute gitlab-rake gitlab:assets:compile like this:

sudo NO_PRIVILEGE_DROP=true USE_DB=false gitlab-rake gitlab:assets:clean gitlab:assets:compile
# user and path might be different if you changed the defaults of
# user['username'], user['group'] and gitlab_rails['dir'] in gitlab.rb
sudo chown -R git:git /var/opt/gitlab/gitlab-rails/tmp/cache

'Short read or OOM loading DB' error

Try cleaning the old Redis session.

Apt error 'The requested URL returned error: 403'

When trying to install GitLab using the apt repo if you receive an error similar to:

W: Failed to fetch https://packages.gitlab.com/gitlab/gitlab-ce/DISTRO/dists/CODENAME/main/source/Sources  The requested URL returned error: 403

check if there is a repository cacher in front of your server, like for example apt-cacher-ng.

Add the following line to apt-cacher-ng config(eg. in /etc/apt-cacher-ng/acng.conf):

PassThroughPattern: (packages\.gitlab\.com|packages-gitlab-com\.s3\.amazonaws\.com|*\.cloudfront\.net)