stage: Systems
group: Distribution
info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments
Troubleshooting Omnibus GitLab installation issues
DETAILS: Tier: Free, Premium, Ultimate Offering: Self-managed
Use this page to learn about common issues users can encounter when installing Omnibus GitLab packages.
Hash Sum mismatch when downloading packages
apt-get install
outputs something like:
E: Failed to fetch https://packages.gitlab.com/gitlab/gitlab-ce/ubuntu/pool/trusty/main/g/gitlab-ce/gitlab-ce_8.1.0-ce.0_amd64.deb Hash Sum mismatch
Run the following to fix this:
sudo rm -rf /var/lib/apt/lists/partial/*
sudo apt-get update
sudo apt-get clean
See Joe Damato's from Packagecloud comment and his blog article for more context.
Another workaround is to download the package manually by selecting the correct package from the CE packages or EE packages repository:
curl -LJO "https://packages.gitlab.com/gitlab/gitlab-ce/packages/ubuntu/trusty/gitlab-ce_8.1.0-ce.0_amd64.deb/download"
dpkg -i gitlab-ce_8.1.0-ce.0_amd64.deb
Installation on openSUSE and SLES platforms warns about unknown key signature
Omnibus GitLab packages are signed with GPG keys in addition to the package repositories providing signed metadata. This ensures authenticity and integrity of the packages that are distributed to the users. However, the package manager used in openSUSE and SLES operating systems may sometime raise false warnings with these signatures, similar to
File 'repomd.xml' from repository 'gitlab_gitlab-ce' is signed with an unknown key '14219A96E15E78F4'. Continue? [yes/no] (no):
File 'repomd.xml' from repository 'gitlab_gitlab-ce' is signed with an unknown key '14219A96E15E78F4'. Continue? [yes/no] (no): yes
This is a known bug with zypper where zypper ignores the gpgkey
keyword in the
repository configuration file. With later versions of Packagecloud, there may be
improvements regarding this, but currently users have to manually agree to
package installation.
So, in openSUSE or SLES systems, if such a warning is displayed, it is safe to continue installation.
apt/yum complains about GPG signatures
You already have GitLab repositories configured, and ran apt-get update
,
apt-get install
or yum install
, and saw errors like the following:
The following signatures couldn’t be verified because the public key is not available: NO_PUBKEY 3F01618A51312F3F
or
https://packages.gitlab.com/gitlab/gitlab-ee/el/7/x86_64/repodata/repomd.xml: [Errno -1] repomd.xml signature could not be verified for gitlab-ee
This is because on April 2020, GitLab changed the GPG keys used to sign metadata of the apt and yum repositories available through the Packagecloud instance. If you see this error, it generally means you do not have the public keys currently used to sign repository metadata in your keyring. To fix this error, follow the steps to fetch the new key.
NoMethodError - undefined method '[]=' for nil:NilClass
Reconfigure shows an error: You ran sudo gitlab-ctl reconfigure
or package upgrade triggered the
reconfigure which produced error similar to:
================================================================================
Recipe Compile Error in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb
================================================================================
NoMethodError
-------------
undefined method '[]=' for nil:NilClass
Cookbook Trace:
---------------
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/config.rb:21:in 'from_file'
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/default.rb:26:in 'from_file'
Relevant File Content:
This error is thrown when /etc/gitlab/gitlab.rb
configuration file contains
configuration that is invalid or unsupported. Double check that there are no
typos or that the configuration file does not contain obsolete configuration.
You can check the latest available configuration by using sudo gitlab-ctl diff-config
or check the latest gitlab.rb.template
.
GitLab is unreachable in my browser
Try specifying an external_url
in
/etc/gitlab/gitlab.rb
. Also check your firewall settings; port 80 (HTTP) or
443 (HTTPS) might be closed on your GitLab server.
Note that specifying the external_url
for GitLab, or any other bundled service (Registry and
Mattermost) doesn't follow the key=value
format that other parts of gitlab.rb
follow. Make sure
that you have them set in the following format:
external_url "https://gitlab.example.com"
registry_external_url "https://registry.example.com"
mattermost_external_url "https://mattermost.example.com"
NOTE:
Don't add the equal sign (=
) between external_url
and the value.
Emails are not being delivered
To test email delivery you can create a new GitLab account for an email that is not used in your GitLab instance yet.
If necessary, you can modify the 'From' field of the emails sent by GitLab with
the following setting in /etc/gitlab/gitlab.rb
:
gitlab_rails['gitlab_email_from'] = 'gitlab@example.com'
Run sudo gitlab-ctl reconfigure
for the change to take effect.
TCP ports for GitLab services are already taken
By default, Puma listens at TCP address 127.0.0.1:8080. NGINX listens on port 80 (HTTP) and/or 443 (HTTPS) on all interfaces.
The ports for Redis, PostgreSQL and Puma can be overridden in
/etc/gitlab/gitlab.rb
as follows:
redis['port'] = 1234
postgresql['port'] = 2345
puma['port'] = 3456
For NGINX port changes please see Setting the NGINX listen port.
Git user does not have SSH access
SELinux-enabled systems
On SELinux-enabled systems the Git user's .ssh
directory or its contents can
get their security context messed up. You can fix this by running sudo gitlab-ctl reconfigure
, which sets the gitlab_shell_t
security context on
/var/opt/gitlab/.ssh
.
To improve this behavior, we set the context permanently using
semanage
. The runtime dependency policycoreutils-python
has been added to the
RPM package for RHEL based operating systems in order to ensure the semanage
command is available.
Diagnose and resolve SELinux issues
Omnibus GitLab detects default path changes in /etc/gitlab/gitlab.rb
and should apply
the correct file contexts.
NOTE:
From GitLab 16.10 forward, administrators can try gitlab-ctl apply-sepolicy
to automatically fix SELinux issues. Consult
gitlab-ctl apply-sepolicy --help
for runtime options.
For installations using custom data path configuration, the administrator may have to manually resolve SELinux issues.
Data paths may be altered via gitlab.rb
, however, a common scenario forces the
use of symlink
paths. Administrators should be cautious, because symlink
paths are
not supported for all scenarios, such as Gitaly data paths.
For example, if /data/gitlab
replaced /var/opt/gitlab
as the base data directory, the following fixes the security context:
sudo semanage fcontext -a -t gitlab_shell_t /data/gitlab/.ssh/
sudo semanage fcontext -a -t gitlab_shell_t /data/gitlab/.ssh/authorized_keys
sudo restorecon -Rv /data/gitlab/
sudo semanage fcontext -a -t gitlab_shell_t /data/gitlab/gitlab-shell/config.yml
sudo restorecon -Rv /data/gitlab/gitlab-shell/
sudo semanage fcontext -a -t gitlab_shell_t /data/gitlab/gitlab-rails/etc/gitlab_shell_secret
sudo restorecon -Rv /data/gitlab/gitlab-rails/
sudo semanage fcontext --list | grep /data/gitlab/
After the policies are applied, you can verify the SSH access is working by getting the welcome message:
ssh -T git@gitlab-hostname
All systems
The Git user is created, by default, with a locked password, shown by '!'
in
/etc/shadow. Unless "UsePam yes" is enabled, the OpenSSH daemon prevents the
Git user from authenticating even with SSH keys. An alternative secure solution
is to unlock the password by replacing '!'
with '*'
in /etc/shadow
. The Git
user is still unable to change the password because it runs in a restricted
shell and the passwd
command for non-superusers requires entering the current
password prior to a new password. The user cannot enter a password that matches
'*'
, which means the account continues to not have a password.
Keep in mind that the Git user must have access to the system so please review
your security settings at /etc/security/access.conf
and make sure the Git user
is not blocked.
FATAL: could not create shared memory segment: Cannot allocate memory
PostgreSQL error The packaged PostgreSQL instance tries to allocate 25% of total memory as
shared memory. On some Linux (virtual) servers, there is less shared memory
available, which prevents PostgreSQL from starting. In
/var/log/gitlab/postgresql/current
:
1885 2014-08-08_16:28:43.71000 FATAL: could not create shared memory segment: Cannot allocate memory
1886 2014-08-08_16:28:43.71002 DETAIL: Failed system call was shmget(key=5432001, size=1126563840, 03600).
1887 2014-08-08_16:28:43.71003 HINT: This error usually means that PostgreSQL's request for a shared memory segment exceeded available memory or swap space, or exceeded your kernel's SHMALL parameter. You can either reduce the request size or reconfigure the kernel with larger SHMALL. To reduce the request size (currently 1126563840 bytes), reduce PostgreSQL's shared memory usage, perhaps by reducing shared_buffers or max_connections.
1888 2014-08-08_16:28:43.71004 The PostgreSQL documentation contains more information about shared memory configuration.
You can manually lower the amount of shared memory PostgreSQL tries to allocate
in /etc/gitlab/gitlab.rb
:
postgresql['shared_buffers'] = "100MB"
Run sudo gitlab-ctl reconfigure
for the change to take effect.
FATAL: could not open shared memory segment "/PostgreSQL.XXXXXXXXXX": Permission denied
PostgreSQL error By default, PostgreSQL tries to detect the shared memory type to use. If you don't
have shared memory enabled, you might see this error in /var/log/gitlab/postgresql/current
.
To fix this, you can disable PostgreSQL's shared memory detection. Set the
following value in /etc/gitlab/gitlab.rb
:
postgresql['dynamic_shared_memory_type'] = 'none'
Run sudo gitlab-ctl reconfigure
for the change to take effect.
FATAL: remaining connection slots are reserved for non-replication superuser connections
PostgreSQL error PostgreSQL has a setting for the maximum number of the concurrent connections to the database server. If you see this error, it means that your GitLab instance is trying to exceed this limit on the number of concurrent connections.
To fix this problem, you have two options:
-
Either increase the max connections value:
-
Edit
/etc/gitlab/gitlab.rb
:postgresql['max_connections'] = 600
-
Reconfigure GitLab:
sudo gitlab-ctl reconfigure
-
-
Or, you can consider using PgBouncer which is a connection pooler for PostgreSQL.
Reconfigure complains about the GLIBC version
$ gitlab-ctl reconfigure
/opt/gitlab/embedded/bin/ruby: /lib64/libc.so.6: version `GLIBC_2.14' not found (required by /opt/gitlab/embedded/lib/libruby.so.2.1)
/opt/gitlab/embedded/bin/ruby: /lib64/libc.so.6: version `GLIBC_2.17' not found (required by /opt/gitlab/embedded/lib/libruby.so.2.1)
This can happen if the omnibus package you installed was built for a different OS release than the one on your server. Double-check that you downloaded and installed the correct Omnibus GitLab package for your operating system.
Reconfigure fails to create the Git user
This can happen if you run sudo gitlab-ctl reconfigure
as the Git user.
Switch to another user.
More importantly: do not give sudo rights to the Git user or to any of the other users used by Omnibus GitLab. Bestowing unnecessary privileges on a system user weakens the security of your system.
Failed to modify kernel parameters with sysctl
If sysctl cannot modify the kernel parameters you could possibly get an error with the following stack trace:
* execute[sysctl] action run
================================================================================
Error executing action `run` on resource 'execute[sysctl]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '255'
---- Begin output of /sbin/sysctl -p /etc/sysctl.conf ----
This is unlikely to happen with non virtualized machines but on a VPS with virtualization like openVZ, container might not have the required module enabled or container doesn't have access to kernel parameters.
Try enabling the module on which sysctl errored out.
There is a reported workaround described in this issue which requires editing the GitLab' internal recipe by supplying the switch which ignores failures. Ignoring errors can have unexpected side effects on the performance of your GitLab server, so it isn't recommended to do so.
Another variation of this error reports the file system is read-only and shows following stack trace:
* execute[load sysctl conf] action run
[execute] sysctl: setting key "kernel.shmall": Read-only file system
sysctl: setting key "kernel.shmmax": Read-only file system
================================================================================
Error executing action `run` on resource 'execute[load sysctl conf]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '255'
---- Begin output of cat /etc/sysctl.conf /etc/sysctl.d/*.conf | sysctl -e -p - ----
STDOUT:
STDERR: sysctl: setting key "kernel.shmall": Read-only file system
sysctl: setting key "kernel.shmmax": Read-only file system
---- End output of cat /etc/sysctl.conf /etc/sysctl.d/*.conf | sysctl -e -p - ----
Ran cat /etc/sysctl.conf /etc/sysctl.d/*.conf | sysctl -e -p - returned 255
This error is also reported to occur in virtual machines only, and the recommended workaround is to set the values in the host. The values needed for GitLab can be found inside the file /opt/gitlab/embedded/etc/90-omnibus-gitlab.conf
in the virtual machine. After setting these values in /etc/sysctl.conf
file in the host OS, run cat /etc/sysctl.conf /etc/sysctl.d/*.conf | sysctl -e -p -
on the host. Then try running gitlab-ctl reconfigure
inside the virtual machine. It should detect that the kernel is already running with the necessary settings, and not raise any errors.
You may have to repeat this process for other lines. For example, reconfigure fails three times, after having added something like this to /etc/sysctl.conf
:
kernel.shmall = 4194304
kernel.sem = 250 32000 32 262
net.core.somaxconn = 2048
kernel.shmmax = 17179869184
You may find it easier to look at the line in the Chef output than to find the file (since the file is different for each error). See the last line of this snippet.
* file[create /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmall.conf kernel.shmall] action create
- create new file /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmall.conf
- update content in file /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmall.conf from none to 6d765d
--- /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmall.conf 2017-11-28 19:09:46.864364952 +0000
+++ /opt/gitlab/embedded/etc/.chef-90-omnibus-gitlab-kernel.shmall.conf kernel.shmall20171128-13622-sduqoj 2017-11-28 19:09:46.864364952 +0000
@@ -1 +1,2 @@
+kernel.shmall = 4194304
I am unable to install Omnibus GitLab without root access
Occasionally people ask if they can install GitLab without root access. This is problematic for several reasons.
Installing the .deb or .rpm
To our knowledge there is no clean way to install Debian or RPM packages as a non-privileged user. You cannot install Omnibus GitLab RPM's because the Omnibus build process does not create source RPM's.
Hassle-free hosting on port 80 and 443
The most common way to deploy GitLab is to have a web server (NGINX/Apache) running on the same server as GitLab, with the web server listening on a privileged (below-1024) TCP port. In Omnibus GitLab we provide this convenience by bundling an automatically configured NGINX service that needs to run its master process as root to open ports 80 and 443.
If this is problematic, administrators installing GitLab can disable the bundled NGINX service, but this puts the burden on them to keep the NGINX configuration in tune with GitLab during application updates.
Isolation between Omnibus services
Bundled services in Omnibus GitLab (GitLab itself, NGINX, PostgreSQL,
Redis, Mattermost) are isolated from each other using Unix user
accounts. Creating and managing these user accounts requires root
access. By default, Omnibus GitLab will create the required Unix
accounts during gitlab-ctl reconfigure
but that behavior can be
disabled.
In principle Omnibus GitLab could do with only 2 user accounts (one
for GitLab and one for Mattermost) if we give each application its own
runit (runsvdir), PostgreSQL and Redis process. But this would be a
major change in the gitlab-ctl reconfigure
Chef code and it would
probably create major upgrade pain for all existing Omnibus GitLab
installations. (We would probably have to rearrange the directory
structure under /var/opt/gitlab
.)
Tweaking the operating system for better performance
During gitlab-ctl reconfigure
we set and install several sysctl
tweaks to improve PostgreSQL performance and increase connection limits.
This can only be done with root access.
gitlab-rake assets:precompile
fails with 'Permission denied'
Some users report that running gitlab-rake assets:precompile
does not work
with the omnibus packages. The short answer to this is: do not run that
command, it is only for GitLab installations from source.
The GitLab web interface uses CSS and JavaScript files, called 'assets' in Ruby
on Rails-speak. In the upstream GitLab repository
these files are stored in a developer-friendly way: easy to read and edit. When
you are a normal user of GitLab, you do not want these files to be in the
developer friendly format however because that makes GitLab slow. This is why
part of the GitLab setup process is to convert the assets from a
developer-friendly format to an end-user friendly (compact, fast) format; that
is what the rake assets:precompile
script is for.
When you install GitLab from source (which was the only way to do it before we
had omnibus packages) you need to convert the assets on your GitLab server
every time you update GitLab. People used to overlook this step and there are
still posts, comments and mails out there on the internet where users recommend
each other to run rake assets:precompile
(which has now been renamed
gitlab:assets:compile
). With the omnibus packages things are different: when
we build the package we compile the assets for you.
When you install GitLab with an omnibus package, the converted assets are
already there! That is why you do not need to run rake assets:precompile
when
you install GitLab from a package.
When gitlab-rake assets:precompile
fails with a permission error it fails for
a good reason from a security standpoint: the fact that the assets cannot
easily be rewritten makes it harder for an attacker to use your GitLab server
to serve evil JavaScript code to the visitors of your GitLab server.
If you want to run GitLab with custom JavaScript or CSS code you are probably better off running GitLab from source, or building your own packages.
If you really know what you are doing,
you can execute gitlab-rake gitlab:assets:compile
like this:
sudo NO_PRIVILEGE_DROP=true USE_DB=false gitlab-rake gitlab:assets:clean gitlab:assets:compile
# user and path might be different if you changed the defaults of
# user['username'], user['group'] and gitlab_rails['dir'] in gitlab.rb
sudo chown -R git:git /var/opt/gitlab/gitlab-rails/tmp/cache
'Short read or OOM loading DB' error
Try cleaning the old Redis session.
Apt error 'The requested URL returned error: 403'
When trying to install GitLab using the apt repo if you receive an error similar to:
W: Failed to fetch https://packages.gitlab.com/gitlab/gitlab-ce/DISTRO/dists/CODENAME/main/source/Sources The requested URL returned error: 403
check if there is a repository cacher in front of your server, like for example apt-cacher-ng
.
Add the following line to apt-cacher-ng config(eg. in /etc/apt-cacher-ng/acng.conf
):
PassThroughPattern: (packages\.gitlab\.com|packages-gitlab-com\.s3\.amazonaws\.com|*\.cloudfront\.net)