diff --git a/doc/settings/nginx.md b/doc/settings/nginx.md index 3ce817c649f771bfcd4317777f854b4ce84b1a0b_ZG9jL3NldHRpbmdzL25naW54Lm1k..bc0ef9473bf9d5ca34bfd18d8a17870d2d991214_ZG9jL3NldHRpbmdzL25naW54Lm1k 100644 --- a/doc/settings/nginx.md +++ b/doc/settings/nginx.md @@ -651,136 +651,3 @@ Don't forget to restart NGINX to load the new configuration (on Debian-based systems `sudo service nginx restart`). - -## Troubleshooting - -### 400 Bad Request: too many Host headers - -Make sure you don't have the `proxy_set_header` configuration in -`nginx['custom_gitlab_server_config']` settings and instead use the -['proxy_set_headers'](ssl/index.md#configure-a-reverse-proxy-or-load-balancer-ssl-termination) configuration in your `gitlab.rb` file. - -### `javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure` - -Linux package installations don't support TLSv1 protocol by default. -This can cause connection issues with some older Java based IDE clients when interacting with -your GitLab instance. -We strongly urge you to upgrade ciphers on your server, similar to what was mentioned -in [this user comment](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/624#note_299061). - -If it is not possible to make this server change, you can default back to the old -behavior by changing the values in your `/etc/gitlab/gitlab.rb`: - -```ruby -nginx['ssl_protocols'] = "TLSv1 TLSv1.1 TLSv1.2 TLSv1.3" -``` - -### Mismatch between private key and certificate - -If you see `x509 certificate routines:X509_check_private_key:key values mismatch)` in the NGINX logs (`/var/log/gitlab/nginx/current` by default for Omnibus), there is a mismatch between your private key and certificate. - -To fix this, you will need to match the correct private key with your certificate. - -To ensure you have the correct key and certificate, you can ensure that the modulus of the private key and certificate match: - -```shell -/opt/gitlab/embedded/bin/openssl rsa -in /etc/gitlab/ssl/gitlab.example.com.key -noout -modulus | /opt/gitlab/embedded/bin/openssl sha256 - -/opt/gitlab/embedded/bin/openssl x509 -in /etc/gitlab/ssl/gitlab.example.com.crt -noout -modulus| /opt/gitlab/embedded/bin/openssl sha256 -``` - -Once you verify that they match, you will need to reconfigure and reload NGINX: - -```shell -sudo gitlab-ctl reconfigure -sudo gitlab-ctl hup nginx -``` - -### Request Entity Too Large - -If you see `Request Entity Too Large` in the [NGINX logs](https://docs.gitlab.com/ee/administration/logs/index.html#nginx-logs), -you will need to increase the [Client Max Body Size](http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size). -You may encounter this error if you have increased the [Max import size](https://docs.gitlab.com/ee/administration/settings/account_and_limit_settings.html#max-import-size). -In a Kubernetes-based GitLab installation, this setting is -[named differently](https://docs.gitlab.com/charts/charts/gitlab/webservice/#proxybodysize). - -To increase the `client_max_body_size`, you will need to set the value in your `/etc/gitlab/gitlab.rb`: - -```ruby -nginx['client_max_body_size'] = '250m' -``` - -Make sure you run `sudo gitlab-ctl reconfigure` and run `sudo gitlab-ctl hup nginx` to cause NGINX to -[reload the with the updated configuration](http://nginx.org/en/docs/control.html) -To increase the `client_max_body_size`: - -1. Edit `/etc/gitlab/gitlab.rb` and set the preferred value: - - ```ruby - nginx['client_max_body_size'] = '250m' - ``` - -1. Reconfigure GitLab, and [HUP](https://nginx.org/en/docs/control.html) - NGINX to cause it to reload with the updated configuration gracefully: - - ```shell - sudo gitlab-ctl reconfigure - sudo gitlab-ctl hup nginx - ``` - -### Security scan is showing a "NGINX HTTP Server Detection" warning - -Some security scanners detect issues when they see the `Server: nginx` http header. Most scanners with this alert will -notify as `Low` or `Info` severity. [See Nessus as an example](https://www.tenable.com/plugins/nessus/106375). - -We recommend ignoring this warning, as the benefit of removing the header is low, and its presence -[helps support the NGINX project in usage statistics](https://trac.nginx.org/nginx/ticket/1644). We do provide a way to turn off the -header with `hide_server_tokens`: - -1. Edit `/etc/gitlab/gitlab.rb` and set the value: - - ```ruby - nginx['hide_server_tokens'] = 'on' - ``` - -1. Reconfigure GitLab, and [hup](https://nginx.org/en/docs/control.html) - NGINX to cause it to reload the with the updated configuration gracefully: - - ```shell - sudo gitlab-ctl reconfigure - sudo gitlab-ctl hup nginx - ``` - -### `Branch 'branch_name' was not found in this project's repository` when Web IDE and external NGINX are used - -If you get this error, check your NGINX configuration file if you have a trailing slash in `proxy_pass` and remove it: - -1. Edit your NGINX configuration file: - - ```plaintext - proxy_pass https://1.2.3.4; - ``` - -1. Restart NGINX: - - ```shell - sudo systemctl restart nginx - ``` - -### GitLab is presenting 502 errors and `worker_connections are not enough` in logs - -If you get a `worker_connections are not enough` error in the logs, configure the NGINX worker connections to a higher value: - -1. Edit `/etc/gitlab/gitlab.rb`: - - ```ruby - gitlab['nginx']['worker_connections'] = 10240 - ``` - -1. Reconfigure GitLab: - - ```shell - sudo gitlab-ctl reconfigure - ``` - -The default value is [10240 connections](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/86f98401113eb843658b63d45d58be8b706216f3/files/gitlab-cookbooks/gitlab/attributes/default.rb#L750). diff --git a/doc/settings/nginx_troubleshooting.md b/doc/settings/nginx_troubleshooting.md new file mode 100644 index 0000000000000000000000000000000000000000..bc0ef9473bf9d5ca34bfd18d8a17870d2d991214_ZG9jL3NldHRpbmdzL25naW54X3Ryb3VibGVzaG9vdGluZy5tZA== --- /dev/null +++ b/doc/settings/nginx_troubleshooting.md @@ -0,0 +1,176 @@ +--- +stage: Systems +group: Distribution +info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://handbook.gitlab.com/handbook/product/ux/technical-writing/#assignments +--- + +# Troubleshooting NGINX + +DETAILS: +**Tier:** Free, Premium, Ultimate +**Offering:** Self-managed + +When [configuring NGINX](nginx.md), you might encounter the following issues. + +## Error: `400 Bad Request: too many Host headers` + +The workaround is to make sure you don't have the `proxy_set_header` configuration in +`nginx['custom_gitlab_server_config']` settings. +Instead, use the +[`roxy_set_headers`](ssl/index.md#configure-a-reverse-proxy-or-load-balancer-ssl-termination) +configuration in your `gitlab.rb` file. + +## Error: `Received fatal alert: handshake_failure` + +You might get an error that states: + +```plaintext +javax.net.ssl.SSLHandshakeException: Received fatal alert: handshake_failure +``` + +This issue occurs when you use an older Java-based IDE client to interact with your GitLab instance. +Those IDEs can use the TLS 1 protocol, which the Linux package installations don't support by default. + +To resolve this issue, upgrade ciphers on your server, similar to the user in +[issue 624](https://gitlab.com/gitlab-org/gitlab-foss/-/issues/624#note_299061). + +If it is not possible to make this server change, you can default back to the old +behavior by changing the values in your `/etc/gitlab/gitlab.rb`: + +```ruby +nginx['ssl_protocols'] = "TLSv1 TLSv1.1 TLSv1.2 TLSv1.3" +``` + +## Mismatch between private key and certificate + +In the [NGINX logs](https://docs.gitlab.com/ee/administration/logs/index.html#nginx-logs), you might find: + +```plaintext +x509 certificate routines:X509_check_private_key:key values mismatch) +``` + +This issue occurs when there is a mismatch between your private key and certificate. + +To resolve this, match the correct private key with your certificate: + +1. To ensure you have the correct key and certificate, check whether the moduli of the private key and + certificate match: + + ```shell + /opt/gitlab/embedded/bin/openssl rsa -in /etc/gitlab/ssl/gitlab.example.com.key -noout -modulus | /opt/gitlab/embedded/bin/openssl sha256 + + /opt/gitlab/embedded/bin/openssl x509 -in /etc/gitlab/ssl/gitlab.example.com.crt -noout -modulus| /opt/gitlab/embedded/bin/openssl sha256 + ``` + +1. After you verify that they match, reconfigure and reload NGINX: + + ```shell + sudo gitlab-ctl reconfigure + sudo gitlab-ctl hup nginx + ``` + +## `Request Entity Too Large` + +In the [NGINX logs](https://docs.gitlab.com/ee/administration/logs/index.html#nginx-logs), you might find: + +```plaintext +Request Entity Too Large +``` + +This issue occurs when you have increased the [max import size](https://docs.gitlab.com/ee/administration/settings/import_and_export_settings.html#max-import-size). + +To resolve this, increase the +[client max body size](http://nginx.org/en/docs/http/ngx_http_core_module.html#client_max_body_size). + +In a Kubernetes-based GitLab installation, this setting is +[named differently](https://docs.gitlab.com/charts/charts/gitlab/webservice/#proxybodysize). + +To increase the value of `client_max_body_size`: + +1. Edit `/etc/gitlab/gitlab.rb` and set the preferred value: + + ```ruby + nginx['client_max_body_size'] = '250m' + ``` + +1. Save the file and [reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#reconfigure-a-linux-package-installation). +1. [`HUP`](https://nginx.org/en/docs/control.html) NGINX to cause it to reload with the updated + configuration gracefully: + + ```shell + sudo gitlab-ctl hup nginx + ``` + +## Security scan warning: `NGINX HTTP Server Detection` + +This issue occurs when some security scanners detect the `Server: nginx` HTTP header. +Most scanners with this alert mark it as `Low` or `Info` severity. +For example, see [Nessus](https://www.tenable.com/plugins/nessus/106375). + +You should ignore this warning, as the benefit of removing the header is low, and its presence +[helps support the NGINX project in usage statistics](https://trac.nginx.org/nginx/ticket/1644). + +The workaround is to turn the header off by using `hide_server_tokens`: + +1. Edit `/etc/gitlab/gitlab.rb` and set the value: + + ```ruby + nginx['hide_server_tokens'] = 'on' + ``` + +1. Save the file and [reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#reconfigure-a-linux-package-installation). +1. [`HUP`](https://nginx.org/en/docs/control.html) NGINX to cause it to reload with the updated + configuration gracefully: + + ```shell + sudo gitlab-ctl hup nginx + ``` + +## Branch not found when using Web IDE and external NGINX + +You might get an error that states: + +```plaintext +Branch 'branch_name' was not found in this project's repository +``` + +This issue occurs when there's a trailing slash in `proxy_pass` in your NGINX configuration file. + +To resolve it: + +1. Edit your NGINX configuration file so there's no trailing slash in `proxy_pass`: + + ```plaintext + proxy_pass https://1.2.3.4; + ``` + +1. Restart NGINX: + + ```shell + sudo systemctl restart nginx + ``` + +## Error: `worker_connections are not enough` + +You might get `502` errors from GitLab, and find the following in +[NGINX logs](https://docs.gitlab.com/ee/administration/logs/index.html#nginx-logs): + +```plaintext +worker_connections are not enough +``` + +This issue occurs when worker connections are set to a value that's too low. + +To resolve it, configure the NGINX worker connections to a higher value: + +1. Edit `/etc/gitlab/gitlab.rb`: + + ```ruby + gitlab['nginx']['worker_connections'] = 10240 + ``` + + 10240 connections is + [the default value](https://gitlab.com/gitlab-org/omnibus-gitlab/-/blob/374b34e2bdc4bccb73665e0dc856ae32d6082d77/files/gitlab-cookbooks/gitlab/attributes/default.rb#L883). + +1. Save the file and [reconfigure GitLab](https://docs.gitlab.com/ee/administration/restart_gitlab.html#reconfigure-a-linux-package-installation) + for the changes to take effect.