--- stage: Systems group: Distribution info: To determine the technical writer assigned to the Stage/Group associated with this page, see https://about.gitlab.com/handbook/product/ux/technical-writing/#assignments --- # Omnibus GitLab logs **(FREE SELF)** GitLab includes an [advanced log system](https://docs.gitlab.com/ee/administration/logs/index.html) where every service and component within GitLab will output system logs. Here are the Omnibus configuration settings and tools for managing these logs. ## Tail logs in a console on the server If you want to 'tail', i.e. view live log updates of GitLab logs you can use `gitlab-ctl tail`. ```shell # Tail all logs; press Ctrl-C to exit sudo gitlab-ctl tail # Drill down to a sub-directory of /var/log/gitlab sudo gitlab-ctl tail gitlab-rails # Drill down to an individual file sudo gitlab-ctl tail nginx/gitlab_error.log ``` ### Tail logs in a console and save to a file Oftentimes, it is useful to both display the logs in the console and save them to a file for later debugging/analysis. You can use the [`tee`](https://en.wikipedia.org/wiki/Tee_(command)) utility to accomplish this. ```shell # Use 'tee' to tail all the logs to STDOUT and write to a file at the same time sudo gitlab-ctl tail | tee --append /tmp/gitlab_tail.log ``` ## Configure default log directories In your `/etc/gitlab/gitlab.rb` file, there are many `log_directory` keys for the various types of logs. Uncomment and update the values for all the logs you want to place elsewhere: ```ruby # For example: gitlab_rails['log_directory'] = "/var/log/gitlab/gitlab-rails" puma['log_directory'] = "/var/log/gitlab/puma" registry['log_directory'] = "/var/log/gitlab/registry" ... ``` Gitaly and Mattermost have different log directory configs: ```ruby gitaly['configuration']['logging']['dir'] = "/var/log/gitlab/registry" mattermost['log_file_directory'] = "/var/log/gitlab/registry" ``` Run `sudo gitlab-ctl reconfigure` to configure your instance with these settings. ## runit logs The [runit-managed](../architecture/index.md#runit) services in Omnibus GitLab generate log data using `svlogd`. - Logs are written to a file called `current`. - Periodically, this log is compressed and renamed using the TAI64N format, for example: `@400000005f8eaf6f1a80ef5c.s`. - The filesystem datestamp on the compressed logs will be consistent with the time GitLab last wrote to that file. - `zmore` and `zgrep` allow viewing and searching through both compressed or uncompressed logs. Read the [svlogd documentation](http://smarden.org/runit/svlogd.8.html) for more information about the files it generates. You can modify svlogd settings via `/etc/gitlab/gitlab.rb` with the following settings: ```ruby # Below are the default values logging['svlogd_size'] = 200 * 1024 * 1024 # rotate after 200 MB of log data logging['svlogd_num'] = 30 # keep 30 rotated log files logging['svlogd_timeout'] = 24 * 60 * 60 # rotate after 24 hours logging['svlogd_filter'] = "gzip" # compress logs with gzip logging['svlogd_udp'] = nil # transmit log messages via UDP logging['svlogd_prefix'] = nil # custom prefix for log messages # Optionally, you can override the prefix for e.g. Nginx nginx['svlogd_prefix'] = "nginx" ``` ## Logrotate The **logrotate** service built into GitLab manages all logs except those captured by **runit**. This service will rotate, compress, and eventually delete the log data such as `gitlab-rails/production.log` and `nginx/gitlab_access.log`. You can configure common logrotate settings, configure per-service logrotate settings, and completely disable logrotate with `/etc/gitlab/gitlab.rb`. ### Configuring common logrotate settings Settings common to all **logrotate** services can be set in the `/etc/gitlab/gitlab.rb` file. These settings correspond to configuration options in the logrotate configuration files for each service. See the logrotate man page (`man logrotate`) for details. ```ruby logging['logrotate_frequency'] = "daily" # rotate logs daily logging['logrotate_maxsize'] = nil # logs will be rotated when they grow bigger than size specified for `maxsize`, even before the specified time interval (daily, weekly, monthly, or yearly) logging['logrotate_size'] = nil # do not rotate by size by default logging['logrotate_rotate'] = 30 # keep 30 rotated logs logging['logrotate_compress'] = "compress" # see 'man logrotate' logging['logrotate_method'] = "copytruncate" # see 'man logrotate' logging['logrotate_postrotate'] = nil # no postrotate command by default logging['logrotate_dateformat'] = nil # use date extensions for rotated files rather than numbers e.g. a value of "-%Y-%m-%d" would give rotated files like production.log-2016-03-09.gz ``` ### Configuring individual service logrotate settings You can customize logrotate settings for each individual service by using `/etc/gitlab/gitlab.rb`. For example, to customize logrotate frequency and size for the `nginx` service, use: ```ruby nginx['logrotate_frequency'] = nil nginx['logrotate_size'] = "200M" ``` ### Disabling logrotate You can also disable the built-in logrotate service with the following setting in `/etc/gitlab/gitlab.rb`: ```ruby logrotate['enable'] = false ``` ### Logrotate `notifempty` setting Since [GitLab 13.6](https://gitlab.com/gitlab-org/omnibus-gitlab/-/merge_requests/3820), the logrotate service runs with a non-configurable default of `notifempty`, resolving the following issues: - Empty logs being rotated unnecessarily, and often many empty logs being stored. - One-off logs that are useful for long term troubleshooting being deleted after 30 days, such as database migration logs. ### Logrotate one-off and empty log handling Logs are now rotated and recreated by **logrotate** as needed, and one-off logs are only rotated when they change. With this setting in place, some tidying can be done: - Empty one-off logs such as `gitlab-rails/gitlab-rails-db-migrate*.log` can be deleted. - Empty logs which were rotated and compressed by older versions of GitLab. These empty logs are usually 20 bytes in size. ### Run logrotate manually Logrotate is a scheduled job but it can also be triggered on-demand. To manually trigger GitLab log rotation with `logrotate`, use the following command: ```shell /opt/gitlab/embedded/sbin/logrotate -fv -s /var/opt/gitlab/logrotate/logrotate.status /var/opt/gitlab/logrotate/logrotate.conf ``` ## UDP log forwarding **(PREMIUM SELF)** Omnibus GitLab can utilize the UDP logging feature in svlogd as well as sending non-svlogd logs to a syslog-compatible remote system using UDP. To configure Omnibus GitLab to send syslog-protocol messages via UDP, use the following settings: ```ruby logging['udp_log_shipping_host'] = '1.2.3.4' # Your syslog server # logging['udp_log_shipping_hostname'] = nil # Optional, defaults the system hostname logging['udp_log_shipping_port'] = 1514 # Optional, defaults to 514 (syslog) ``` NOTE: Setting `udp_log_shipping_host` will [add a `svlogd_prefix`](https://gitlab.com/gitlab-org/omnibus-gitlab/blob/master/files/gitlab-cookbooks/gitlab/libraries/logging.rb) for the specified hostname and service for each of the [runit-managed](../architecture/index.md#runit) services. Example log messages: ```plaintext Jun 26 06:33:46 ubuntu1204-test production.log: Started GET "/root/my-project/import" for 127.0.0.1 at 2014-06-26 06:33:46 -0700 Jun 26 06:33:46 ubuntu1204-test production.log: Processing by ProjectsController#import as HTML Jun 26 06:33:46 ubuntu1204-test production.log: Parameters: {"id"=>"root/my-project"} Jun 26 06:33:46 ubuntu1204-test production.log: Completed 200 OK in 122ms (Views: 71.9ms | ActiveRecord: 12.2ms) Jun 26 06:33:46 ubuntu1204-test gitlab_access.log: 172.16.228.1 - - [26/Jun/2014:06:33:46 -0700] "GET /root/my-project/import HTTP/1.1" 200 5775 "https://172.16.228.169/root/my-project/import" "Mozilla/5.0 (Macintosh; Intel Mac OS X 10_9_3) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/35.0.1916.153 Safari/537.36" 2014-06-26_13:33:46.49866 ubuntu1204-test sidekiq: 2014-06-26T13:33:46Z 18107 TID-7nbj0 Sidekiq::Extensions::DelayedMailer JID-bbfb118dd1db20f6c39f5b50 INFO: start 2014-06-26_13:33:46.52608 ubuntu1204-test sidekiq: 2014-06-26T13:33:46Z 18107 TID-7muoc RepositoryImportWorker JID-57ee926c3655fcfa062338ae INFO: start ``` ## Using a custom NGINX log format By default the NGINX access logs will use a version of the 'combined' NGINX format, designed to hide potentially sensitive information embedded in query strings. If you want to use a custom log format string you can specify it in `/etc/gitlab/gitlab.rb` - see [the NGINX documentation](http://nginx.org/en/docs/http/ngx_http_log_module.html#log_format) for format details. ```ruby nginx['log_format'] = 'my format string $foo $bar' mattermost_nginx['log_format'] = 'my format string $foo $bar' ``` ## JSON logging Structured logs can be exported via JSON to be parsed by Elasticsearch, Splunk, or another log management system. [Beginning in Omnibus GitLab 12.0](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4102), the JSON format is enabled by default for all services that support it. NOTE: PostgreSQL does not support JSON logging without an external plugin. However, it does support logging in CSV format: ```ruby postgresql['log_destination'] = 'csvlog' postgresql['logging_collector'] = 'on' ``` A restart of the database is required for this to take effect. For more details, see the [PostgreSQL documentation](https://www.postgresql.org/docs/12/runtime-config-logging.html). ## Text logging Customers with established log ingestion systems may not wish to use the JSON log format. Text formatting can be configured by setting the following in `/etc/gitlab/gitlab.rb` and then running `gitlab-ctl reconfigure` afterward: ```ruby gitaly['logging_format'] = '' gitlab_shell['log_format'] = 'text' gitlab_workhorse['log_format'] = 'text' registry['log_formatter'] = 'text' sidekiq['log_format'] = 'text' gitlab_pages['log_format'] = 'text' ``` NOTE: There are a few variations in attribute names for the log format depending on the service involved (for example, Container Registry uses `log_formatter`, Gitaly and Praefect both use `logging_format`). See [Issue #4280](https://gitlab.com/gitlab-org/omnibus-gitlab/-/issues/4280) for more details. ## rbtrace GitLab ships with [`rbtrace`](https://github.com/tmm1/rbtrace), which allows you to trace Ruby code, view all running threads, take memory dumps, and more. However, this is not enabled by default. To enable it, define the `ENABLE_RBTRACE` variable to the environment. In Omnibus: ```ruby gitlab_rails['env'] = {"ENABLE_RBTRACE" => "1"} ``` Then reconfigure the system and restart Puma and Sidekiq. To run this in Omnibus, run as root: ```ruby /opt/gitlab/embedded/bin/ruby /opt/gitlab/embedded/bin/rbtrace ``` ## Configuring log level/verbosity You can configure the minimum log levels (verbosity) for the Container Registry, GitLab Shell and Gitaly: 1. Edit `/etc/gitlab/gitlab.rb` and set the log levels: ```ruby registry['log_level'] = 'info' gitlab_shell['log_level'] = 'INFO' gitaly['logging_level'] = 'warn' ``` 1. Reconfigure GitLab: ```shell sudo gitlab-ctl reconfigure ``` NOTE: You [cannot edit](https://gitlab.com/groups/gitlab-org/-/epics/6034) the `log_level` for other GitLab logs, for example `production_json.log`, `sidekiq.log`, and so on. ## Setting a custom log group GitLab supports assigning a custom group to the configured [log directories](#configure-default-log-directories) A global `logging['log_group']` setting in your `/etc/gitlab/gitlab.rb` file can be configured as well as per-service `log_group` settings such as `gitaly['log_group']`. You will need to run `sudo gitlab-ctl reconfigure` to configure your instance when adding `log_group` settings. Setting a global or per-service `log_group` will: - Change the permissions on the per-service log directories (or all log directories if using the global setting) to `0750` to allow the configured group members to read the contents of the log directory. - Configure [runit](#runit-logs) to write and rotate logs using the specified `log_group` : either per-service or for all runit-managed services. ### Custom log group limitations Logs for services not managed by runit (e.g. the `gitlab-rails` logs in `/var/log/gitlab/gitlab-rails`) will not inherit the configured `log_group` setting. The group must already exist on the host - Omnibus will not create the group when running `sudo gitlab-ctl reconfigure`.