Read about our upcoming Code of Conduct on this issue

logs.md 21.7 KB
Newer Older
1
# Log system
2
3
4
5
6

GitLab has an advanced log system where everything is logged so that you
can analyze your instance using various system log files. In addition to
system log files, GitLab Enterprise Edition comes with Audit Events.
Find more about them [in Audit Events
7
documentation](audit_events.md)
8
9
10
11

System log files are typically plain text in a standard log file format.
This guide talks about how to read and use these system log files.

12
13
14
15
## `production_json.log`

This file lives in `/var/log/gitlab/gitlab-rails/production_json.log` for
Omnibus GitLab packages or in `/home/git/gitlab/log/production_json.log` for
16
installations from source. (When GitLab is running in an environment
17
18
19
20
other than production, the corresponding logfile is shown here.)

It contains a structured log for Rails controller requests received from
GitLab, thanks to [Lograge](https://github.com/roidrage/lograge/). Note that
Stan Hu's avatar
Stan Hu committed
21
requests from the API are logged to a separate file in `api_json.log`.
22
23
24
25

Each line contains a JSON line that can be ingested by Elasticsearch, Splunk, etc. For example:

```json
26
{"method":"GET","path":"/gitlab/gitlab-foss/issues/1234","format":"html","controller":"Projects::IssuesController","action":"show","status":200,"duration":229.03,"view":174.07,"db":13.24,"time":"2017-08-08T20:15:54.821Z","params":[{"key":"param_key","value":"param_value"}],"remote_ip":"18.245.0.1","user_id":1,"username":"admin","gitaly_calls":76,"gitaly_duration":7.41,"queue_duration": 112.47}
27
28
```

29
30
31
In this example, you can see this was a GET request for a specific
issue. Notice each line also contains performance data. All times are in
milliseconds:
32

33
34
1. `duration`: total time taken to retrieve the request
1. `queue_duration`: total time that the request was queued inside GitLab Workhorse
35
36
37
1. `view`: total time taken inside the Rails views
1. `db`: total time to retrieve data from the database
1. `gitaly_calls`: total number of calls made to Gitaly
38
1. `gitaly_duration`: total time taken by Gitaly calls
39

40
41
User clone/fetch activity using http transport appears in this log as `action: git_upload_pack`.

42
43
44
In addition, the log contains the IP address from which the request originated
(`remote_ip`) as well as the user's ID (`user_id`), and username (`username`).

45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
NOTE: **Note:** Starting with GitLab 12.5, if an error occurs, an
`exception` field is included with `class`, `message`, and
`backtrace`. Previous versions included an `error` field instead of
`exception.class` and `exception.message`. For example:

```json
{
  "method": "GET",
  "path": "/admin",
  "format": "html",
  "controller": "Admin::DashboardController",
  "action": "index",
  "status": 500,
  "duration": 2584.11,
  "view": 0,
  "db": 9.21,
  "time": "2019-11-14T13:12:46.156Z",
  "params": [],
  "remote_ip": "127.0.0.1",
  "user_id": 1,
  "username": "root",
  "ua": "Mozilla/5.0 (Macintosh; Intel Mac OS X 10.14; rv:70.0) Gecko/20100101 Firefox/70.0",
  "queue_duration": 274.35,
  "correlation_id": "KjDVUhNvvV3",
  "cpu_s": 2.837645135999999,
70
71
72
73
74
75
76
77
78
79
80
81
  "exception.class": "NameError",
  "exception.message": "undefined local variable or method `adsf' for #<Admin::DashboardController:0x00007ff3c9648588>",
  "exception.backtrace": [
    "app/controllers/admin/dashboard_controller.rb:11:in `index'",
    "ee/app/controllers/ee/admin/dashboard_controller.rb:14:in `index'",
    "ee/lib/gitlab/ip_address_state.rb:10:in `with'",
    "ee/app/controllers/ee/application_controller.rb:43:in `set_current_ip_address'",
    "lib/gitlab/session.rb:11:in `with_session'",
    "app/controllers/application_controller.rb:450:in `set_session_storage'",
    "app/controllers/application_controller.rb:444:in `set_locale'",
    "ee/lib/gitlab/jira/middleware.rb:19:in `call'"
  ]
82
83
84
}
```

85
## `production.log`
86
87

This file lives in `/var/log/gitlab/gitlab-rails/production.log` for
88
Omnibus GitLab packages or in `/home/git/gitlab/log/production.log` for
89
installations from source. (When GitLab is running in an environment
90
other than production, the corresponding logfile is shown here.)
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121

It contains information about all performed requests. You can see the
URL and type of request, IP address and what exactly parts of code were
involved to service this particular request. Also you can see all SQL
request that have been performed and how much time it took. This task is
more useful for GitLab contributors and developers. Use part of this log
file when you are going to report bug. For example:

```
Started GET "/gitlabhq/yaml_db/tree/master" for 168.111.56.1 at 2015-02-12 19:34:53 +0200
Processing by Projects::TreeController#show as HTML
  Parameters: {"project_id"=>"gitlabhq/yaml_db", "id"=>"master"}

  ... [CUT OUT]

  Namespaces"."created_at" DESC, "namespaces"."id" DESC LIMIT 1 [["id", 26]]
  CACHE (0.0ms) SELECT  "members".* FROM "members"  WHERE "members"."source_type" = 'Project' AND "members"."type" IN ('ProjectMember') AND "members"."source_id" = $1 AND "members"."source_type" = $2 AND "members"."user_id" = 1  ORDER BY "members"."created_at" DESC, "members"."id" DESC LIMIT 1  [["source_id", 18], ["source_type", "Project"]]
  CACHE (0.0ms) SELECT  "members".* FROM "members"  WHERE "members"."source_type" = 'Project' AND "members".
  (1.4ms) SELECT COUNT(*) FROM "merge_requests"  WHERE "merge_requests"."target_project_id" = $1 AND ("merge_requests"."state" IN ('opened','reopened')) [["target_project_id", 18]]
  Rendered layouts/nav/_project.html.haml (28.0ms)
  Rendered layouts/_collapse_button.html.haml (0.2ms)
  Rendered layouts/_flash.html.haml (0.1ms)
  Rendered layouts/_page.html.haml (32.9ms)
Completed 200 OK in 166ms (Views: 117.4ms | ActiveRecord: 27.2ms)
```

In this example we can see that server processed an HTTP request with URL
`/gitlabhq/yaml_db/tree/master` from IP 168.111.56.1 at 2015-02-12
19:34:53 +0200. Also we can see that request was processed by
`Projects::TreeController`.

Stan Hu's avatar
Stan Hu committed
122
123
124
125
126
127
128
129
130
## `api_json.log`

Introduced in GitLab 10.0, this file lives in
`/var/log/gitlab/gitlab-rails/api_json.log` for Omnibus GitLab packages or in
`/home/git/gitlab/log/api_json.log` for installations from source.

It helps you see requests made directly to the API. For example:

```json
131
{"time":"2018-10-29T12:49:42.123Z","severity":"INFO","duration":709.08,"db":14.59,"view":694.49,"status":200,"method":"GET","path":"/api/v4/projects","params":[{"key":"action","value":"git-upload-pack"},{"key":"changes","value":"_any"},{"key":"key_id","value":"secret"},{"key":"secret_token","value":"[FILTERED]"}],"host":"localhost","remote_ip":"::1","ua":"Ruby","route":"/api/:version/projects","user_id":1,"username":"root","queue_duration":100.31,"gitaly_calls":30,"gitaly_duration":5.36}
Stan Hu's avatar
Stan Hu committed
132
133
134
135
136
137
```

This entry above shows an access to an internal endpoint to check whether an
associated SSH key can download the project in question via a `git fetch` or
`git clone`. In this example, we see:

138
139
1. `duration`: total time in milliseconds taken to retrieve the request
1. `queue_duration`: total time in milliseconds that the request was queued inside GitLab Workhorse
Stan Hu's avatar
Stan Hu committed
140
141
142
143
144
1. `method`: The HTTP method used to make the request
1. `path`: The relative path of the query
1. `params`: Key-value pairs passed in a query string or HTTP body. Sensitive parameters (e.g. passwords, tokens, etc.) are filtered out.
1. `ua`: The User-Agent of the requester

145
## `application.log`
146
147

This file lives in `/var/log/gitlab/gitlab-rails/application.log` for
148
Omnibus GitLab packages or in `/home/git/gitlab/log/application.log` for
149
150
151
152
153
154
155
156
157
158
159
160
161
installations from source.

It helps you discover events happening in your instance such as user creation,
project removing and so on. For example:

```
October 06, 2014 11:56: User "Administrator" (admin@example.com) was created
October 06, 2014 11:56: Documentcloud created a new project "Documentcloud / Underscore"
October 06, 2014 11:56: Gitlab Org created a new project "Gitlab Org / Gitlab Ce"
October 07, 2014 11:25: User "Claudie Hodkiewicz" (nasir_stehr@olson.co.uk)  was removed
October 07, 2014 11:25: Project "project133" was removed
```

162
163
164
165
166
167
## `integrations_json.log`

This file lives in `/var/log/gitlab/gitlab-rails/integrations_json.log` for
Omnibus GitLab packages or in `/home/git/gitlab/log/integrations_json.log` for
installations from source.

Takuya Noguchi's avatar
Takuya Noguchi committed
168
It contains information about [integrations](../user/project/integrations/project_services.md) activities such as Jira, Asana and Irker services. It uses JSON format like the example below:
169
170
171

``` json
{"severity":"ERROR","time":"2018-09-06T14:56:20.439Z","service_class":"JiraService","project_id":8,"project_path":"h5bp/html5-boilerplate","message":"Error sending message","client_url":"http://jira.gitlap.com:8080","error":"execution expired"}
172
{"severity":"INFO","time":"2018-09-06T17:15:16.365Z","service_class":"JiraService","project_id":3,"project_path":"namespace2/project2","message":"Successfully posted","client_url":"http://jira.example.com"}
173
174
```

175
176
177
178
179
180
181
182
183
184
185
186
187
188
189
190
191
192
193
## `kubernetes.log`

Introduced in GitLab 11.6. This file lives in
`/var/log/gitlab/gitlab-rails/kubernetes.log` for Omnibus GitLab
packages or in `/home/git/gitlab/log/kubernetes.log` for
installations from source.

It logs information related to the Kubernetes Integration including errors
during installing cluster applications on your GitLab managed Kubernetes
clusters.

Each line contains a JSON line that can be ingested by Elasticsearch, Splunk,
etc. For example:

```json
{"severity":"ERROR","time":"2018-11-23T15:14:54.652Z","exception":"Kubeclient::HttpError","error_code":401,"service":"Clusters::Applications::CheckInstallationProgressService","app_id":14,"project_ids":[1],"group_ids":[],"message":"Unauthorized"}
{"severity":"ERROR","time":"2018-11-23T15:42:11.647Z","exception":"Kubeclient::HttpError","error_code":null,"service":"Clusters::Applications::InstallService","app_id":2,"project_ids":[19],"group_ids":[],"message":"SSL_connect returned=1 errno=0 state=error: certificate verify failed (unable to get local issuer certificate)"}
```

Stan Hu's avatar
Stan Hu committed
194
## `git_json.log`
195

Stan Hu's avatar
Stan Hu committed
196
197
This file lives in `/var/log/gitlab/gitlab-rails/git_json.log` for
Omnibus GitLab packages or in `/home/git/gitlab/log/git_json.log` for
198
199
installations from source.

Stan Hu's avatar
Stan Hu committed
200
NOTE: **Note:**
Stan Hu's avatar
Stan Hu committed
201
202
After 12.2, this file was renamed from `githost.log` to
`git_json.log` and stored in JSON format.
Stan Hu's avatar
Stan Hu committed
203

204
205
206
207
208
209
GitLab has to interact with Git repositories but in some rare cases
something can go wrong and in this case you will know what exactly
happened. This log file contains all failed requests from GitLab to Git
repositories. In the majority of cases this file will be useful for developers
only. For example:

Stan Hu's avatar
Stan Hu committed
210
211
212
213
214
215
216
```json
{
   "severity":"ERROR",
   "time":"2019-07-19T22:16:12.528Z",
   "correlation_id":"FeGxww5Hj64",
   "message":"Command failed [1]: /usr/bin/git --git-dir=/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/group184/gitlabhq/.git --work-tree=/Users/vsizov/gitlab-development-kit/gitlab/tmp/tests/gitlab-satellites/group184/gitlabhq merge --no-ff -mMerge branch 'feature_conflict' into 'feature' source/feature_conflict\n\nerror: failed to push some refs to '/Users/vsizov/gitlab-development-kit/repositories/gitlabhq/gitlab_git.git'"
}
217
218
```

219
220
221
222
223
224
225
226
227
228
229
230
231
232
## `audit_json.log`

This file lives in `/var/log/gitlab/gitlab-rails/audit_json.log` for
Omnibus GitLab packages or in `/home/git/gitlab/log/audit_json.log` for
installations from source.

Changes to group or project settings are logged to this file. For example:

```json
{"severity":"INFO","time":"2018-10-17T17:38:22.523Z","author_id":3,"entity_id":2,"entity_type":"Project","change":"visibility","from":"Private","to":"Public","author_name":"John Doe4","target_id":2,"target_type":"Project","target_details":"namespace2/project2"}
{"severity":"INFO","time":"2018-10-17T17:38:22.830Z","author_id":5,"entity_id":3,"entity_type":"Project","change":"name","from":"John Doe7 / project3","to":"John Doe7 / new name","author_name":"John Doe6","target_id":3,"target_type":"Project","target_details":"namespace3/project3"}
{"severity":"INFO","time":"2018-10-17T17:38:23.175Z","author_id":7,"entity_id":4,"entity_type":"Project","change":"path","from":"","to":"namespace4/newpath","author_name":"John Doe8","target_id":4,"target_type":"Project","target_details":"namespace4/newpath"}
```

233
## `sidekiq.log`
234
235

This file lives in `/var/log/gitlab/gitlab-rails/sidekiq.log` for
236
Omnibus GitLab packages or in `/home/git/gitlab/log/sidekiq.log` for
237
238
239
240
241
242
243
244
245
246
247
installations from source.

GitLab uses background jobs for processing tasks which can take a long
time. All information about processing these jobs are written down to
this file. For example:

```
2014-06-10T07:55:20Z 2037 TID-tm504 ERROR: /opt/bitnami/apps/discourse/htdocs/vendor/bundle/ruby/1.9.1/gems/redis-3.0.7/lib/redis/client.rb:228:in `read'
2014-06-10T18:18:26Z 14299 TID-55uqo INFO: Booting Sidekiq 3.0.0 with redis options {:url=>"redis://localhost:6379/0", :namespace=>"sidekiq"}
```

Stan Hu's avatar
Stan Hu committed
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
Instead of the format above, you can opt to generate JSON logs for
Sidekiq. For example:

```json
{"severity":"INFO","time":"2018-04-03T22:57:22.071Z","queue":"cronjob:update_all_mirrors","args":[],"class":"UpdateAllMirrorsWorker","retry":false,"queue_namespace":"cronjob","jid":"06aeaa3b0aadacf9981f368e","created_at":"2018-04-03T22:57:21.930Z","enqueued_at":"2018-04-03T22:57:21.931Z","pid":10077,"message":"UpdateAllMirrorsWorker JID-06aeaa3b0aadacf9981f368e: done: 0.139 sec","job_status":"done","duration":0.139,"completed_at":"2018-04-03T22:57:22.071Z"}
```

For Omnibus GitLab installations, add the configuration option:

```ruby
sidekiq['log_format'] = 'json'
```

For source installations, edit the `gitlab.yml` and set the Sidekiq
`log_format` configuration option:

```yaml
  ## Sidekiq
  sidekiq:
    log_format: json
```

270
## `gitlab-shell.log`
271
272

This file lives in `/var/log/gitlab/gitlab-shell/gitlab-shell.log` for
273
Omnibus GitLab packages or in `/home/git/gitlab-shell/gitlab-shell.log` for
274
275
installations from source.

276
GitLab Shell is used by GitLab for executing Git commands and provide
277
278
279
280
281
282
283
SSH access to Git repositories. For example:

```
I, [2015-02-13T06:17:00.671315 #9291]  INFO -- : Adding project root/example.git at </var/opt/gitlab/git-data/repositories/root/dcdcdcdcd.git>.
I, [2015-02-13T06:17:00.679433 #9291]  INFO -- : Moving existing hooks directory and symlinking global hooks directory for /var/opt/gitlab/git-data/repositories/root/example.git.
```

284
User clone/fetch activity using SSH transport appears in this log as `executing git command <gitaly-upload-pack...`.
285

Greg Myers's avatar
Greg Myers committed
286
## `unicorn_stderr.log`
287
288

This file lives in `/var/log/gitlab/unicorn/unicorn_stderr.log` for
289
Omnibus GitLab packages or in `/home/git/gitlab/log/unicorn_stderr.log` for
290
291
292
293
294
installations from source.

Unicorn is a high-performance forking Web server which is used for
serving the GitLab application. You can look at this log if, for
example, your application does not respond. This log contains all
295
information about the state of Unicorn processes at any given time.
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311

```
I, [2015-02-13T06:14:46.680381 #9047]  INFO -- : Refreshing Gem list
I, [2015-02-13T06:14:56.931002 #9047]  INFO -- : listening on addr=127.0.0.1:8080 fd=12
I, [2015-02-13T06:14:56.931381 #9047]  INFO -- : listening on addr=/var/opt/gitlab/gitlab-rails/sockets/gitlab.socket fd=13
I, [2015-02-13T06:14:56.936638 #9047]  INFO -- : master process ready
I, [2015-02-13T06:14:56.946504 #9092]  INFO -- : worker=0 spawned pid=9092
I, [2015-02-13T06:14:56.946943 #9092]  INFO -- : worker=0 ready
I, [2015-02-13T06:14:56.947892 #9094]  INFO -- : worker=1 spawned pid=9094
I, [2015-02-13T06:14:56.948181 #9094]  INFO -- : worker=1 ready
W, [2015-02-13T07:16:01.312916 #9094]  WARN -- : #<Unicorn::HttpServer:0x0000000208f618>: worker (pid: 9094) exceeds memory limit (320626688 bytes > 247066940 bytes)
W, [2015-02-13T07:16:01.313000 #9094]  WARN -- : Unicorn::WorkerKiller send SIGQUIT (pid: 9094) alive: 3621 sec (trial 1)
I, [2015-02-13T07:16:01.530733 #9047]  INFO -- : reaped #<Process::Status: pid 9094 exit 0> worker=1
I, [2015-02-13T07:16:01.534501 #13379]  INFO -- : worker=1 spawned pid=13379
I, [2015-02-13T07:16:01.534848 #13379]  INFO -- : worker=1 ready
```
312
313
314
315

## `repocheck.log`

This file lives in `/var/log/gitlab/gitlab-rails/repocheck.log` for
316
Omnibus GitLab packages or in `/home/git/gitlab/log/repocheck.log` for
317
318
319
320
installations from source.

It logs information whenever a [repository check is run][repocheck] on a project.

321
322
323
324
325
326
## `importer.log`

Introduced in GitLab 11.3. This file lives in `/var/log/gitlab/gitlab-rails/importer.log` for
Omnibus GitLab packages or in `/home/git/gitlab/log/importer.log` for
installations from source.

327
## `auth.log`
328
329
330
331
332

Introduced in GitLab 12.0. This file lives in `/var/log/gitlab/gitlab-rails/auth.log` for
Omnibus GitLab packages or in `/home/git/gitlab/log/auth.log` for
installations from source.

333
334
335
336
This log records:

- Information whenever [Rack Attack] registers an abusive request.
- Requests over the [Rate Limit] on raw endpoints.
337
- [Protected paths] abusive requests.
338

Mayra Cabrera's avatar
Mayra Cabrera committed
339
NOTE: **Note:**
340
341
From [%12.1](https://gitlab.com/gitlab-org/gitlab-foss/issues/62756), user id and username are also
recorded on this log, if available.
Mayra Cabrera's avatar
Mayra Cabrera committed
342

343
344
## `graphql_json.log`

345
> [Introduced](https://gitlab.com/gitlab-org/gitlab-foss/issues/59587) in GitLab 12.0.
charlie ablett's avatar
charlie ablett committed
346

347
348
349
350
This file lives in `/var/log/gitlab/gitlab-rails/graphql_json.log` for
Omnibus GitLab packages or in `/home/git/gitlab/log/graphql_json.log` for
installations from source.

charlie ablett's avatar
charlie ablett committed
351
GraphQL queries are recorded in that file. For example:
352
353

```json
charlie ablett's avatar
charlie ablett committed
354
{"query_string":"query IntrospectionQuery{__schema {queryType { name },mutationType { name }}}...(etc)","variables":{"a":1,"b":2},"complexity":181,"depth":1,"duration":7}
355
356
```

357
358
359
360
361
362
## `migrations.log`

Introduced in GitLab 12.3. This file lives in `/var/log/gitlab/gitlab-rails/migrations.log` for
Omnibus GitLab packages or in `/home/git/gitlab/log/migrations.log` for
installations from source.

363
364
365
366
367
368
369
370
371
372
373
## `mail_room_json.log` (default)

> [Introduced](https://gitlab.com/gitlab-org/gitlab/merge_requests/19186) in GitLab 12.6.

This file lives in `/var/log/gitlab/mail_room/mail_room_json.log` for
Omnibus GitLab packages or in `/home/git/gitlab/log/mail_room_json.log` for
installations from source.

This structured log file records internal activity in the `mail_room` gem.
Its name and path are configurable, so the name and path may not match the above.

Stan Hu's avatar
Stan Hu committed
374
375
## Reconfigure Logs

376
377
Reconfigure log files live in `/var/log/gitlab/reconfigure` for Omnibus GitLab
packages. Installations from source don't have reconfigure logs. A reconfigure log
Stan Hu's avatar
Stan Hu committed
378
379
380
381
382
is populated whenever `gitlab-ctl reconfigure` is run manually or as part of an upgrade.

Reconfigure logs files are named according to the UNIX timestamp of when the reconfigure
was initiated, such as `1509705644.log`

383
## `sidekiq_exporter.log` and `web_exporter.log`
384
385

If Prometheus metrics and the Sidekiq Exporter are both enabled, Sidekiq will
386
start a Web server and listen to the defined port (default: 8082). Access logs
387
388
389
390
will be generated in `/var/log/gitlab/gitlab-rails/sidekiq_exporter.log` for
Omnibus GitLab packages or in `/home/git/gitlab/log/sidekiq_exporter.log` for
installations from source.

391
392
393
394
395
396
If Prometheus metrics and the Web Exporter are both enabled, Unicorn/Puma will
start a Web server and listen to the defined port (default: 8083). Access logs
will be generated in `/var/log/gitlab/gitlab-rails/web_exporter.log` for
Omnibus GitLab packages or in `/home/git/gitlab/log/web_exporter.log` for
installations from source.

397
## `database_load_balancing.log` **(PREMIUM ONLY)**
398

399
> [Introduced](https://gitlab.com/gitlab-org/gitlab/merge_requests/15442) in GitLab 12.3.
400

401
Contains details of GitLab's [Database Load Balancing](database_load_balancing.md).
402
403
404
405
It is stored at:

- `/var/log/gitlab/gitlab-rails/database_load_balancing.log` for Omnibus GitLab packages.
- `/home/git/gitlab/log/database_load_balancing.log` for installations from source.
406

407
408
409
410
411
412
413
414
415
416
417
418
419
420
421
422
423
## `elasticsearch.log`

Introduced in GitLab 12.6. This file lives in
`/var/log/gitlab/gitlab-rails/elasticsearch.log` for Omnibus GitLab
packages or in `/home/git/gitlab/log/elasticsearch.log` for installations
from source.

It logs information related to the Elasticsearch Integration including
errors during indexing or searching Elasticsearch.

Each line contains a JSON line that can be ingested by Elasticsearch, Splunk,
etc. For example:

```json
{"severity":"DEBUG","time":"2019-10-17T06:23:13.227Z","correlation_id":null,"message":"redacted_search_result","class_name":"Milestone","id":2,"ability":"read_milestone","current_user_id":2,"query":"project"}
```

424
425
426
427
428
429
430
431
432
433
434
435
436
437
438
439
440
441
442
443
444
445
446
447
448
449
## `exceptions_json.log`

> [Introduced](https://gitlab.com/gitlab-org/gitlab/merge_requests/17819) in GitLab 12.6.

This file lives in
`/var/log/gitlab/gitlab-rails/exceptions_json.log` for Omnibus GitLab
packages or in `/home/git/gitlab/log/exceptions_json.log` for installations
from source.

It logs the information about exceptions being tracked by `Gitlab::ErrorTracking` which provides standard and consistent way of [processing rescued exceptions](https://gitlab.com/gitlab-org/gitlab/blob/master/doc/development/logging.md#exception-handling).

Each line contains a JSON line that can be ingested by Elasticsearch. For example:

```json
{
  "severity": "ERROR",
  "time": "2019-12-17T11:49:29.485Z",
  "correlation_id": "AbDVUrrTvM1",
  "extra.server": {
    "os": {
      "name": "Darwin",
      "version": "Darwin Kernel Version 19.2.0",
      "build": "19.2.0",
    },
    "runtime": {
      "name": "ruby",
450
      "version": "ruby 2.6.5p114 (2019-10-01 revision 67812) [x86_64-darwin18]"
451
452
453
454
455
456
457
458
459
460
461
462
463
464
    }
  },
  "extra.project_id": 55,
  "extra.relation_key": "milestones",
  "extra.relation_index": 1,
  "exception.class": "NoMethodError",
  "exception.message": "undefined method `strong_memoize' for #<Gitlab::ImportExport::RelationFactory:0x00007fb5d917c4b0>",
  "exception.backtrace": [
    "lib/gitlab/import_export/relation_factory.rb:329:in `unique_relation?'",
    "lib/gitlab/import_export/relation_factory.rb:345:in `find_or_create_object!'"
  ]
}
```

465
466
467
468
[repocheck]: repository_checks.md
[Rack Attack]: ../security/rack_attack.md
[Rate Limit]: ../user/admin_area/settings/rate_limits_on_raw_endpoints.md
[Protected paths]: ../user/admin_area/settings/protected_paths.md