redis.md 39.8 KB
Newer Older
1
2
3
4
---
type: reference
---

5
# Configuring Redis for Scaling and High Availability
Drew Blessing's avatar
Drew Blessing committed
6

7
## Provide your own Redis instance **(CORE ONLY)**
8
9
10

The following are the requirements for providing your own Redis instance:

11
12
13
14
- GitLab 12.0 and later requires Redis version 3.2 or higher. Version 3.2 or higher is recommend as this is
  what ships with the GitLab Omnibus package. Older Redis versions do not
  support an optional count argument to SPOP which is now required for
  [Merge Trains](../../ci/merge_request_pipelines/pipelines_for_merged_results/merge_trains/index.md).
15
16
17
18
19
20
21
22
23
24
- Standalone Redis or Redis high availability with Sentinel are supported. Redis
  Cluster is not supported.
- Managed Redis from cloud providers such as AWS Elasticache will work. If these
  services support high availability, be sure it is not the Redis Cluster type.

Note the Redis node's IP address or hostname, port, and password (if required).
These will be necessary when configuring the GitLab application servers later.

## Redis in a Scaled Environment

25
26
27
This section is relevant for [Scaled Architecture](README.md#scalable-architecture-examples)
environments including [Basic Scaling](README.md#basic-scaling) and
[Full Scaling](README.md#full-scaling).
28

29
### Provide your own Redis instance **(CORE ONLY)**
30

31
32
33
34
If you want to use your own deployed Redis instance(s),
see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only)
for more details. However, you can use the GitLab Omnibus package to easily
deploy the bundled Redis.
35

36
### Standalone Redis using GitLab Omnibus **(CORE ONLY)**
37
38
39
40
41
42

The GitLab Omnibus package can be used to configure a standalone Redis server.
In this configuration Redis is not highly available, and represents a single
point of failure. However, in a scaled environment the objective is to allow
the environment to handle more users or to increase throughput. Redis itself
is generally stable and can handle many requests so it is an acceptable
43
trade off to have only a single instance. See [Scaling and High Availability](README.md)
44
45
46
47
48
49
for an overview of GitLab scaling and high availability options.

The steps below are the minimum necessary to configure a Redis server with
Omnibus:

1. SSH into the Redis server.
Cindy Pallares's avatar
Cindy Pallares committed
50
1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab
51
52
53
54
55
   package you want using **steps 1 and 2** from the GitLab downloads page.
     - Do not complete any other steps on the download page.

1. Edit `/etc/gitlab/gitlab.rb` and add the contents:

56
57
58
59
60
61
62
63
64
65
66
67
68
   ```ruby
   ## Enable Redis
   redis['enable'] = true

   ## Disable all other services
   sidekiq['enable'] = false
   gitlab_workhorse['enable'] = false
   unicorn['enable'] = false
   postgresql['enable'] = false
   nginx['enable'] = false
   prometheus['enable'] = false
   alertmanager['enable'] = false
   pgbouncer_exporter['enable'] = false
69
   gitlab_exporter['enable'] = false
70
71
72
   gitaly['enable'] = false

   redis['bind'] = '0.0.0.0'
73
   redis['port'] = 6379
74
75
   redis['password'] = 'SECRET_PASSWORD_HERE'

76
   gitlab_rails['enable'] = false
77
   ```
78
79
80
81
82

1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
1. Note the Redis node's IP address or hostname, port, and
   Redis password. These will be necessary when configuring the GitLab
   application servers later.
83
1. [Enable Monitoring](#enable-monitoring)
84
85
86
87
88

Advanced configuration options are supported and can be added if
needed.

Continue configuration of other components by going
89
[back to Scaled Architectures](README.md#scalable-architecture-examples)
90
91
92

## Redis with High Availability

93
94
95
96
This section is relevant for [High Availability Architecture](README.md#high-availability-architecture-examples)
environments including [Horizontal](README.md#horizontal),
[Hybrid](README.md#hybrid), and
[Fully Distributed](README.md#fully-distributed).
97

98
### Provide your own Redis instance **(CORE ONLY)**
99

100
101
102
103
If you want to use your own deployed Redis instance(s),
see [Provide your own Redis instance](#provide-your-own-redis-instance-core-only)
for more details. However, you can use the GitLab Omnibus package to easily
deploy the bundled Redis.
104

105
### High Availability with GitLab Omnibus **(PREMIUM ONLY)**
106
107

> Experimental Redis Sentinel support was [introduced in GitLab 8.11][ce-1877].
108
109
110
111
Starting with 8.14, Redis Sentinel is no longer experimental.
If you've used it with versions `< 8.14` before, please check the updated
documentation here.

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
112
113
High Availability with [Redis] is possible using a **Master** x **Slave**
topology with a [Redis Sentinel][sentinel] service to watch and automatically
114
start the failover procedure.
Drew Blessing's avatar
Drew Blessing committed
115

116
You can choose to install and manage Redis and Sentinel yourself, use
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
117
118
a hosted cloud solution or you can use the one that comes bundled with
Omnibus GitLab packages.
Drew Blessing's avatar
Drew Blessing committed
119

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
120
> **Notes:**
Evan Read's avatar
Evan Read committed
121
>
122
> - Redis requires authentication for High Availability. See
123
>  [Redis Security](https://redis.io/topics/security) documentation for more
124
125
126
127
128
129
130
131
132
133
134
135
>  information. We recommend using a combination of a Redis password and tight
>  firewall rules to secure your Redis service.
> - You are highly encouraged to read the [Redis Sentinel][sentinel] documentation
>  before configuring Redis HA with GitLab to fully understand the topology and
>  architecture.
> - This is the documentation for the Omnibus GitLab packages. For installations
>  from source, follow the [Redis HA source installation](redis_source.md) guide.
> - Redis Sentinel daemon is bundled with Omnibus GitLab Enterprise Edition only.
>  For configuring Sentinel with the Omnibus GitLab Community Edition and
>  installations from source, read the
>  [Available configuration setups](#available-configuration-setups) section
>  below.
136

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
137
## Overview
138

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
139
Before diving into the details of setting up Redis and Redis Sentinel for HA,
140
141
make sure you read this Overview section to better understand how the components
are tied together.
142

143
144
145
146
147
You need at least `3` independent machines: physical, or VMs running into
distinct physical machines. It is essential that all master and slaves Redis
instances run in different machines. If you fail to provision the machines in
that specific way, any issue with the shared environment can bring your entire
setup down.
148

Lee Matos's avatar
Lee Matos committed
149
150
It is OK to run a Sentinel alongside of a master or slave Redis instance.
There should be no more than one Sentinel on the same machine though.
151

Lee Matos's avatar
Lee Matos committed
152
You also need to take into consideration the underlying network topology,
153
154
155
making sure you have redundant connectivity between Redis / Sentinel and
GitLab instances, otherwise the networks will become a single point of
failure.
156

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
157
158
Make sure that you read this document once as a whole before configuring the
components below.
159

160
> **Notes:**
Evan Read's avatar
Evan Read committed
161
>
162
163
164
165
> - Starting with GitLab `8.11`, you can configure a list of Redis Sentinel
>   servers that will monitor a group of Redis servers to provide failover support.
> - Starting with GitLab `8.14`, the Omnibus GitLab Enterprise Edition package
>  comes with Redis Sentinel daemon built-in.
Ben Bodenmiller's avatar
Ben Bodenmiller committed
166

167
High Availability with Redis requires a few things:
168

169
- Multiple Redis instances
170
171
- Run Redis in a **Master** x **Slave** topology
- Multiple Sentinel instances
172
- Application support and visibility to all Sentinel and Redis instances
173

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
174
Redis Sentinel can handle the most important tasks in an HA environment and that's
175
to help keep servers online with minimal to no downtime. Redis Sentinel:
176

177
- Monitors **Master** and **Slaves** instances to see if they are available
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
178
- Promotes a **Slave** to **Master** when the **Master** fails
179
- Demotes a **Master** to **Slave** when the failed **Master** comes back online
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
180
  (to prevent data-partitioning)
181
182
- Can be queried by the application to always connect to the current **Master**
  server
183

184
185
186
When a **Master** fails to respond, it's the application's responsibility
(in our case GitLab) to handle timeout and reconnect (querying a **Sentinel**
for a new **Master**).
187

188
To get a better understanding on how to correctly set up Sentinel, please read
189
the [Redis Sentinel documentation](https://redis.io/topics/sentinel) first, as
190
failing to configure it correctly can lead to data loss or can bring your
191
whole cluster down, invalidating the failover effort.
192

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
193
194
195
### Recommended setup

For a minimal setup, you will install the Omnibus GitLab package in `3`
196
**independent** machines, both with **Redis** and **Sentinel**:
197

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
198
199
200
- Redis Master + Sentinel
- Redis Slave + Sentinel
- Redis Slave + Sentinel
201

202
If you are not sure or don't understand why and where the amount of nodes come
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
203
204
from, read [Redis setup overview](#redis-setup-overview) and
[Sentinel setup overview](#sentinel-setup-overview).
205

206
207
For a recommended setup that can resist more failures, you will install
the Omnibus GitLab package in `5` **independent** machines, both with
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
208
**Redis** and **Sentinel**:
209

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
210
211
212
213
214
- Redis Master + Sentinel
- Redis Slave + Sentinel
- Redis Slave + Sentinel
- Redis Slave + Sentinel
- Redis Slave + Sentinel
215

216
### Redis setup overview
217

218
You must have at least `3` Redis servers: `1` Master, `2` Slaves, and they
Lee Matos's avatar
Lee Matos committed
219
need to each be on independent machines (see explanation above).
220

221
222
223
You can have additional Redis nodes, that will help survive a situation
where more nodes goes down. Whenever there is only `2` nodes online, a failover
will not be initiated.
224

225
226
As an example, if you have `6` Redis nodes, a maximum of `3` can be
simultaneously down.
227

228
229
Please note that there are different requirements for Sentinel nodes.
If you host them in the same Redis machines, you may need to take
230
that restrictions into consideration when calculating the amount of
231
nodes to be provisioned. See [Sentinel setup overview](#sentinel-setup-overview)
232
documentation for more information.
233

234
235
All Redis nodes should be configured the same way and with similar server specs, as
in a failover situation, any **Slave** can be promoted as the new **Master** by
236
the Sentinel servers.
237

238
239
240
The replication requires authentication, so you need to define a password to
protect all Redis nodes and the Sentinels. They will all share the same
password, and all instances must be able to talk to
241
each other over the network.
242

243
### Sentinel setup overview
244

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
245
Sentinels watch both other Sentinels and Redis nodes. Whenever a Sentinel
246
detects that a Redis node is not responding, it will announce that to the
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
247
248
other Sentinels. They have to reach the **quorum**, that is the minimum amount
of Sentinels that agrees a node is down, in order to be able to start a failover.
249

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
250
251
Whenever the **quorum** is met, the **majority** of all known Sentinel nodes
need to be available and reachable, so that they can elect the Sentinel **leader**
252
who will take all the decisions to restore the service availability by:
253

254
255
256
257
- Promoting a new **Master**
- Reconfiguring the other **Slaves** and make them point to the new **Master**
- Announce the new **Master** to every other Sentinel peer
- Reconfigure the old **Master** and demote to **Slave** when it comes back online
258

259
You must have at least `3` Redis Sentinel servers, and they need to
Ville Skyttä's avatar
Ville Skyttä committed
260
be each in an independent machine (that are believed to fail independently),
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
261
ideally in different geographical areas.
262

263
264
265
You can configure them in the same machines where you've configured the other
Redis servers, but understand that if a whole node goes down, you loose both
a Sentinel and a Redis instance.
266

267
268
The number of sentinels should ideally always be an **odd** number, for the
consensus algorithm to be effective in the case of a failure.
269

270
271
272
In a `3` nodes topology, you can only afford `1` Sentinel node going down.
Whenever the **majority** of the Sentinels goes down, the network partition
protection prevents destructive actions and a failover **will not be started**.
273

274
Here are some examples:
275

276
277
- With `5` or `6` sentinels, a maximum of `2` can go down for a failover begin.
- With `7` sentinels, a maximum of `3` nodes can go down.
278

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
279
The **Leader** election can sometimes fail the voting round when **consensus**
280
281
282
is not achieved (see the odd number of nodes requirement above). In that case,
a new attempt will be made after the amount of time defined in
`sentinel['failover_timeout']` (in milliseconds).
283

284
285
286
>**Note:**
We will see where `sentinel['failover_timeout']` is defined later.

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
287
The `failover_timeout` variable has a lot of different use cases. According to
288
the official documentation:
289
290
291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307

- The time needed to re-start a failover after a previous failover was
  already tried against the same master by a given Sentinel, is two
  times the failover timeout.

- The time needed for a slave replicating to a wrong master according
  to a Sentinel current configuration, to be forced to replicate
  with the right master, is exactly the failover timeout (counting since
  the moment a Sentinel detected the misconfiguration).

- The time needed to cancel a failover that is already in progress but
  did not produced any configuration change (SLAVEOF NO ONE yet not
  acknowledged by the promoted slave).

- The maximum time a failover in progress waits for all the slaves to be
  reconfigured as slaves of the new master. However even after this time
  the slaves will be reconfigured by the Sentinels anyway, but not with
  the exact parallel-syncs progression as specified.

308
309
310
311
312
313
314
315
316
317
### Available configuration setups

Based on your infrastructure setup and how you have installed GitLab, there are
multiple ways to configure Redis HA. Omnibus GitLab packages have Redis and/or
Redis Sentinel bundled with them so you only need to focus on configuration.
Pick the one that suits your needs.

- [Installations from source][source]: You need to install Redis and Sentinel
  yourself. Use the [Redis HA installation from source](redis_source.md)
  documentation.
318
- [Omnibus GitLab **Community Edition** (CE) package](https://about.gitlab.com/install/?version=ce): Redis is bundled, so you
319
320
321
322
  can use the package with only the Redis service enabled as described in steps
  1 and 2 of this document (works for both master and slave setups). To install
  and configure Sentinel, jump directly to the Sentinel section in the
  [Redis HA installation from source](redis_source.md#step-3-configuring-the-redis-sentinel-instances) documentation.
323
- [Omnibus GitLab **Enterprise Edition** (EE) package](https://about.gitlab.com/install/?version=ee): Both Redis and Sentinel
324
  are bundled in the package, so you can use the EE package to set up the whole
325
326
327
328
329
330
331
332
  Redis HA infrastructure (master, slave and Sentinel) which is described in
  this document.
- If you have installed GitLab using the Omnibus GitLab packages (CE or EE),
  but you want to use your own external Redis server, follow steps 1-3 in the
  [Redis HA installation from source](redis_source.md) documentation, then go
  straight to step 4 in this guide to
  [set up the GitLab application](#step-4-configuring-the-gitlab-application).

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
333
## Configuring Redis HA
334

335
This is the section where we install and set up the new Redis instances.
336

337
> **Notes:**
Evan Read's avatar
Evan Read committed
338
>
339
340
341
342
343
344
> - We assume that you have installed GitLab and all HA components from scratch. If you
>   already have it installed and running, read how to
>   [switch from a single-machine installation to Redis HA](#switching-from-an-existing-single-machine-installation-to-redis-ha).
> - Redis nodes (both master and slaves) will need the same password defined in
>   `redis['password']`. At any time during a failover the Sentinels can
>  reconfigure a node and change its status from master to slave and vice versa.
345

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
346
### Prerequisites
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
347

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
348
The prerequisites for a HA Redis setup are the following:
349

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
350
351
1. Provision the minimum required number of instances as specified in the
   [recommended setup](#recommended-setup) section.
Lee Matos's avatar
Lee Matos committed
352
353
354
1. We **Do not** recommend installing Redis or Redis Sentinel in the same machines your
   GitLab application is running on as this weakens your HA configuration. You can however opt in to install Redis
   and Sentinel in the same machine.
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
355
356
357
358
359
1. All Redis nodes must be able to talk to each other and accept incoming
   connections over Redis (`6379`) and Sentinel (`26379`) ports (unless you
   change the default ones).
1. The server that hosts the GitLab application must be able to access the
   Redis nodes.
360
1. Protect the nodes from access from external networks ([Internet][it]), using
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
361
   firewall.
362

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
363
### Step 1. Configuring the master Redis instance
364

365
1. SSH into the **master** Redis server.
Cindy Pallares's avatar
Cindy Pallares committed
366
1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab
367
   package you want using **steps 1 and 2** from the GitLab downloads page.
368
369
370
   - Make sure you select the correct Omnibus package, with the same version
     and type (Community, Enterprise editions) of your current install.
   - Do not complete any other steps on the download page.
371

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
372
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
373

374
375
376
   ```ruby
   # Specify server role as 'redis_master_role'
   roles ['redis_master_role']
377

378
379
380
381
382
   # IP address pointing to a local IP that the other machines can reach to.
   # You can also set bind to '0.0.0.0' which listen in all interfaces.
   # If you really need to bind to an external accessible IP, make
   # sure you add extra firewall rules to prevent unauthorized access.
   redis['bind'] = '10.0.0.1'
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
383

384
385
386
   # Define a port so Redis can listen for TCP requests which will allow other
   # machines to connect to it.
   redis['port'] = 6379
387

388
389
390
   # Set up password authentication for Redis (use the same password in all nodes).
   redis['password'] = 'redis-password-goes-here'
   ```
391

392
393
394
1. Only the primary GitLab application server should handle migrations. To
   prevent database migrations from running on upgrade, add the following
   configuration to your `/etc/gitlab/gitlab.rb` file:
395

396
   ```ruby
397
398
   gitlab_rails['auto_migrate'] = false
   ```
399

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
400
1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
401

402
> Note: You can specify multiple roles like sentinel and Redis as:
403
> `roles ['redis_sentinel_role', 'redis_master_role']`. Read more about high
Evan Read's avatar
Evan Read committed
404
> availability roles at <https://docs.gitlab.com/omnibus/roles/>.
405

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
406
### Step 2. Configuring the slave Redis instances
407

408
1. SSH into the **slave** Redis server.
Cindy Pallares's avatar
Cindy Pallares committed
409
1. [Download/install](https://about.gitlab.com/install/) the Omnibus GitLab
410
   package you want using **steps 1 and 2** from the GitLab downloads page.
411
412
413
   - Make sure you select the correct Omnibus package, with the same version
     and type (Community, Enterprise editions) of your current install.
   - Do not complete any other steps on the download page.
414

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
415
1. Edit `/etc/gitlab/gitlab.rb` and add the contents:
416

417
418
419
   ```ruby
   # Specify server role as 'redis_slave_role'
   roles ['redis_slave_role']
420

421
422
423
424
425
   # IP address pointing to a local IP that the other machines can reach to.
   # You can also set bind to '0.0.0.0' which listen in all interfaces.
   # If you really need to bind to an external accessible IP, make
   # sure you add extra firewall rules to prevent unauthorized access.
   redis['bind'] = '10.0.0.2'
426

427
428
429
   # Define a port so Redis can listen for TCP requests which will allow other
   # machines to connect to it.
   redis['port'] = 6379
430

431
432
   # The same password for Redis authentication you set up for the master node.
   redis['password'] = 'redis-password-goes-here'
433

434
435
   # The IP of the master Redis node.
   redis['master_ip'] = '10.0.0.1'
436

437
438
439
440
   # Port of master Redis server, uncomment to change to non default. Defaults
   # to `6379`.
   #redis['master_port'] = 6379
   ```
441

442
1. To prevent reconfigure from running automatically on upgrade, run:
443

444
   ```shell
445
446
   sudo touch /etc/gitlab/skip-auto-reconfigure
   ```
447

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
448
449
1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
1. Go through the steps again for all the other slave nodes.
450

451
> Note: You can specify multiple roles like sentinel and Redis as:
452
> `roles ['redis_sentinel_role', 'redis_slave_role']`. Read more about high
Evan Read's avatar
Evan Read committed
453
> availability roles at <https://docs.gitlab.com/omnibus/roles/>.
454

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
455
---
456

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
457
These values don't have to be changed again in `/etc/gitlab/gitlab.rb` after
458
459
460
a failover, as the nodes will be managed by the Sentinels, and even after a
`gitlab-ctl reconfigure`, they will get their configuration restored by
the same Sentinels.
461

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
462
### Step 3. Configuring the Redis Sentinel instances
463

464
>**Note:**
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
465
466
467
468
Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. The
following section assumes you are using Omnibus GitLab Enterprise Edition.
For the Omnibus Community Edition and installations from source, follow the
[Redis HA source install](redis_source.md) guide.
469

470
471
472
473
474
475
NOTE: **Note:** If you are using an external Redis Sentinel instance, be sure
to exclude the `requirepass` parameter from the Sentinel
configuration. This parameter will cause clients to report `NOAUTH
Authentication required.`. [Redis Sentinel 3.2.x does not support
password authentication](https://github.com/antirez/redis/issues/3279).

476
477
Now that the Redis servers are all set up, let's configure the Sentinel
servers.
478

479
If you are not sure if your Redis servers are working and replicating
Marcel Amirault's avatar
Marcel Amirault committed
480
correctly, please read the [Troubleshooting Replication](#troubleshooting-redis-replication)
481
482
483
and fix it before proceeding with Sentinel setup.

You must have at least `3` Redis Sentinel servers, and they need to
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
484
be each in an independent machine. You can configure them in the same
485
486
machines where you've configured the other Redis servers.

487
With GitLab Enterprise Edition, you can use the Omnibus package to set up
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
488
489
490
491
multiple machines with the Sentinel daemon.

---

492
1. SSH into the server that will host Redis Sentinel.
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
493
494
495
1. **You can omit this step if the Sentinels will be hosted in the same node as
   the other Redis instances.**

496
   [Download/install](https://about.gitlab.com/install/) the
497
498
499
500
501
   Omnibus GitLab Enterprise Edition package using **steps 1 and 2** from the
   GitLab downloads page.
   - Make sure you select the correct Omnibus package, with the same version
     the GitLab application is running.
   - Do not complete any other steps on the download page.
502

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
503
504
505
1. Edit `/etc/gitlab/gitlab.rb` and add the contents (if you are installing the
   Sentinels in the same node as the other Redis instances, some values might
   be duplicate below):
506

507
508
509
510
511
512
513
514
515
516
517
518
519
520
521
522
523
524
525
526
527
528
529
530
531
532
533
534
535
536
537
538
539
540
541
542
543
544
545
546
547
548
549
550
551
552
553
554
555
556
557
558
559
560
561
562
563
564
565
566
567
568
569
570
   ```ruby
   roles ['redis_sentinel_role']

   # Must be the same in every sentinel node
   redis['master_name'] = 'gitlab-redis'

   # The same password for Redis authentication you set up for the master node.
   redis['master_password'] = 'redis-password-goes-here'

   # The IP of the master Redis node.
   redis['master_ip'] = '10.0.0.1'

   # Define a port so Redis can listen for TCP requests which will allow other
   # machines to connect to it.
   redis['port'] = 6379

   # Port of master Redis server, uncomment to change to non default. Defaults
   # to `6379`.
   #redis['master_port'] = 6379

   ## Configure Sentinel
   sentinel['bind'] = '10.0.0.1'

   # Port that Sentinel listens on, uncomment to change to non default. Defaults
   # to `26379`.
   # sentinel['port'] = 26379

   ## Quorum must reflect the amount of voting sentinels it take to start a failover.
   ## Value must NOT be greater then the amount of sentinels.
   ##
   ## The quorum can be used to tune Sentinel in two ways:
   ## 1. If a the quorum is set to a value smaller than the majority of Sentinels
   ##    we deploy, we are basically making Sentinel more sensible to master failures,
   ##    triggering a failover as soon as even just a minority of Sentinels is no longer
   ##    able to talk with the master.
   ## 1. If a quorum is set to a value greater than the majority of Sentinels, we are
   ##    making Sentinel able to failover only when there are a very large number (larger
   ##    than majority) of well connected Sentinels which agree about the master being down.s
   sentinel['quorum'] = 2

   ## Consider unresponsive server down after x amount of ms.
   # sentinel['down_after_milliseconds'] = 10000

   ## Specifies the failover timeout in milliseconds. It is used in many ways:
   ##
   ## - The time needed to re-start a failover after a previous failover was
   ##   already tried against the same master by a given Sentinel, is two
   ##   times the failover timeout.
   ##
   ## - The time needed for a slave replicating to a wrong master according
   ##   to a Sentinel current configuration, to be forced to replicate
   ##   with the right master, is exactly the failover timeout (counting since
   ##   the moment a Sentinel detected the misconfiguration).
   ##
   ## - The time needed to cancel a failover that is already in progress but
   ##   did not produced any configuration change (SLAVEOF NO ONE yet not
   ##   acknowledged by the promoted slave).
   ##
   ## - The maximum time a failover in progress waits for all the slaves to be
   ##   reconfigured as slaves of the new master. However even after this time
   ##   the slaves will be reconfigured by the Sentinels anyway, but not with
   ##   the exact parallel-syncs progression as specified.
   # sentinel['failover_timeout'] = 60000
   ```
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
571
572
573

1. To prevent database migrations from running on upgrade, run:

574
   ```shell
575
576
   sudo touch /etc/gitlab/skip-auto-reconfigure
   ```
577

578
   Only the primary GitLab application server should handle migrations.
579

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
580
581
1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
1. Go through the steps again for all the other Sentinel nodes.
582

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
583
### Step 4. Configuring the GitLab application
584

585
586
The final part is to inform the main GitLab application server of the Redis
Sentinels servers and authentication credentials.
587

588
589
590
You can enable or disable Sentinel support at any time in new or existing
installations. From the GitLab application perspective, all it requires is
the correct credentials for the Sentinel nodes.
591

592
While it doesn't require a list of all Sentinel nodes, in case of a failure,
593
it needs to access at least one of the listed.
594

595
596
>**Note:**
The following steps should be performed in the [GitLab application server](gitlab.md)
597
which ideally should not have Redis or Sentinels on it for a HA setup.
598

599
1. SSH into the server where the GitLab application is installed.
600
1. Edit `/etc/gitlab/gitlab.rb` and add/change the following lines:
601

602
603
604
605
606
607
608
609
610
611
612
613
614
615
   ```ruby
   ## Must be the same in every sentinel node
   redis['master_name'] = 'gitlab-redis'

   ## The same password for Redis authentication you set up for the master node.
   redis['master_password'] = 'redis-password-goes-here'

   ## A list of sentinels with `host` and `port`
   gitlab_rails['redis_sentinels'] = [
     {'host' => '10.0.0.1', 'port' => 26379},
     {'host' => '10.0.0.2', 'port' => 26379},
     {'host' => '10.0.0.3', 'port' => 26379}
   ]
   ```
616

617
1. [Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
618

619
620
621
622
623
624
625
626
627
628
629
630
631
632
## Switching from an existing single-machine installation to Redis HA

If you already have a single-machine GitLab install running, you will need to
replicate from this machine first, before de-activating the Redis instance
inside it.

Your single-machine install will be the initial **Master**, and the `3` others
should be configured as **Slave** pointing to this machine.

After replication catches up, you will need to stop services in the
single-machine install, to rotate the **Master** to one of the new nodes.

Make the required changes in configuration and restart the new nodes again.

633
To disable Redis in the single install, edit `/etc/gitlab/gitlab.rb`:
634
635
636

```ruby
redis['enable'] = false
637
638
```

639
640
If you fail to replicate first, you may loose data (unprocessed background jobs).

641
## Example of a minimal configuration with 1 master, 2 slaves and 3 Sentinels
642

643
644
645
646
647
>**Note:**
Redis Sentinel is bundled with Omnibus GitLab Enterprise Edition only. For
different setups, read the
[available configuration setups](#available-configuration-setups) section.

648
649
650
651
In this example we consider that all servers have an internal network
interface with IPs in the `10.0.0.x` range, and that they can connect
to each other using these IPs.

652
In a real world usage, you would also set up firewall rules to prevent
653
unauthorized access from other machines and block traffic from the
654
655
656
outside (Internet).

We will use the same `3` nodes with **Redis** + **Sentinel** topology
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
657
658
discussed in [Redis setup overview](#redis-setup-overview) and
[Sentinel setup overview](#sentinel-setup-overview) documentation.
659
660
661

Here is a list and description of each **machine** and the assigned **IP**:

662
663
664
665
- `10.0.0.1`: Redis Master + Sentinel 1
- `10.0.0.2`: Redis Slave 1 + Sentinel 2
- `10.0.0.3`: Redis Slave 2 + Sentinel 3
- `10.0.0.4`: GitLab application
666
667
668
669
670
671

Please note that after the initial configuration, if a failover is initiated
by the Sentinel nodes, the Redis nodes will be reconfigured and the **Master**
will change permanently (including in `redis.conf`) from one node to the other,
until a new failover is initiated again.

672
The same thing will happen with `sentinel.conf` that will be overridden after the
673
674
initial execution, after any new sentinel node starts watching the **Master**,
or a failover promotes a different **Master** node.
675

676
### Example configuration for Redis master and Sentinel 1
677

678
In `/etc/gitlab/gitlab.rb`:
679
680

```ruby
681
roles ['redis_sentinel_role', 'redis_master_role']
682
redis['bind'] = '10.0.0.1'
683
redis['port'] = 6379
684
redis['password'] = 'redis-password-goes-here'
685
redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node
686
redis['master_password'] = 'redis-password-goes-here' # the same value defined in redis['password'] in the master instance
687
redis['master_ip'] = '10.0.0.1' # ip of the initial master redis instance
688
689
690
691
692
693
694
#redis['master_port'] = 6379 # port of the initial master redis instance, uncomment to change to non default
sentinel['bind'] = '10.0.0.1'
# sentinel['port'] = 26379 # uncomment to change default port
sentinel['quorum'] = 2
# sentinel['down_after_milliseconds'] = 10000
# sentinel['failover_timeout'] = 60000
```
695

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
696
[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.
697

698
### Example configuration for Redis slave 1 and Sentinel 2
699

700
In `/etc/gitlab/gitlab.rb`:
701
702

```ruby
703
roles ['redis_sentinel_role', 'redis_slave_role']
704
705
706
707
708
709
redis['bind'] = '10.0.0.2'
redis['port'] = 6379
redis['password'] = 'redis-password-goes-here'
redis['master_password'] = 'redis-password-goes-here'
redis['master_ip'] = '10.0.0.1' # IP of master Redis server
#redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default
710
711
712
713
redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node
sentinel['bind'] = '10.0.0.2'
# sentinel['port'] = 26379 # uncomment to change default port
sentinel['quorum'] = 2
714
# sentinel['down_after_milliseconds'] = 10000
715
716
# sentinel['failover_timeout'] = 60000
```
717

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
718
719
[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.

720
### Example configuration for Redis slave 2 and Sentinel 3
721
722

In `/etc/gitlab/gitlab.rb`:
723

724
```ruby
725
roles ['redis_sentinel_role', 'redis_slave_role']
726
727
728
729
730
731
redis['bind'] = '10.0.0.3'
redis['port'] = 6379
redis['password'] = 'redis-password-goes-here'
redis['master_password'] = 'redis-password-goes-here'
redis['master_ip'] = '10.0.0.1' # IP of master Redis server
#redis['master_port'] = 6379 # Port of master Redis server, uncomment to change to non default
732
redis['master_name'] = 'gitlab-redis' # must be the same in every sentinel node
733
sentinel['bind'] = '10.0.0.3'
734
735
736
# sentinel['port'] = 26379 # uncomment to change default port
sentinel['quorum'] = 2
# sentinel['down_after_milliseconds'] = 10000
737
738
739
# sentinel['failover_timeout'] = 60000
```

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
740
741
742
743
744
[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.

### Example configuration for the GitLab application

In `/etc/gitlab/gitlab.rb`:
745
746

```ruby
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
747
redis['master_name'] = 'gitlab-redis'
748
redis['master_password'] = 'redis-password-goes-here'
Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
749
750
751
752
753
754
755
756
757
gitlab_rails['redis_sentinels'] = [
  {'host' => '10.0.0.1', 'port' => 26379},
  {'host' => '10.0.0.2', 'port' => 26379},
  {'host' => '10.0.0.3', 'port' => 26379}
]
```

[Reconfigure Omnibus GitLab][reconfigure] for the changes to take effect.

758
759
760
761
## Enable Monitoring

> [Introduced](https://gitlab.com/gitlab-org/omnibus-gitlab/issues/3786) in GitLab 12.0.

762
763
764
If you enable Monitoring, it must be enabled on **all** Redis servers.

1. Make sure to collect [`CONSUL_SERVER_NODES`](database.md#consul-information), which are the IP addresses or DNS records of the Consul server nodes, for the next step. Note they are presented as `Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z`
765

766
1. Create/edit `/etc/gitlab/gitlab.rb` and add the following configuration:
767

768
769
770
771
   ```ruby
   # Enable service discovery for Prometheus
   consul['enable'] = true
   consul['monitoring_service_discovery'] =  true
772

773
774
775
776
777
778
   # Replace placeholders
   # Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z
   # with the addresses of the Consul server nodes
   consul['configuration'] = {
      retry_join: %w(Y.Y.Y.Y consul1.gitlab.example.com Z.Z.Z.Z),
   }
779

780
781
782
783
   # Set the network addresses that the exporters will listen on
   node_exporter['listen_address'] = '0.0.0.0:9100'
   redis_exporter['listen_address'] = '0.0.0.0:9121'
   ```
784

785
1. Run `sudo gitlab-ctl reconfigure` to compile the configuration.
786

787
788
789
790
791
## Advanced configuration

Omnibus GitLab configures some things behind the curtains to make the sysadmins'
lives easier. If you want to know what happens underneath keep reading.

792
793
794
795
796
797
798
799
### Running multiple Redis clusters

GitLab supports running [separate Redis clusters for different persistent
classes](https://docs.gitlab.com/omnibus/settings/redis.html#running-with-multiple-redis-instances):
cache, queues, and shared_state. To make this work with Sentinel:

1. Set the appropriate variable in `/etc/gitlab/gitlab.rb` for each instance you are using:

800
801
802
803
804
805
   ```ruby
   gitlab_rails['redis_cache_instance'] = REDIS_CACHE_URL
   gitlab_rails['redis_queues_instance'] = REDIS_QUEUES_URL
   gitlab_rails['redis_shared_state_instance'] = REDIS_SHARED_STATE_URL
   ```

806
807
    **Note**: Redis URLs should be in the format: `redis://:PASSWORD@SENTINEL_MASTER_NAME`

808
809
810
   1. PASSWORD is the plaintext password for the Redis instance
   1. SENTINEL_MASTER_NAME is the Sentinel master name (e.g. `gitlab-redis-cache`)

811
812
1. Include an array of hashes with host/port combinations, such as the following:

813
814
815
816
817
818
819
820
821
822
823
824
825
826
827
   ```ruby
   gitlab_rails['redis_cache_sentinels'] = [
     { host: REDIS_CACHE_SENTINEL_HOST, port: PORT1 },
     { host: REDIS_CACHE_SENTINEL_HOST2, port: PORT2 }
   ]
   gitlab_rails['redis_queues_sentinels'] = [
     { host: REDIS_QUEUES_SENTINEL_HOST, port: PORT1 },
     { host: REDIS_QUEUES_SENTINEL_HOST2, port: PORT2 }
   ]
   gitlab_rails['redis_shared_state_sentinels'] = [
     { host: SHARED_STATE_SENTINEL_HOST, port: PORT1 },
     { host: SHARED_STATE_SENTINEL_HOST2, port: PORT2 }
   ]
   ```

828
829
1. Note that for each persistence class, GitLab will default to using the
   configuration specified in `gitlab_rails['redis_sentinels']` unless
George Tsiolis's avatar
George Tsiolis committed
830
   overridden by the settings above.
831
832
833
834
835
1. Be sure to include BOTH configuration options for each persistent classes. For example,
   if you choose to configure a cache instance, you must specify both `gitlab_rails['redis_cache_instance']`
   and `gitlab_rails['redis_cache_sentinels']` for GitLab to generate the proper configuration files.
1. Run `gitlab-ctl reconfigure`

836
837
### Control running services

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
838
839
In the previous example, we've used `redis_sentinel_role` and
`redis_master_role` which simplifies the amount of configuration changes.
840
841
842
843
844
845
846
847

If you want more control, here is what each one sets for you automatically
when enabled:

```ruby
## Redis Sentinel Role
redis_sentinel_role['enable'] = true

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
848
# When Sentinel Role is enabled, the following services are also enabled
849
850
sentinel['enable'] = true

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
851
# The following services are disabled
852
853
854
855
856
857
858
redis['enable'] = false
bootstrap['enable'] = false
nginx['enable'] = false
postgresql['enable'] = false
gitlab_rails['enable'] = false
mailroom['enable'] = false

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
859
-------
860

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
861
## Redis master/slave Role
862
863
redis_master_role['enable'] = true # enable only one of them
redis_slave_role['enable'] = true # enable only one of them
864

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
865
866
867
# When Redis Master or Slave role are enabled, the following services are
# enabled/disabled. Note that if Redis and Sentinel roles are combined, both
# services will be enabled.
868

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
869
# The following services are disabled
870
871
872
873
874
875
sentinel['enable'] = false
bootstrap['enable'] = false
nginx['enable'] = false
postgresql['enable'] = false
gitlab_rails['enable'] = false
mailroom['enable'] = false
876

Achilleas Pipinellis's avatar
Achilleas Pipinellis committed
877
# For Redis Slave role, also change this setting from default 'true' to 'false':
878
879
redis['master'] = false
```
880

881
You can find the relevant attributes defined in [gitlab_rails.rb][omnifile].
882

883
## Troubleshooting
884

885
886
There are a lot of moving parts that needs to be taken care carefully
in order for the HA setup to work as expected.
887

888
Before proceeding with the troubleshooting below, check your firewall rules:
889

890
- Redis machines
891
892
  - Accept TCP connection in `6379`
  - Connect to the other Redis machines via TCP in `6379`
893
- Sentinel machines
894
895
896
  - Accept TCP connection in `26379`
  - Connect to other Sentinel machines via TCP in `26379`
  - Connect to the Redis machines via TCP in `6379`
897

898
### Troubleshooting Redis replication
899

900
You can check if everything is correct by connecting to each server using
901
`redis-cli` application, and sending the `info replication` command as below.
902

903
```shell
904
/opt/gitlab/embedded/bin/redis-cli -h <redis-host-or-ip> -a '<redis-password>' info replication
905
```
906

907
When connected to a `master` Redis, you will see the number of connected
908
909
`slaves`, and a list of each with connection details:

910
```plaintext
911
912
913
914
915
916
917
918
919
920
# Replication
role:master
connected_slaves:1
slave0:ip=10.133.5.21,port=6379,state=online,offset=208037514,lag=1
master_repl_offset:208037658
repl_backlog_active:1
repl_backlog_size:1048576
repl_backlog_first_byte_offset:206989083
repl_backlog_histlen:1048576
```
921

922
923
When it's a `slave`, you will see details of the master connection and if
its `up` or `down`:
924

925
```plaintext
926
927
928
929
930
931
932
933
934
935
936
937
938
939
940
941
942
# Replication
role:slave
master_host:10.133.1.58
master_port:6379
master_link_status:up
master_last_io_seconds_ago:1
master_sync_in_progress:0
slave_repl_offset:208096498
slave_priority:100
slave_read_only:1
connected_slaves:0
master_repl_offset:0
repl_backlog_active:0
repl_backlog_size:1048576
repl_backlog_first_byte_offset:0
repl_backlog_histlen:0
```
943

944
### Troubleshooting Sentinel
945
946
947
948
949
950
951
952

If you get an error like: `Redis::CannotConnectError: No sentinels available.`,
there may be something wrong with your configuration files or it can be related
to [this issue][gh-531].

You must make sure you are defining the same value in `redis['master_name']`
and `redis['master_pasword']` as you defined for your sentinel node.

953
The way the Redis connector `redis-rb` works with sentinel is a bit
954
955
956
non-intuitive. We try to hide the complexity in omnibus, but it still requires
a few extra configs.

957
---
Drew Blessing's avatar
Drew Blessing committed
958

959
To make sure your configuration is correct:
Drew Blessing's avatar
Drew Blessing committed
960

961
962
963
1. SSH into your GitLab application server
1. Enter the Rails console:

964
   ```shell
965
966
   # For Omnibus installations
   sudo gitlab-rails console
967

968
969
970
   # For source installations
   sudo -u git rails console production
   ```
971
972

1. Run in the console:
Drew Blessing's avatar
Drew Blessing committed
973

974
975
976
977
   ```ruby
   redis = Redis.new(Gitlab::Redis::SharedState.params)
   redis.info
   ```
Drew Blessing's avatar
Drew Blessing committed
978

979
   Keep this screen open and try to simulate a failover below.
Drew Blessing's avatar
Drew Blessing committed
980

981
982
1. To simulate a failover on master Redis, SSH into the Redis server and run:

983
   ```shell
984
985
986
   # port must match your master redis port, and the sleep time must be a few seconds bigger than defined one
    redis-cli -h localhost -p 6379 DEBUG sleep 20
   ```
Achilleas Pipinellis's avatar