INSTALL_HEPTAPOD.md 12.5 KB
Newer Older
1
2
# Heptapod installation guide

3
## Important message about migrations
4

5
6
Do **not** migrate directly from Heptapod 0.8 to 0.12.
See the dedicated section at the end of this document
7

8
9
10
11
12
## Installing from source

Heptapod installation is similar to GitLab's except that

- some components have to be taken from our Mercurial repositories
13
- we have additional components for Mercurial.
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
- some specific configuration has to be set in place.
- we have more constraints, many of which will be hopefully lifted in
  forthcoming versions

### Limitations

the Rails application, GitLab Shell, Mercurial and the repositories must
be in a single system.

### Fetching the standard GitLab components

Please use the mirror URL for initial and repetitive clonings.

- Rails application (replacement for `gitlab`): https://mirror.octobus.net/heptapod/heptapod
- Heptapod Shell (replacement for GitLab Shell): https://mirror.octobus.net/heptapod/heptapod-shell
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
- Heptapod Workhorse (replacement for GitLab Workhorse): https://mirror.octobus.net/heptapod/heptapod-workhorse

All these have version numbers that are independent (but sometimes coincide) with
their upstream counterparts:

- `VERSION` is the upstream version
- `HEPTAPOD_VERSION` is the Heptapod specific version
- The tag for Heptapod version x.y.z is `heptapod-x.y.z`.
- All tags taking the form `vx.y.z` are upstream tags and must be ignored.

See heptapod#352 for explanation and rationale of the version policy.

As is the case with upstream GitLab, the repository for the Rails application
plays the role of the main repository. It contains additional version files to
specify the versions of Heptapod forked components:
44
45
46

Example:

47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70
- checkout the Rails application at the wished Heptapod version

  ```
     $ hg clone https://mirror.octobus.net/heptapod/heptapod
     $ cd heptapod
     $ hg update heptapod-0.17.0rc1

- check version files:

  ```
     $ ls HEPTAPOD_*_VERSION
     HEPTAPOD_SHELL_VERSION  HEPTAPOD_WORKHORSE_VERSION
     $ cat HEPTAPOD_SHELL_VERSION
     13.7.0
  ```

- checkout a forked component for the wished Heptapod version

  ```
     $ hg clone https://mirror.octobus.net/heptapod/heptapod-shell
     $ cd heptapod-shell
     $ hg up heptapod-13.7.0
  ```

71

72
All other standard GitLab components (Gitaly, etc) have to be taken
73
from the canonical upstream locations, at the version specified at the root
74
of the Rails application sources (`GITALY_SERVER_VERSION` etc).
75

76
### Installing Mercurial components
77
78
79
80

We need a specific Mercurial version, a WSGI server (usually Gunicorn), and
some additional Python packages from source.

81
82
#### Python versions

83
84
85
As of Heptapod 0.17, Heptapod supports Python 3.7 and 3.8 only, with 3.8 being
the recommended version.
Python 2 is not supported any more. Python 3.9 is not supported yet.
86
87
88
89
90

In theory, the minimal Python 3 version should be 3.6.2, but it is completely
untested, whereas developer setups have routinely been running 3.7.

#### Install the Python packages
91
92
93
94
95
96

You can install them system-wide if that suits you (Mercurial has very few
dependencies, it's very unlikely it'll clash with system libraries) or with
`pip install --user` or in a virtualenv. What matters for proper operation is
that they are all importable from the final Mercurial executable.

97
98
99
Note: if you're interested in using Mercurial enhanced with Rust,
you have to preinstall it (see next section about that)

100
101
102
103
Install the Python packages to be taken from pypi.org using the
[requirements file](python/requirements.txt):

```
104
$PIP install -U -r python/requirements.txt
105
106
```

107
108
where `$PIP` is `pip --user` or `some/virtualenv/bin/pip` according to your
choice.
109

110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
167
168
169
170
171
172
173
#### Variant: Mercurial enhanced with Rust

This provides some serious performances boosts, some of them activated by
boolean flags that can be set, e.g., in the global HGRC file.

The Mercurial source distribution includes the Rust code,
but `pip` won't compile it by default.

Here are the steps do to that:

- prerequisite: Rust toolchain (cargo, rustc), version ≥ 1.41.1 as of
  Mercurial 5.6. The general policy of Mercurial is to depend on the version
  available in Debian stable at the time of the Mercurial release.
- use the requirements file intended for Rust instead of the standard one:

  ```
  $PIP install -r python/requirements-rust.txt
  ```
- check: `/path/to/just/installed/hg debuginstall`. Here's an excerpt of
  the expected output:

  ```
  checking Python security support (sni,tls1.0,tls1.1,tls1.2)
  checking Rust extensions (installed)
  checking Mercurial version (5.6.1)
  checking Mercurial custom build ()
  checking module policy (rust+c)
  ```

  "module policy" is the configuration for the runtime usage of native Python
  modules:

  - `rust+c`: use both Python modules written in C and Rust – fail if they
    are missing. This is the default configuration for Mercurial compiled with
    Rust.
  - `c`: use only Python modules written in C – fail if they are missing.
    This is the default configuration for the default compilation of Mercurial
    (C modules, no Rust).

#### Disabling the Rust parts of Mercurial at run time
The Rust parts of Mercurial are still considered experimental.

In some cases, you may want to disable them. This could be for bug
investigation or simply to use a single build script for
installations with different policies.

The resulting Mercurial runs the Rust code by default, but this can be
disabled at runtime.

1. Run the internal Mercurial HTTP service, and in HGitaly with the
   following environment variable:

   ```
   HGMODULEPOLICY=c
   ```

2. In the `mercurial` section of the Heptapod Shell *and* Heptapod Rails
   configuration files, add this:

   ```
   mercurial:
     module_policy: c
   ```

174
175
### Configuration

176
177
178
You need to

1. define Mercurial settings, including some necessary paths
179
2. create Mercurial internal services
180
181
182
183
184
3. configure GitLab components to use the proper Mercurial with those settings
4. take care of few special cases, depending on the Heptapod version

#### Defining Mercurial settings

185
186
187
The `heptapod` Python project, listed in `requirements.txt`,
ships with a configuration file, `required.hgrc`, that performs most
of the required and default Mercurial settings for Heptapod
188
189
(using the `heptapod` extension, permission policy, etc.)

190
191
192
193
194
195
196
197
198
199
200
201
202
203
204
You'll find it where your `$PIP` has installed `heptapod`, which you can
find out like this:

```
export PY_HEPTAPOD=`$PYTHON -c "import heptapod, os; print(os.path.dirname(heptapod.__file__))"`
```

where `$PYTHON` is the python interpreter that's consistent with `$PIP`. So,
for instance if you're using a virtualenv, and `$PIP` is
`/path/to/virtualenv/bin/pip`, then `$PYTHON` is
`/path/to/virtualenv/bin/python`. If you're using
the global pip, then `$PYTHON` is most likely just `python` or `python2`.

You'll need to use `$PY_HEPTAPOD/required.hgrc` and add your local
installation details.
205
206

A simple way is to create a `heptapod.hgrc` file using the `%include` directive.
207
See the example in `$PY_HEPTAPOD/heptapod.example.hgrc`.
208

209
210
211
212
213
214
215
216
217
218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
##### Heptapod >= 0.13: configuring Mercurial to notify the Rails application

Starting with Heptapod 0.13, Mercurial will send notifications about incoming
changesets directly.

For that to work, you need tell it where to find the `.gitlab_shell_secret`
file and configure the internal API address. This is done in the `[heptapod]`
section of your `heptapod.hgrc`:

```
[heptapod]
gitlab-internal-api-secret-file=/path/to/.gitlab_shell_secret
gitlab-internal-api-url=http+unix://%2Fpath%2Fto%2Fgitlab.socket
```

Notes:

* the default internal API URL is `http://localhost:8080`.
* the older `heptapod.gitlab-shell` setting is still supported to find
  the `.gitlab_shell_secret` file

If you're upgrading from Heptapod 0.12 *and* the Rails application is binding
to `localhost:8080` (the default with unicorn) then the configuration should
be readily compatible.


235
### Internal Mercurial HTTP service
236

237
238
HTTP access to Mercurial repository content is provided by an internal HTTP
service that sits behind the Heptapod Workhorse reverse proxy.
239
240

Warning: do NOT have it listen to any other interface than the loopback:
241
its security model is to trust Heptapod Workhorse blindly.
242
243
244
245
246
247
248
249
250
251
252
253

It also needs to load your hgrc file(s). Here's an example, assuming `gunicorn`
is on the path.

```
  HEPTAPOD_HGRC=/path/to/your/heptapod.hgrc gunicorn -w 5 \
      -b 127.0.0.1:8000 \
      --error-logfile /tmp/gunicorn.log \
      --capture-output \
      heptapod.wsgi:hgserve
```

254
255
Again, this `gunicorn` must be able to load the Heptapod Python packages, so
you should use the one that's been installed with `requirements.txt`.
256
257
If you're using the dual virtualenv setup for Python 3 with fallback on
Python 2, then use the symbolic link in the path to gunicorn.
258
259
260

Notes:

261
- Heptapod is not able yet to use Unix domain sockets in this service
262
263
264
- If you want to use another loopback URL, you must also pass it to Heptapod
  Workhorse with the `-hgBackend` option (whose default is
  ``http://localhost:8000``)
265

266
267
268
269
270
271
272
273
274
275
276
277
278
279
280

### new in Heptapod 0.17: HGitaly service

Mercurial repository content is exposed to various GitLab components by the
HGitaly service. Eventually, it will become the only way GitLab can access it,
except when proxying HTTP user requests.

In Heptapod 0.17, it is providing the backend for archive downloads of all
Mercurial projects and powers the (very experimental) Mercurial native
projects.

Again, it needs to know about your HGRC. Here's an example, assuming the `hg`
executable to be on `$PATH`:

```
281
  HGRCPATH=/path/to/your/heptapod.hgrc hg \
282
283
284
285
286
287
288
289
290
291
      --config extensions.hgitaly= \
      hgitaly-serve \
      --listen unix:///path/to/hgitaly.socket
```

Startup and shutdown messages will go to stdout and stderr, as well as
potentially rare cases of uncatched exception (a bug in itself).

Actual logs are emitted as configured in the HGRC file.

292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
Here is a sample systemd service unit file, just fix the paths and/or the
system user and it should do the job.

```
[Unit]
Description=HGitaly, internal Heptapod service for Mercurial handling
After=network.target

[Install]
WantedBy=multi-user.target

[Service]
User=heptapod
Group=heptapod
Environment=HGRCPATH=/path/to/your/heptapod.hgrc
ExecStart=/path/to/virtualenv/bin/hg --config extensions.hgitaly= hgitaly-serve --listen unix:///path/to/hgitaly.socket
Restart=on-failure
```
310

311
312
313
314
#### Using Mercurial in GitLab

Edit the configuration files for the Rails application
(see `config/gitlab.yml.example`) and GitLab Shell (see `config.yml.example`),
315
316
317
318
319
to provide

- the full path to your `heptapod.hgrc` file and to the `hg` executable in
  the `mercurial` section.
- the path to the HGitaly socket in the `repositories` section
320

321
Notes:
322

323
324
 - in those YaML files, `hgrc` must always be a list. This is for separation
   in several files with different persistence properties.
325

326
327
328
329
330
331
332
333
334
335
336
337
338
339
340
341
342
343
### Validation with the functional tests

- start from an empty set of data (do not even set the instance root password)
- install tox (Python3)
- grab the functional tests:
  `hg clone https://dev.heptapod.net/heptapod/heptapod-tests`
- run the tests as the same user as the GitLab components, with something
  like:

  ```
  tox -- --heptapod-ssh-port 2222 \
           --heptapod-url http://localhost:3000 \
           --heptapod-source-install \
           --heptapod-repositories-root ~/heptapod/gdk/gl10-5/repositories
  ```

  (for a quicker first test, you can add `-k push_basic`)

344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
363
364
365
366
367
368
369
370
371
372
373
374
375
376
## Migration notes

With GitLab migrations in general, it's not possible to jump between too much
different versions. There are some intermediate pauses to respect.

Here is how that translates for Heptapod

### from Heptapod 0.8 to 0.12

Because Heptapod 0.12 is a jump from GitLab CE 10.5 to 12.2, it is not
possible to upgrade directly, because major GitLab version bump must be done
[one at a time](https://docs.gitlab.com/ce/policy/maintenance.html#upgrade-recommendations).

Instead, the recommended way is to go through a series of Heptapod intermediate
versions that are meant to support the migration only:

- Heptapod 0.9 (GitLab CE 10.8)
- Heptapod 0.10 (GitLab CE 11.0)
- Heptapod 0.11 (Gitlab CE 11.8)

At each step, start the Rails application and wait for the background
migrations to be finished. See GitLab [documentation](https://docs.gitlab.com/ce/update/#checking-for-background-migrations-before-upgrading) for how to check
that.

For source installs, one only needs to update the Rails application for these
intermediate versions, and it's best not even to start the services that handle
connections from clients (Workhorse, SSH).

For Docker installs, we will provide a series of images tailored for the
migration needs.

**WARNING do not use Heptapod 0.9 to 0.11 for anything but data migration.**