INSTALL_HEPTAPOD.md 14.9 KB
Newer Older
1 2
# Heptapod installation guide

3
## Important message about migrations
4

5 6
Do **not** migrate directly from Heptapod 0.8 to 0.12.
See the dedicated section at the end of this document
7

8 9 10 11 12
## Installing from source

Heptapod installation is similar to GitLab's except that

- some components have to be taken from our Mercurial repositories
13
   and built specifically.
14
- we have additional components for Mercurial.
15 16 17 18 19 20 21 22 23 24 25 26 27 28 29
- some specific configuration has to be set in place.
- we have more constraints, many of which will be hopefully lifted in
  forthcoming versions

### Limitations

the Rails application, GitLab Shell, Mercurial and the repositories must
be in a single system.

### Fetching the standard GitLab components

Please use the mirror URL for initial and repetitive clonings.

- Rails application (replacement for `gitlab`): https://mirror.octobus.net/heptapod/heptapod
- Heptapod Shell (replacement for GitLab Shell): https://mirror.octobus.net/heptapod/heptapod-shell
30 31 32
- As of Heptapod 0.20.0rc1, Heptapod Workhorse sources are bundled with the
  Rails application and don't have a separate repository any more. See the
  dedicated section below for how to build it.
33 34 35 36 37 38 39 40 41 42 43 44 45 46

All these have version numbers that are independent (but sometimes coincide) with
their upstream counterparts:

- `VERSION` is the upstream version
- `HEPTAPOD_VERSION` is the Heptapod specific version
- The tag for Heptapod version x.y.z is `heptapod-x.y.z`.
- All tags taking the form `vx.y.z` are upstream tags and must be ignored.

See heptapod#352 for explanation and rationale of the version policy.

As is the case with upstream GitLab, the repository for the Rails application
plays the role of the main repository. It contains additional version files to
specify the versions of Heptapod forked components:
47 48 49

Example:

50 51 52 53 54 55 56 57 58 59 60
- checkout the Rails application at the wished Heptapod version

  ```
     $ hg clone https://mirror.octobus.net/heptapod/heptapod
     $ cd heptapod
     $ hg update heptapod-0.17.0rc1

- check version files:

  ```
     $ ls HEPTAPOD_*_VERSION
61
     HEPTAPOD_SHELL_VERSION
62 63 64 65 66 67 68 69 70 71 72 73
     $ cat HEPTAPOD_SHELL_VERSION
     13.7.0
  ```

- checkout a forked component for the wished Heptapod version

  ```
     $ hg clone https://mirror.octobus.net/heptapod/heptapod-shell
     $ cd heptapod-shell
     $ hg up heptapod-13.7.0
  ```

74
All other standard GitLab components (Gitaly, etc) have to be taken
75
from the canonical upstream locations, at the version specified at the root
76
of the Rails application sources (`GITALY_SERVER_VERSION` etc).
77

78 79 80 81 82 83 84 85 86 87 88 89 90 91 92 93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131 132
### Building Heptapod Workhorse

Warning: this is new in Heptapod 0.20 series, representing a serious change from
previous instructions.

If you are installing Heptapod 0.19, please consult the [present file in the support branch for 0.19](https://foss.heptapod.net/heptapod/heptapod/-/blob/branch/heptapod-0-19/INSTALL_HEPTAPOD.md) instead.

In Heptapod 0.20, Heptapod Workhorse sources are bundled with the Rails
application, in the `workhorse/` subdirectory. There is no separate Mercurial
repository any more.

Heptapod tracking issue for this is heptapod#402. This parallels the
[same move by upstream](https://gitlab.com/groups/gitlab-org/-/epics/4826).

The `heptapod` branch of the Heptapod Workhorse repository is now officially
obsolete and closed. It does not have the changes for version 0.20. This
repository will get updates in the `heptapod-stable` branch only, until the
end of life of Heptapod 0.19 (on March 22nd, 2021).

Upstream is nearing the end of several months of coexistence between their own
separate repository and the `workhorse/` subdirectory, with automatic
migration of changes etc. By contrast, Heptapod did it in one shot,
between the 0.19 and 0.20 version series.

Upstream documentation recommends to use a rake task to build Workhorse. In
GitLab 13.9.0, this task was updated to work on the bundled version, and works
readily in the Heptapod case:

1. Choose a directory path to host the built binaries, for example
   `/path/to/workhorse/bin`, but make sure this directory does not exist yet.

   Note: the rake task will actually initialize the target directory
   with a `git clone` command, just to bring in a README file.
   If the directory already exists and is
   not a Git clone already, in particular if this is an old Heptapod Workhorse
   Mercurial clone, this will result in an error.

2. Launch the rake task from the Rails application checkout:

   ```
   bundle exec rake 'gitlab:workhorse:install[/path/to/workhorse/bin]`
   ```

   (some shells don't need those quotes. `zsh` does.)

3. Check that this is really Heptapod Workhorse:

   ```
   $ /path/to/workhorse/bin/gitlab-workhorse -version
   gitlab-workhorse heptapod-0.20.0rc1-20210221.191739
   ```

Another option is to run `make` in the `workhorse/` subdirectory. The binaries
will be produced there, including `gitlab-workhorse`, the main one.

133
### Installing Mercurial components
134 135 136 137

We need a specific Mercurial version, a WSGI server (usually Gunicorn), and
some additional Python packages from source.

138 139
#### Python versions

140 141 142
As of Heptapod 0.17, Heptapod supports Python 3.7 and 3.8 only, with 3.8 being
the recommended version.
Python 2 is not supported any more. Python 3.9 is not supported yet.
143 144 145 146 147

In theory, the minimal Python 3 version should be 3.6.2, but it is completely
untested, whereas developer setups have routinely been running 3.7.

#### Install the Python packages
148 149 150 151 152 153

You can install them system-wide if that suits you (Mercurial has very few
dependencies, it's very unlikely it'll clash with system libraries) or with
`pip install --user` or in a virtualenv. What matters for proper operation is
that they are all importable from the final Mercurial executable.

154 155 156
Note: if you're interested in using Mercurial enhanced with Rust,
you have to preinstall it (see next section about that)

157 158 159 160
Install the Python packages to be taken from pypi.org using the
[requirements file](python/requirements.txt):

```
161
$PIP install -U -r python/requirements.txt
162
```
163

164 165
where `$PIP` is `pip --user` or `some/virtualenv/bin/pip` according to your
choice.
166

167 168 169 170 171 172 173 174 175 176 177 178 179 180 181 182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207 208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225 226 227 228 229 230
#### Variant: Mercurial enhanced with Rust

This provides some serious performances boosts, some of them activated by
boolean flags that can be set, e.g., in the global HGRC file.

The Mercurial source distribution includes the Rust code,
but `pip` won't compile it by default.

Here are the steps do to that:

- prerequisite: Rust toolchain (cargo, rustc), version ≥ 1.41.1 as of
  Mercurial 5.6. The general policy of Mercurial is to depend on the version
  available in Debian stable at the time of the Mercurial release.
- use the requirements file intended for Rust instead of the standard one:

  ```
  $PIP install -r python/requirements-rust.txt
  ```
- check: `/path/to/just/installed/hg debuginstall`. Here's an excerpt of
  the expected output:

  ```
  checking Python security support (sni,tls1.0,tls1.1,tls1.2)
  checking Rust extensions (installed)
  checking Mercurial version (5.6.1)
  checking Mercurial custom build ()
  checking module policy (rust+c)
  ```

  "module policy" is the configuration for the runtime usage of native Python
  modules:

  - `rust+c`: use both Python modules written in C and Rust – fail if they
    are missing. This is the default configuration for Mercurial compiled with
    Rust.
  - `c`: use only Python modules written in C – fail if they are missing.
    This is the default configuration for the default compilation of Mercurial
    (C modules, no Rust).

#### Disabling the Rust parts of Mercurial at run time
The Rust parts of Mercurial are still considered experimental.

In some cases, you may want to disable them. This could be for bug
investigation or simply to use a single build script for
installations with different policies.

The resulting Mercurial runs the Rust code by default, but this can be
disabled at runtime.

1. Run the internal Mercurial HTTP service, and in HGitaly with the
   following environment variable:

   ```
   HGMODULEPOLICY=c
   ```

2. In the `mercurial` section of the Heptapod Shell *and* Heptapod Rails
   configuration files, add this:

   ```
   mercurial:
     module_policy: c
   ```

231 232
### Configuration

233 234 235
You need to

1. define Mercurial settings, including some necessary paths
236
2. create Mercurial internal services
237 238 239 240 241
3. configure GitLab components to use the proper Mercurial with those settings
4. take care of few special cases, depending on the Heptapod version

#### Defining Mercurial settings

242 243 244
The `heptapod` Python project, listed in `requirements.txt`,
ships with a configuration file, `required.hgrc`, that performs most
of the required and default Mercurial settings for Heptapod
245 246
(using the `heptapod` extension, permission policy, etc.)

247 248 249 250 251 252 253 254 255 256 257 258 259 260 261
You'll find it where your `$PIP` has installed `heptapod`, which you can
find out like this:

```
export PY_HEPTAPOD=`$PYTHON -c "import heptapod, os; print(os.path.dirname(heptapod.__file__))"`
```

where `$PYTHON` is the python interpreter that's consistent with `$PIP`. So,
for instance if you're using a virtualenv, and `$PIP` is
`/path/to/virtualenv/bin/pip`, then `$PYTHON` is
`/path/to/virtualenv/bin/python`. If you're using
the global pip, then `$PYTHON` is most likely just `python` or `python2`.

You'll need to use `$PY_HEPTAPOD/required.hgrc` and add your local
installation details.
262 263

A simple way is to create a `heptapod.hgrc` file using the `%include` directive.
264
See the example in `$PY_HEPTAPOD/heptapod.example.hgrc`.
265

266 267 268 269 270 271 272 273 274 275 276 277 278 279 280 281 282 283 284 285 286 287 288 289 290 291
##### Heptapod >= 0.13: configuring Mercurial to notify the Rails application

Starting with Heptapod 0.13, Mercurial will send notifications about incoming
changesets directly.

For that to work, you need tell it where to find the `.gitlab_shell_secret`
file and configure the internal API address. This is done in the `[heptapod]`
section of your `heptapod.hgrc`:

```
[heptapod]
gitlab-internal-api-secret-file=/path/to/.gitlab_shell_secret
gitlab-internal-api-url=http+unix://%2Fpath%2Fto%2Fgitlab.socket
```

Notes:

* the default internal API URL is `http://localhost:8080`.
* the older `heptapod.gitlab-shell` setting is still supported to find
  the `.gitlab_shell_secret` file

If you're upgrading from Heptapod 0.12 *and* the Rails application is binding
to `localhost:8080` (the default with unicorn) then the configuration should
be readily compatible.


292
### Internal Mercurial HTTP service
293

294 295
HTTP access to Mercurial repository content is provided by an internal HTTP
service that sits behind the Heptapod Workhorse reverse proxy.
296 297

Warning: do NOT have it listen to any other interface than the loopback:
298
its security model is to trust Heptapod Workhorse blindly.
299 300 301 302 303 304 305 306 307 308 309 310

It also needs to load your hgrc file(s). Here's an example, assuming `gunicorn`
is on the path.

```
  HEPTAPOD_HGRC=/path/to/your/heptapod.hgrc gunicorn -w 5 \
      -b 127.0.0.1:8000 \
      --error-logfile /tmp/gunicorn.log \
      --capture-output \
      heptapod.wsgi:hgserve
```

311 312
Again, this `gunicorn` must be able to load the Heptapod Python packages, so
you should use the one that's been installed with `requirements.txt`.
313 314
If you're using the dual virtualenv setup for Python 3 with fallback on
Python 2, then use the symbolic link in the path to gunicorn.
315 316 317

Notes:

318
- Heptapod is not able yet to use Unix domain sockets in this service
319 320 321
- If you want to use another loopback URL, you must also pass it to Heptapod
  Workhorse with the `-hgBackend` option (whose default is
  ``http://localhost:8000``)
322

323 324 325 326 327 328 329 330 331 332 333 334 335 336 337

### new in Heptapod 0.17: HGitaly service

Mercurial repository content is exposed to various GitLab components by the
HGitaly service. Eventually, it will become the only way GitLab can access it,
except when proxying HTTP user requests.

In Heptapod 0.17, it is providing the backend for archive downloads of all
Mercurial projects and powers the (very experimental) Mercurial native
projects.

Again, it needs to know about your HGRC. Here's an example, assuming the `hg`
executable to be on `$PATH`:

```
338
  HGRCPATH=/path/to/your/heptapod.hgrc hg \
339 340 341 342 343 344 345 346 347 348
      --config extensions.hgitaly= \
      hgitaly-serve \
      --listen unix:///path/to/hgitaly.socket
```

Startup and shutdown messages will go to stdout and stderr, as well as
potentially rare cases of uncatched exception (a bug in itself).

Actual logs are emitted as configured in the HGRC file.

349 350 351 352 353 354 355 356 357 358 359 360 361 362 363 364 365 366
Here is a sample systemd service unit file, just fix the paths and/or the
system user and it should do the job.

```
[Unit]
Description=HGitaly, internal Heptapod service for Mercurial handling
After=network.target

[Install]
WantedBy=multi-user.target

[Service]
User=heptapod
Group=heptapod
Environment=HGRCPATH=/path/to/your/heptapod.hgrc
ExecStart=/path/to/virtualenv/bin/hg --config extensions.hgitaly= hgitaly-serve --listen unix:///path/to/hgitaly.socket
Restart=on-failure
```
367

368 369 370 371
#### Using Mercurial in GitLab

Edit the configuration files for the Rails application
(see `config/gitlab.yml.example`) and GitLab Shell (see `config.yml.example`),
372 373 374 375 376
to provide

- the full path to your `heptapod.hgrc` file and to the `hg` executable in
  the `mercurial` section.
- the path to the HGitaly socket in the `repositories` section
377

378
Notes:
379

380 381
 - in those YaML files, `hgrc` must always be a list. This is for separation
   in several files with different persistence properties.
382

383 384 385 386 387 388 389 390 391 392 393 394 395 396 397 398 399 400
### Validation with the functional tests

- start from an empty set of data (do not even set the instance root password)
- install tox (Python3)
- grab the functional tests:
  `hg clone https://dev.heptapod.net/heptapod/heptapod-tests`
- run the tests as the same user as the GitLab components, with something
  like:

  ```
  tox -- --heptapod-ssh-port 2222 \
           --heptapod-url http://localhost:3000 \
           --heptapod-source-install \
           --heptapod-repositories-root ~/heptapod/gdk/gl10-5/repositories
  ```

  (for a quicker first test, you can add `-k push_basic`)

401 402 403 404 405 406 407 408 409 410 411 412 413 414 415 416 417 418 419 420 421 422 423 424 425 426 427 428 429 430 431 432 433
## Migration notes

With GitLab migrations in general, it's not possible to jump between too much
different versions. There are some intermediate pauses to respect.

Here is how that translates for Heptapod

### from Heptapod 0.8 to 0.12

Because Heptapod 0.12 is a jump from GitLab CE 10.5 to 12.2, it is not
possible to upgrade directly, because major GitLab version bump must be done
[one at a time](https://docs.gitlab.com/ce/policy/maintenance.html#upgrade-recommendations).

Instead, the recommended way is to go through a series of Heptapod intermediate
versions that are meant to support the migration only:

- Heptapod 0.9 (GitLab CE 10.8)
- Heptapod 0.10 (GitLab CE 11.0)
- Heptapod 0.11 (Gitlab CE 11.8)

At each step, start the Rails application and wait for the background
migrations to be finished. See GitLab [documentation](https://docs.gitlab.com/ce/update/#checking-for-background-migrations-before-upgrading) for how to check
that.

For source installs, one only needs to update the Rails application for these
intermediate versions, and it's best not even to start the services that handle
connections from clients (Workhorse, SSH).

For Docker installs, we will provide a series of images tailored for the
migration needs.

**WARNING do not use Heptapod 0.9 to 0.11 for anything but data migration.**