INSTALL_HEPTAPOD.md 9.69 KB
Newer Older
1 2
# Heptapod installation guide

3
## Important: migrating from Heptapod 0.8 to 0.12
4 5 6 7 8 9 10 11 12 13 14 15 16 17 18 19 20 21 22 23 24 25 26 27 28 29

Because Heptapod 0.12 is a jump from GitLab CE 10.5 to 12.2, it is not
possible to upgrade directly, because major GitLab version bump must be done
[one at a time](https://docs.gitlab.com/ce/policy/maintenance.html#upgrade-recommendations).

Instead, the recommended way is to go through a series of Heptapod intermediate
versions that are meant to support the migration only:

- Heptapod 0.9 (GitLab CE 10.8)
- Heptapod 0.10 (GitLab CE 11.0)
- Heptapod 0.11 (Gitlab CE 11.8)

At each step, start the Rails application and wait for the background
migrations to be finished. See GitLab [documentation](https://docs.gitlab.com/ce/update/#checking-for-background-migrations-before-upgrading) for how to check
that.

For source installs, one only needs to update the Rails application for these
intermediate versions, and it's best not even to start the services that handle
connections from clients (Workhorse, SSH).

For Docker installs, we will provide a series of images tailored for the
migration needs.

**WARNING do not use Heptapod 0.9 to 0.11 for anything but data migration.**


30 31 32 33 34 35 36 37 38 39 40 41 42 43 44 45 46 47 48 49 50 51 52 53 54
## Installing from source

Heptapod installation is similar to GitLab's except that

- some components have to be taken from our Mercurial repositories
- we have an additional component for Mercurial.
- some specific configuration has to be set in place.
- we have more constraints, many of which will be hopefully lifted in
  forthcoming versions

### Limitations

the Rails application, GitLab Shell, Mercurial and the repositories must
be in a single system.

### Fetching the standard GitLab components

We try and maintain a consistent tagging policy across the various Mercurial
repositories. For example, in version 0.8.0rc1, all of them have a
`heptapod-0.8.0rc1` tag.

Please use the mirror URL for initial and repetitive clonings.

- Rails application (replacement for `gitlab`): https://mirror.octobus.net/heptapod/heptapod
- Heptapod Shell (replacement for GitLab Shell): https://mirror.octobus.net/heptapod/heptapod-shell
55
- Optional in Heptapod < 0.13, Heptapod Workhorse (replacement for GitLab Workhorse): https://mirror.octobus.net/heptapod/heptapod-workhorse
56 57 58 59 60 61 62 63 64

Example:

```
  hg clone https://mirror.octobus.net/heptapod/heptapod
  cd heptapod
  hg update heptapod-0.8.0rc1
```

65
All other standard GitLab components (Gitaly, etc) have to be taken
66
from the canonical upstream locations, at the version specified at the root
67
of the Rails application sources (`GITALY_SERVER_VERSION` etc).
68

69 70 71 72 73 74 75 76 77 78 79 80 81 82 83 84 85 86 87
#### Heptapod Shell: activating the Golang version of Mercurial SSH support

This is for Heptapod 0.13 from 0.13.1 onwards. The Go programming language
version will become the only one in Heptapod 0.14, same as in in GitLab ≥ 12.4.

In the `config.yml` file of GitLab / Heptapod Shell, put this:

```
# Migration to Go: anything listed here has two implementations. Use these flags
# to try the new implementations out, or to revert to the old behaviour if there
# problems arise.
migration:
  enabled: true
  features: ["hg"]
```

The `config.yml.example` file in the sources was also updated to include
this new `hg` feature among the possible values.

88
### Installing Mercurial components
89 90 91 92

We need a specific Mercurial version, a WSGI server (usually Gunicorn), and
some additional Python packages from source.

93 94 95 96 97 98 99 100 101 102 103 104 105 106 107 108 109 110 111 112 113 114 115 116 117 118 119 120 121 122 123 124 125 126 127 128 129 130 131
#### Python versions

Before version 0.13.0, Heptapod runs with Python 2.7 only.

Heptapod 0.13.0 supports Python 2.7 and Python ≥ 3.7, with Python 3 still
considered a technology preview.

Python 3 will become default or even mandatory with Heptapod 0.14.

On Python 3, the preferred version should be 3.8.

In theory, the minimal Python 3 version should be 3.6.2, but it is completely
untested, whereas developer setups have routinely been running 3.7.
For the record the Docker images currently ship with 3.8.3.

#### Be ready to fallback to Python 2

We encourage you to run Heptapod with Python 3, but you must be aware that
this can cause some friction, especially in interactions with external
systems. Therefore you should **prepare the means to switch easily between
Python 2 and Python 3**. This way, if you encounter problems,  you'll be able to qualify them (thank you in advance for the resulting bug report)
and you'll have a workaround.

An easy way to do that is to prepare a virtualenv for each Python version,
and point a symbolic link to the one to actually use.

Assuming you'll reference only that link in the
various configuration files (see below), to switch Python versions, you'll only
need to

1. change the link target
2. restart `gunicorn` (internal Mercurial service).
   Almost no perceptible interruption.
3. restart the Rails application so that the Python version displayed in the
   help page and admin dashboard is the right one. This is a noticeable
   interruption of service, but you can delay it if that's a problem.


#### Install the Python packages
132 133 134 135 136 137

You can install them system-wide if that suits you (Mercurial has very few
dependencies, it's very unlikely it'll clash with system libraries) or with
`pip install --user` or in a virtualenv. What matters for proper operation is
that they are all importable from the final Mercurial executable.

138 139 140 141
Install the Python packages to be taken from pypi.org using the
[requirements file](python/requirements.txt):

```
142
$PIP install -U -r python/requirements.txt
143
```
144

145
where `$PIP` is `pip --user` or `some/virtualenv/bin/pip` according to your choice (see above how to prepare for Python 3 with easy fallback onto Python 2).
146 147 148

### Configuration

149 150 151 152 153 154 155 156 157
You need to

1. define Mercurial settings, including some necessary paths
2. create an internal service for Mercurial
3. configure GitLab components to use the proper Mercurial with those settings
4. take care of few special cases, depending on the Heptapod version

#### Defining Mercurial settings

158 159 160
The `heptapod` Python project, listed in `requirements.txt`,
ships with a configuration file, `required.hgrc`, that performs most
of the required and default Mercurial settings for Heptapod
161 162
(using the `heptapod` extension, permission policy, etc.)

163 164 165 166 167 168 169 170 171 172 173 174 175 176 177
You'll find it where your `$PIP` has installed `heptapod`, which you can
find out like this:

```
export PY_HEPTAPOD=`$PYTHON -c "import heptapod, os; print(os.path.dirname(heptapod.__file__))"`
```

where `$PYTHON` is the python interpreter that's consistent with `$PIP`. So,
for instance if you're using a virtualenv, and `$PIP` is
`/path/to/virtualenv/bin/pip`, then `$PYTHON` is
`/path/to/virtualenv/bin/python`. If you're using
the global pip, then `$PYTHON` is most likely just `python` or `python2`.

You'll need to use `$PY_HEPTAPOD/required.hgrc` and add your local
installation details.
178 179

A simple way is to create a `heptapod.hgrc` file using the `%include` directive.
180
See the example in `$PY_HEPTAPOD/heptapod.example.hgrc`.
181

182 183 184 185 186 187 188 189 190 191 192 193 194 195 196 197 198 199 200 201 202 203 204 205 206 207
##### Heptapod >= 0.13: configuring Mercurial to notify the Rails application

Starting with Heptapod 0.13, Mercurial will send notifications about incoming
changesets directly.

For that to work, you need tell it where to find the `.gitlab_shell_secret`
file and configure the internal API address. This is done in the `[heptapod]`
section of your `heptapod.hgrc`:

```
[heptapod]
gitlab-internal-api-secret-file=/path/to/.gitlab_shell_secret
gitlab-internal-api-url=http+unix://%2Fpath%2Fto%2Fgitlab.socket
```

Notes:

* the default internal API URL is `http://localhost:8080`.
* the older `heptapod.gitlab-shell` setting is still supported to find
  the `.gitlab_shell_secret` file

If you're upgrading from Heptapod 0.12 *and* the Rails application is binding
to `localhost:8080` (the default with unicorn) then the configuration should
be readily compatible.


208 209 210 211 212 213 214 215 216 217 218 219 220 221 222 223 224 225
### Internal Mercurial service

Repository content is exposed to GitLab through an internal HTTP service.

Warning: do NOT have it listen to any other interface than the loopback:
its security model is to trust the Rails application blindly.

It also needs to load your hgrc file(s). Here's an example, assuming `gunicorn`
is on the path.

```
  HEPTAPOD_HGRC=/path/to/your/heptapod.hgrc gunicorn -w 5 \
      -b 127.0.0.1:8000 \
      --error-logfile /tmp/gunicorn.log \
      --capture-output \
      heptapod.wsgi:hgserve
```

226 227
Again, this `gunicorn` must be able to load the Heptapod Python packages, so
you should use the one that's been installed with `requirements.txt`.
228 229
If you're using the dual virtualenv setup for Python 3 with fallback on
Python 2, then use the symbolic link in the path to gunicorn.
230 231 232

Notes:

233
- Heptapod is not able yet to use Unix domain sockets in this service
234 235 236
- If you want to use another loopback URL, configure the Rails app
  accordingly, in `config/gitlab.yml`.

237 238 239 240 241 242 243
#### Using Mercurial in GitLab

Edit the configuration files for the Rails application
(see `config/gitlab.yml.example`) and GitLab Shell (see `config.yml.example`),
to provide the full path to your `heptapod.hgrc` file, and the `hg` executable
in the `mercurial` section.

244 245 246 247 248 249
Notes:

 - in those YaML files, `hgrc` must always be a list. This is for separation
   in several files with different persistence properties.
 - if you're using the double virtualenv approach to testing Python 3, remember
   to use the symbolic link for the path to the `hg` executable.
250

251 252 253 254 255 256 257 258 259 260 261 262 263 264 265 266 267 268
### Validation with the functional tests

- start from an empty set of data (do not even set the instance root password)
- install tox (Python3)
- grab the functional tests:
  `hg clone https://dev.heptapod.net/heptapod/heptapod-tests`
- run the tests as the same user as the GitLab components, with something
  like:

  ```
  tox -- --heptapod-ssh-port 2222 \
           --heptapod-url http://localhost:3000 \
           --heptapod-source-install \
           --heptapod-repositories-root ~/heptapod/gdk/gl10-5/repositories
  ```

  (for a quicker first test, you can add `-k push_basic`)