Read about our upcoming Code of Conduct on this issue

README.md 17 KB
Newer Older
1
# Heptapod Development Kit (HDK)
Jacob Vosmaer's avatar
Jacob Vosmaer committed
2

3
Configure and manage a [Heptapod](https://heptapod.net) development
Jacob Vosmaer's avatar
Jacob Vosmaer committed
4
environment.
Jacob Vosmaer's avatar
Jacob Vosmaer committed
5

6
7
The Heptapod Development Kit is the fork of the GitLab Development Kit (GDK)
meant for Heptapod development. Most of its features are just provided by the
Georges Racinet's avatar
Georges Racinet committed
8
GDK, and there may be interesting functionalities that aren't supported yet.
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32

## HDK overview

### Getting help

The HDK is intended to be the main tool for Heptapod development, but is
a fairly recent effort in itself, with proper fork and all-inclusive build
published in early July, 2020.

The HDK is still too young to have been extensively tested on fresh systems,
so don't be deterred if you encounter problems, come and talk to us on our
[Mattermost Development channel](https://mattermost.heptapod.net/heptapod/channels/development) and we'll sort them out together.

### Prerequisites

The Heptapod Development Kit has the same prerequisites (Ruby, Golang…) than
the GDK, and some more. Don't miss especially the
[Prepare your computer](doc/prepare.md) page.

Note: `rbenv` is the only Ruby environment manager with which we currently have
experience.

A good way to ensure all GDK prerequisites are fulfilled would be to bootstrap
a regular GDK environment before turning to the HDK variant, but that's by no
Georges Racinet's avatar
Georges Racinet committed
33
means necessary. See [Getting started with GDK](#getting-started).
34
35
36
37

### Additional dependencies

- python >= 3.7
38
39
- python `venv` module (part of Python itself, but sometimes packaged
  apart by Linux distributions)
40
- tox (the one provided by any current Linux distribution should be fine)
41
- mercurial (no need to have the latest and greatest, it will be installed
42
  anyway in the Heptapod Python virtualenvs)
43
44
45

### Heptapod workspaces and share pool

46
47
An Heptapod workspace is a clone of the HDK, configured and initialized to
be of service for a whole development cycle (e.g, Heptapod 0.19) from first
48
experiments to main development series, to stable and finally oldstable status.
49

50
51
52
There is a special case to work on release branches, which should be of interest
for release managers only (see the dedicated section below).

53
54
The HDK does not currently have facilities to switch development cycle, for
various reasons. Notably, this would entail a big database migration, which is
55
56
57
not very convenient to rollback anyway. *But* it has a system to switch series
inside a given development cycle, e.g, from main development series to stable
(see dedicated sections below).
58

59
60
Therefore it is advised to create a new workspace for each new development cycle
and to put them in a common parent directory.
61

62
63
64
65
66
67
68
Here's an example of such a layout:

```
$ tree -L 1 -d heptapod
heptapod
├── 0.19
├── 0.20
69
├── 0.20-release
70
71
72
73
74
75
76
├── downloaded-artifacts
└── hg-share-pool
```

The HDK will automatically share big downloads and changesets for the Heptapod
components across the workspaces in a common directory, so that
bootstrapping a new workspace does not waste much bandwidth.
77
78
79

This is done with the `share` extension that ships with the Mercurial standard
distribution, with a pool named `hg-share-pool` in that same directory (this
80
81
path is not configurable as of this writing). The listing above illustrates
that.
82
83
84
85
86
87
88
89
90
91
92
93
94
95

### Creating a Heptapod workspace

The procedure is somewhat different than with the GDK: we don't run
`gdk init`, because that has the URL of the Git repository for the GDK
hardcoded.

1. Checkout the `heptapod` branch of the HDK:

   ```
   ~/heptapod $ hg clone -b heptapod https://foss.heptapod.net/heptapod/heptapod-development-kit hdk-workspace
   ~/heptapod $ cd hdk-workspace
   ```

96
97
98
99
2. (Optional) specify the Heptapod series you want to start with

   For example, Heptapod 0.19 is the currrent stable. To initialize a new workspace for it,
   one needs to require the stable series explicitely beforehand:
100
101
102
103
104
105
106
107

   ```
   ~/heptapod/hdk-workspace $ echo "heptapod_series: stable" > gdk.yml
   ```

   (check `HEPTAPOD_REVISIONS` in [config.rb](lib/gdk/config.rb) to know all
   currently supported series).

108
109
110
111
112
   Warning: in the case of stable series, it may also be necessary to checkout
   the `heptapod-stable` branch of the HDK. We try and make the `heptapod`
   branch suitable for all series, but still need a separate branch when there
   are lots of structural changes in a development cycle.

113
114
115
116
117
118
3. Register the new workspace to the GDK utilities

   ```
   ~/heptapod/hdk-workspace $ make init
   ```

119
120
121
122
123
124
   For this, you'll need:
   - the Ruby environment manager, e.g., `rbenv`, to
     be in service
   - the [Ruby version](.ruby-version) used by the HDK to be installed:
     if you're using `rbenv`, just run
     `rbenv install --skip-existing` in the workspace before this step.
125

126
127
128
129
130
4. Grab and build everything. This can be very long, depending on how many
   new versions of various components have to be downloaded and built.

   At the end of the process, the GDK cheatsheet is displayed, along with
   the default credentials for the new development instance.
131
132

   ```
133
   ~/heptapod/hdk-workspace $ make install
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
152
153
154
155
156
157
158
159
160
161
162
163
164
165
166
   (...)
   ------------------------------------------------------------
   Setup finished!
   ------------------------------------------------------------

   (in /home/gracinet/heptapod/hdk/hdk-test)


              `                        `
             :s:                      :s:
            `oso`                    `oso.
            +sss+                    +sss+
           :sssss:                  -sssss:
          `ossssso`                `ossssso`
          +sssssss+                +sssssss+
         -ooooooooo-++++++++++++++-ooooooooo-
        `:/+++++++++osssssssssssso+++++++++/:`
        -///+++++++++ssssssssssss+++++++++///-
       .//////+++++++osssssssssso+++++++//////.
       :///////+++++++osssssssso+++++++///////:
        .:///////++++++ssssssss++++++///////:.`
          `-://///+++++osssssso+++++/////:-`
             `-:////++++osssso++++////:-`
                .-:///++osssso++///:-.
                  `.://++osso++//:.`
                     `-:/+oo+/:-`
                        `-++-`


   # GitLab Development Kit

   Usage: gdk <command> [<args>]

167
   (...) long list of commands (...)
168
169

   # Development admin account: root / 5iveL!fe
170
171
   ```

172
173
174
175
176
   You can display the cheat sheet at any time with:

   ```
   gdk help
   ```
177

178
179
5. Start those services that aren't running yet and open the application in
   your web browser
180
181
182

   ```
   ~/heptapod/hdk-workspace $ gdk start
183
   ~/heptapod/hdk-workspace $ firefox http://localhost:3000
184
185
   ```

186
187
188
189
190
### Updating a Heptapod workspace

By contrast with upstream GDK, it's not possible to update a workspace in
place, except in a few special cases, with dedicated tools.

191
#### Switch from default to stable
192
193
194
195
196
197
198
199
200
201

Each time there's a new Heptapod default series, the stable series is bumped
to contain what was previously the default series.

This means that developers that hang on to a HDK workspace predating the
creation of the new default series already almost have a workspace for the
new stable. They still need to perform the small updates of various development
branches to their stable counterparts so that subsequent work is targetting the
stable branches, and to get the latest updates on the stable branches.

Georges Racinet's avatar
Georges Racinet committed
202
There is a small script to perform this, with added sanity checks:
203

Georges Racinet's avatar
Georges Racinet committed
204
```
205
  ./heptapod-switch-new-stable
Georges Racinet's avatar
Georges Racinet committed
206
```
207

208
209
210
211
212
213
214
215
216
217
#### Switch from stable to oldstable

This works in the same way as the switch from default to stable, with an
additional flag:

```
  ./heptapod-switch-new-stable --oldstable
```


218
219
220
221
222
223
224
225
226
227
228
229
230
231
232
233
234
235
236
237
238
239
240
241
242
243
244
245
246
247
248
249
250
251
252
253
254
255
256
257
258
259
260
261
262
263
264
265
266
267
268
269
270
271
272
273
274
275
276
277
278
279
280
281
282
283
284
285
286
287
288
289
290
### Running Heptapod functional tests

The Heptapod
[functional test suite](https://foss.heptapod.net/heptapod/heptapod-tests)
is a major part of our development and release process:

- it is the only thing tying all the pieces together
- not so many Heptapod specifics are covered by RSpec integration tests as
  of this writing

The HDK Makefile creates the `heptapod-tests-run` script, which passes all
the needed port and user options for your workspace.

Note: some tests will issue `gdk` commands, so you need the
Ruby environment manager to be activated.

#### Launching all the functional tests

Just run the script from the top of your workspace:

```
   ~/heptapod/hdk-workspace $ gdk start
   ~/heptapod/hdk-workspace $ ./heptapod-tests-run
```

This is expected to take about half an hour.

#### Partial launches

The functional tests are written with [pytest]() and launched via `tox`,
which takes care of managing the virtualenvs for the tests themselves.

All additional arguments to the script are directly passed to the `pytest`
command. Here's a keyword selection on the fully qualified test name:

```
~/heptapod/hdk-workspace $ ./heptapod-tests-run -k push_basic
++ dirname ./heptapod-tests-run
+ cd ./heptapod-tests
+ tox -- --heptapod-url http://localhost:3001 --heptapod-ssh-port 3222 --heptapod-ssh-user gracinet --heptapod-reverse-call-host localhost --heptapod-gdk --heptapod-gdk-root /home/gracinet/heptapod/hdk-workspace --heptapod-repositories-root /home/gracinet/heptapod/hdk-workspace/repositories -k push_basic
GLOB sdist-make: /home/gracinet/heptapod/hdk-workspace/heptapod-tests/setup.py
py3 inst-nodeps: /home/gracinet/heptapod/hdk-workspace/heptapod-tests/.tox/.tmp/package/1/heptapod-tests-1.0.0.zip
py3 installed: attrs==19.3.0,certifi==2020.6.20,chardet==3.0.4,docker==4.2.2,heptapod-tests==1.0.0,idna==2.10,iniconfig==1.0.1,more-itertools==8.4.0,packaging==20.4,pluggy==0.13.1,py==1.9.0,pyparsing==2.4.7,pytest==6.0.1,pytest-parallel==0.1.0,requests==2.24.0,selenium==3.141.0,six==1.15.0,tblib==1.7.0,toml==0.10.1,urllib3==1.25.10,websocket-client==0.57.0
py3 run-test-pre: PYTHONHASHSEED='4038199019'
py3 run-test: commands[0] | py.test --heptapod-url http://localhost:3001 --heptapod-ssh-port 3222 --heptapod-ssh-user gracinet --heptapod-reverse-call-host localhost --heptapod-gdk --heptapod-gdk-root /home/gracinet/heptapod/hdk-workspace --heptapod-repositories-root /home/gracinet/heptapod/hdk-workspace/repositories -k push_basic
============================= test session starts ==============================
platform linux -- Python 3.8.3, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
cachedir: heptapod-tests/.tox/py3/.pytest_cache
rootdir: /home/gracinet/heptapod/hdk-workspace
plugins: parallel-0.1.0
collected 105 items / 104 deselected / 1 selected


====================== 1 passed, 104 deselected in 20.41s ======================
___________________________________ summary ____________________________________
  py3: commands succeeded
  congratulations :)
```

If you aren't familiar with pytest, you can get a full list of options, including
the Heptapod specific ones by running:

```
~/heptapod/hdk-workspace $ ./heptapod-tests-run --help
```

It is interesting to launch the tests from within the `heptapod-tests` clone, so that
you'll get auto-completion for test files:

```
~/heptapod/hdk-workspace/heptapod-tests $ ../heptapod-tests-run tests/test_merge_requests.py
```

291
292
293
294
295
296
297
298
299
300
301
302
303
304
305
306
307
308
309
310
311
312
313
314
315
316
317
318
319
320
321
322
323
324
325
326
327
328
329
### Working on release branches

Due to the [way we track upstream GitLab](heptapod#127), each minor version
has its own branch,
starting roughly at the first release candidate, then moving on to the first
final release, and then patch versions. The branch is typically closed without
back merge to the development branches at the end of the cycle, just as it
happens with GitLab's stable branches.

In the main repository, release branch names are `heptapod-0-19`,
`heptapod-0-20` etc. At any point in time, we can have up to two active
release branches, each one getting merges from the development branches. For
instance, short after this writing, the latest release branch will be
`heptapod-0-20`, while `heptapod-0-19` will be the stable release branch:
Changesets from the `heptapod` branch will get merged into `heptapod-0-20` for
releases, while those from the `heptapod-stable` branch will get merged into
`heptapod-0.19`. When the `heptapod-0-21` branch will start, `heptapod-0-19`
should stop receiving updates and will be closed not long afterwards.

HDK workspaces can be created for the release branches, by using the
`release` and `release-stable` series (set before running `make`, as usual).

Example, assuming the current development version to be 0.20:

```
~/heptapod/0.20-release $ echo "heptapod_series: release" > gdk.yml
```

Warning: the HDK clone needs to be up to date in order to resolve the target
Heptapod version correctly.

#### Switching a release HDK to stable status

When the stable Heptapod branch is bumped, one needs to:

- update the HDK, e.g., `hg pull -u`
- run `heptapod-switch-new-stable`


330
331
332
333
334
335
336
337
338
339
340
341
342
343
344
345
346
347
348
349
350
351
352
353
354
355
356
357
358
359
360
361
362
### HDK download caches

The HDK comes with a global cache system for downloads that the GitLab
Development Kit does not provide.

Thanks to these, it is reasonably fast to create a new workspace: only
the first workspace installation does perform the long downloads. In fact,
rather than attempting to update a workspace, the recommanded way is to
recreate it.

- Mercurial clones: using a share pool located by default in the
  `hg-share-pool` sibling directory of the current workspace
  (`../hg-share-pool`)
- Artifacts: the installation process retrieves some tarballs. They are stored
  in the `downloaded-artifacts` sibling directory of the current workspace
  and kept indefinitely. You may want to clean the ones you know not to be
  useful any more.

The paths for these download cache directories are fully configurable in
`~/.gdk.yml`. Here's an example:

```
download_caches:
  hg: "/data/hdk/hg-share-pool"
  artifacts: "/data/hdk/artifacts"
trusted_directories:  # managed by `make init` (`gdk trust`)
- /home/user/heptapod/workspace1
- /home/user/heptapod/workspace2
```

This caching system could be considered for upstreaming to the
Gitlab Development Kit.

363
364
365
366
367
368
369
370
371
### Logs

#### Mercurial and HGitaly

The Mercurial log also contains all HGitaly logs, except those related
to the service startup sequence.

It is located by default at `log/mercurial.log`

372

373
## GDK overview
Jacob Vosmaer's avatar
Jacob Vosmaer committed
374

375
376
(this section and the following ones are original `README` content from the
GDK, unchanged)
Jacob Vosmaer's avatar
Jacob Vosmaer committed
377

Craig Norris's avatar
Craig Norris committed
378
379
380
The GitLab Development Kit (GDK) helps you install a GitLab instance on your
workstation. It includes a collection of GitLab requirements, such as Ruby,
Node.js, Go, PostgreSQL, Redis, and more.
Ray Paik's avatar
Ray Paik committed
381

Craig Norris's avatar
Craig Norris committed
382
383
384
The GDK is recommended for anyone contributing to the GitLab codebase, whether a
GitLab team member or a member of the wider community. It allows you to test
your changes locally on your workstation in an isolated manner. This can speed
385
up the time it takes to make successful contributions.
Ray Paik's avatar
Ray Paik committed
386

Ash McKenzie's avatar
Ash McKenzie committed
387
## Goals
388

Ash McKenzie's avatar
Ash McKenzie committed
389
390
391
392
- Provide developer tooling to install, update, and develop against a local GitLab instance.
- Offer GDK users an automated method for installing [required software](https://docs.gitlab.com/ee/install/requirements.html#software-requirements).
- Out of the box, only enable the services GitLab requires to operate.
- Support native operating systems as listed below.
393

394
## Installation
395

396
You can install GDK using the following methods. Some are:
397

398
399
- Supported and frequently tested.
- Not supported, but we welcome merge requests to improve them.
Jacob Vosmaer's avatar
Jacob Vosmaer committed
400

401
### Supported methods
402

403
The following installation methods are supported, actively maintained, and tested:
Jacob Vosmaer's avatar
Jacob Vosmaer committed
404

405
- [One-line installation](doc/index.md#one-line-installation):
406

407
  ```shell
408
  curl "https://gitlab.com/gitlab-org/gitlab-development-kit/-/raw/main/support/install" | bash
409
  ```
410

411
- [Simple installation](doc/index.md) on your local system. Requires at least
412
  8GB RAM and 12GB disk space. Available for [supported platforms](#supported-platforms).
413

414
- [Gitpod](doc/howto/gitpod.md).
Grzegorz Bizon's avatar
Grzegorz Bizon committed
415

416
417
418
419
420
421
422
423
424
425
426
427
428
429
430
431
432
433
### Supported platforms

| Operating system | Versions            |
|:-----------------|:--------------------|
| macOS            | 11, 10.15, 10.14    |
| Ubuntu           | 20.10, 20.04, 18.04 |
| Debian           | 10, 9               |
| Arch             | latest              |
| Manjaro          | latest              |

#### macOS with Apple silicon

Running GDK on macOS with [Apple silicon](https://en.wikipedia.org/wiki/Apple_silicon)
requires [Rosetta 2](https://en.wikipedia.org/wiki/Rosetta_(software)#Rosetta_2).

Follow [Native Apple silicon support](https://gitlab.com/gitlab-org/gitlab-development-kit/-/issues/1159)
for updates on supporting Apple silicon natively.

434
### Unsupported methods
Grzegorz Bizon's avatar
Grzegorz Bizon committed
435

436
437
The following documentation is provided for those who can benefit from it, but aren't
supported installation methods:
438

439
440
441
442
- [Advanced installation](doc/advanced.md) on your local system. Requires at least
  8GB RAM and 12GB disk space.
- [Vagrant](doc/howto/vagrant.md).
- [Minikube](doc/howto/kubernetes/minikube.md).
Ash McKenzie's avatar
Ash McKenzie committed
443

444
## Post-installation
Jacob Vosmaer's avatar
Jacob Vosmaer committed
445

446
447
- [Use GDK](doc/howto/index.md).
- [Update an existing installation](doc/index.md#update-gdk).
Jacob Vosmaer's avatar
Jacob Vosmaer committed
448

Ash McKenzie's avatar
Ash McKenzie committed
449
## Getting help
Jacob Vosmaer's avatar
Jacob Vosmaer committed
450

Ash McKenzie's avatar
Ash McKenzie committed
451
452
453
454
- We encourage you to [create a new issue](https://gitlab.com/gitlab-org/gitlab-development-kit/-/issues/new).
- GitLab team members can use the `#gdk` channel on the GitLab Slack workspace.
- Wider community members can use the [Gitter contributors room](https://gitter.im/gitlab/contributors)
  or [GitLab Forum](https://forum.gitlab.com/c/community/community-contributions/15).
455

Ash McKenzie's avatar
Ash McKenzie committed
456
## Contributing to GitLab Development Kit
457

Ash McKenzie's avatar
Ash McKenzie committed
458
459
Contributions are welcome; see [`CONTRIBUTING.md`](CONTRIBUTING.md)
for more details.
460

Jacob Vosmaer's avatar
Jacob Vosmaer committed
461
462
## License

Craig Norris's avatar
Craig Norris committed
463
464
The GitLab Development Kit is distributed under the MIT license; see the
[LICENSE](LICENSE) file.