OSUOSL: experiment without the Cellar cache
When we started using our own Container Registry, registry.heptapod.net
, we quickly had to work around bandwidth issues between Clever Cloud services (in this case the S3-compatible Cellar, which actually holds container layers) and OSUOSL. It was by then around 2-3 MB/s.
At the time, the solution was to introduce a caching layer at OSUOSL. In theory, the configuration should not have tampered with uploads, but in practice it does, which makes for long debugging sessions that we cannot really afford. The consequence was the introduction of the container-registry-push
tag, to avoid building images on runners that don't have the capability to upload them.
After some changes in our swarm of Runners, we only have two instance runners available for container builds: osci-01-docker-x86_64
and clever-cloud
. The OSCI runner is currently in trouble and the Clever runner cannot be used in some cases. This is bad.
I've just took some fresh bandwidth measurements and got 17 MB/s to get a 1 GB file from Cellar to OSUOSL. This looks acceptable enough to at least experiment removing this caching layer from OSUOSL runners. In turn, this should make them also able to use shared CI caches (less efficiently than the clever-cloud
runner, obviously).
Task list
-
Detach osuosl-01
andosuosl-02
from the Cellar cache, tag themcontainer-registry-push
-
Test Docker image production on instance runners -
Configure S3 shared CI caches on Omnibus Heptapod dedicated runners ( osuosl-openstack-01-omnibus-heptapod
andosuosl-openstack-02-omnibus-heptapod
, same systems actually, with different configurations) -
Confirm proper cache behaviour on the Omnibus runners with the upcoming builds for Heptapod 0.36.3 -
Generalize