Startup failure for Nginx caches at OSUOSL
Conclusion: need more robustness in Nginx service startup
Today, both Nginx servers at OSUOSL that provide caching for the S3 behind registry.heptapod.net were down, resulting in new Docker images unavailable for OSUOSL runners.
There was a planned outage yesterday, and they didn't start properly at boot:
Mar 27 13:31:32 heptapod-ci-01.bak.milne.osuosl.org systemd: Starting A high performance web server and a reverse proxy server... Mar 27 13:31:34 heptapod-ci-01.bak.milne.osuosl.org nginx: nginx: [emerg] bind() to 10.x.y,z:443 failed (99: Cannot assign requested address)
Of course, the manual restart I performed a few minutes ago worked without a fuss, and a new container image got downloaded successfully.
We should add a mixin unit for automatic retry (with a substantial delay) and/or wait on the interface, if possible.
If this happens again: The Docker daemon message is a bit misleading in that it complains about the last one it tried only:
error pulling image configuration: Get https://cellar-c2.services.clever-cloud.com/heptapod-foss-container-registry/docker/registry/v2/blobs/sha256/56/56d66d0b0cb495442532a429d6841e2d15471ca0c042f328bc7afbfab0c123c3/data?X-Amz-Algorithm=AWS4-HMAC-SHA256&X-Amz-Credential=REDACTEDdummy%2Fs3%2Faws4_request&X-Amz-Date=20210328T172019Z&X-Amz-Expires=1200&X-Amz-SignedHeaders=host&X-Amz-Signature=9fbabe8dbec1186495e2c581a161e2d77775bf742870a15a522589fcc55a1b10: dial tcp 10.x.y.z:443: connect: connection refused
I restarted the one it complained about, retried the failing job that was reported to me, then looked at hints for DNS round-robin issues while pulling images… and only then thought of checking the other one (was also down). So it turns out I was actually unintentionally testing the fallback mode: it works.