Read about our upcoming Code of Conduct on this issue

Commit 7a9511f8 authored by Georges Racinet's avatar Georges Racinet 🦑
Browse files

heptapod#582: making Heptapod 0.27 the stable series

Pipeline #29609 passed with stage
in 3 minutes and 25 seconds
......@@ -52,20 +52,6 @@
script:
- make verify test
go:1.14:
extends: .test
image: golang:1.14
after_script:
- make coverage
coverage: '/\d+.\d+%/'
go:1.15:
extends: .test
image: golang:1.15
after_script:
- make coverage
coverage: '/\d+.\d+%/'
go:1.16:
extends: .test
image: golang:1.16
......@@ -73,5 +59,12 @@
- make coverage
coverage: '/\d+.\d+%/'
go:1.17:
extends: .test
image: golang:1.17
after_script:
- make coverage
coverage: '/\d+.\d+%/'
race:
extends: .test
......@@ -76,6 +69,6 @@
race:
extends: .test
image: golang:1.16
image: golang:1.17
script:
- make test_golang_race
......
......@@ -35,3 +35,4 @@
aefbb5e3eac5411fc526a88d3df1762ecbdc2e76 heptapod-13.19.0
b84ce63181d5c0d07585be867dbe957eb6d35348 heptapod-13.19.1
f593b81abfe52ca2083440f3f470dce3d13079da heptapod-13.19.2
b15301c706769652286af112a4437dc8cdfac86e heptapod-13.21.0
v13.21.1
- Only validate SSL cert file exists if a value is supplied !527
v13.21.0
- Change default logging format to JSON !476
- Shutdown sshd gracefully !484
- Provide liveness and readiness probes !494
- Add tracing instrumentation to http client !495
- Log same correlation_id on auth keys check of ssh connections !501
- fix: validate client cert paths exist on disk before proceeding !508
- Modify regex to prevent partial matches
v13.20.0
- Remove bin/authorized_keys !491
- Add a make install command !490
- Create PROCESS.md page with Security release process !488
- Fix the Geo SSH push proxy hanging !487
- Standardize logging timestamp format !485
v13.19.1
- Modify regex to prevent partial matches
......
......@@ -9,6 +9,10 @@
The corresponding tag is heptapod-x.y.z
## 13.21.0
- Jump to upstream GitLab Shell v13.21.1 (GitLab 14.4 series)
## 13.19.2
- Added support for the new `NO_GIT` flag, telling Mercurial not to
......
.PHONY: validate verify verify_ruby verify_golang test test_ruby test_golang coverage coverage_golang setup _install build compile check clean
.PHONY: validate verify verify_ruby verify_golang test test_ruby test_golang coverage coverage_golang setup _script_install build compile check clean install
GO_SOURCES := $(shell find . -name '*.go')
VERSION_STRING := $(shell ./compute_version || echo unknown)
......@@ -6,8 +6,10 @@
BUILD_TAGS := tracer_static tracer_static_jaeger continuous_profiler_stackdriver
GOBUILD_FLAGS := -ldflags "-X main.Version=$(VERSION_STRING) -X main.BuildTime=$(BUILD_TIME)" -tags "$(BUILD_TAGS)"
PREFIX ?= /usr/local
validate: verify test
verify: verify_golang
verify_golang:
......@@ -9,9 +11,12 @@
validate: verify test
verify: verify_golang
verify_golang:
gofmt -s -l $(GO_SOURCES)
gofmt -s -l $(GO_SOURCES) | awk '{ print } END { if (NR > 0) { print "Please run make fmt"; exit 1 } }'
fmt:
gofmt -w -s $(GO_SOURCES)
test: test_ruby test_golang
......@@ -30,5 +35,5 @@
coverage_golang:
[ -f cover.out ] && go tool cover -func cover.out
setup: _install bin/gitlab-shell
setup: _script_install bin/gitlab-shell
......@@ -34,5 +39,5 @@
_install:
_script_install:
bin/install
build: bin/gitlab-shell
......@@ -45,3 +50,12 @@
clean:
rm -f bin/check bin/gitlab-shell bin/gitlab-shell-authorized-keys-check bin/gitlab-shell-authorized-principals-check bin/gitlab-sshd
install: compile
mkdir -p $(DESTDIR)$(PREFIX)/bin/
install -m755 bin/check $(DESTDIR)$(PREFIX)/bin/check
install -m755 bin/gitlab-shell $(DESTDIR)$(PREFIX)/bin/gitlab-shell
install -m755 bin/gitlab-shell $(DESTDIR)$(PREFIX)/bin/gitlab-shell-authorized-keys-check
install -m755 bin/gitlab-shell $(DESTDIR)$(PREFIX)/bin/gitlab-shell-authorized-principals-check
install -m755 bin/gitlab-shell $(DESTDIR)$(PREFIX)/bin/gitlab-sshd
## Releasing a new version
GitLab Shell is versioned by git tags, and the version used by the Rails
application is stored in
[`GITLAB_SHELL_VERSION`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/GITLAB_SHELL_VERSION).
For each version, there is a raw version and a tag version:
- The **raw version** is the version number. For instance, `15.2.8`.
- The **tag version** is the raw version prefixed with `v`. For instance, `v15.2.8`.
To release a new version of GitLab Shell and have that version available to the
Rails application:
1. Create a merge request to update the [`CHANGELOG`](CHANGELOG) with the
**tag version** and the [`VERSION`](VERSION) file with the **raw version**.
2. Ask a maintainer to review and merge the merge request. If you're already a
maintainer, second maintainer review is not required.
3. Add a new git tag with the **tag version**.
4. Update `GITLAB_SHELL_VERSION` in the Rails application to the **raw
version**. (Note: this can be done as a separate MR to that, or in and MR
that will make use of the latest GitLab Shell changes.)
## Security releases
GitLab Shell is included in the packages we create for GitLab, and each version of GitLab specifies the version of GitLab Shell it uses in the `GITLAB_SHELL_VERSION` file, so security fixes in GitLab Shell are tightly coupled to the [GitLab security release](https://about.gitlab.com/handbook/engineering/workflow/#security-issues) workflow.
For a security fix in GitLab Shell, two sets of merge requests are required:
* The fix itself, in the `gitlab-org/security/gitlab-shell` repository and its backports to the previous versions of GitLab Shell
* Merge requests to change the versions of GitLab Shell included in the GitLab security release, in the `gitlab-org/security/gitlab` repository
The first step could be to create a merge request with a fix targeting `main` in `gitlab-org/security/gitlab-shell`. When the merge request is approved by maintainers, backports targeting previous 3 versions of GitLab Shell must be created. The stable branches for those versions may not exist, feel free to ask a maintainer to create ones. The stable branches must be created out of the GitLab Shell tags/version used by the 3 previous GitLab releases. In order to find out the GitLab Shell version that is used on a particular GitLab stable release, the following steps may be helpful:
```shell
git fetch security 13-9-stable-ee
git show refs/remotes/security/13-9-stable-ee:GITLAB_SHELL_VERSION
```
These steps display the version that is used by `13.9` version of GitLab.
Close to the GitLab security release, a maintainer should merge the fix and backports and cut all the necessary GitLab Shell versions. This allows bumping the `GITLAB_SHELL_VERSION` for `gitlab-org/security/gitlab`. The GitLab merge request will be handled by the general GitLab security release process.
Once the security release is done, a GitLab Shell maintainer is responsible for syncing tags and `main` to the `gitlab-org/gitlab-shell` repository.
......@@ -11,7 +11,7 @@
When you access the GitLab server over SSH then GitLab Shell will:
1. Limits you to predefined git commands (git push, git pull).
1. Limit you to predefined git commands (git push, git pull).
1. Call the GitLab Rails API to check if you are authorized, and what Gitaly server your repository is on
1. Copy data back and forth between the SSH client and the Gitaly server
......@@ -38,13 +38,9 @@
We follow the [Golang Release Policy](https://golang.org/doc/devel/release.html#policy)
of supporting the current stable version and the previous two major versions.
## Setup
make setup
## Check
Checks if GitLab API access and redis via internal API can be reached:
make check
......@@ -45,9 +41,33 @@
## Check
Checks if GitLab API access and redis via internal API can be reached:
make check
## Compile
Builds the `gitlab-shell` binaries, placing them into `bin/`.
make compile
## Install
Builds the `gitlab-shell` binaries and installs them onto the filesystem. The
default location is `/usr/local`, but can be controlled by use of the `PREFIX`
and `DESTDIR` environment variables.
make install
## Setup
This command is intended for use when installing GitLab from source on a single
machine. In addition to compiling the gitlab-shell binaries, it ensures that
various paths on the filesystem exist with the correct permissions. Do not run
it unless instructed to by your installation method documentation.
make setup
## Testing
Run tests:
......@@ -82,14 +102,5 @@
Starting with GitLab 8.12, GitLab supports Git LFS authentication through SSH.
## Releasing a new version
GitLab Shell is versioned by git tags, and the version used by the Rails
application is stored in
[`GITLAB_SHELL_VERSION`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/GITLAB_SHELL_VERSION).
For each version, there is a raw version and a tag version:
- The **raw version** is the version number. For instance, `15.2.8`.
- The **tag version** is the raw version prefixed with `v`. For instance, `v15.2.8`.
## Releasing
......@@ -95,15 +106,5 @@
To release a new version of GitLab Shell and have that version available to the
Rails application:
1. Create a merge request to update the [`CHANGELOG`](CHANGELOG) with the
**tag version** and the [`VERSION`](VERSION) file with the **raw version**.
2. Ask a maintainer to review and merge the merge request. If you're already a
maintainer, second maintainer review is not required.
3. Add a new git tag with the **tag version**.
4. Update `GITLAB_SHELL_VERSION` in the Rails application to the **raw
version**. (Note: this can be done as a separate MR to that, or in and MR
that will make use of the latest GitLab Shell changes.)
See [PROCESS.md](./PROCESS.md)
## Contributing
......
#!/bin/sh
# Legacy script used for AuthorizedKeysCommand when configured without username.
# Executes gitlab-shell-authorized-keys-check with "git" as expected and actual
# username and with the passed key.
#
# TODO: Remove this in https://gitlab.com/gitlab-org/gitlab-shell/issues/209.
$(dirname $0)/gitlab-shell-authorized-keys-check git git $1
......@@ -11,5 +11,4 @@
"strings"
"testing"
"github.com/sirupsen/logrus"
"github.com/stretchr/testify/require"
......@@ -15,6 +14,7 @@
"github.com/stretchr/testify/require"
"gitlab.com/gitlab-org/gitlab-shell/client/testserver"
"gitlab.com/gitlab-org/gitlab-shell/internal/testhelper"
)
func TestClients(t *testing.T) {
......@@ -16,11 +16,9 @@
"gitlab.com/gitlab-org/gitlab-shell/client/testserver"
"gitlab.com/gitlab-org/gitlab-shell/internal/testhelper"
)
func TestClients(t *testing.T) {
testDirCleanup, err := testhelper.PrepareTestRootDir()
require.NoError(t, err)
defer testDirCleanup()
testhelper.PrepareTestRootDir(t)
testCases := []struct {
desc string
......@@ -61,7 +59,8 @@
secret := "sssh, it's a secret"
httpClient := NewHTTPClient(url, tc.relativeURLRoot, tc.caFile, "", false, 1)
httpClient, err := NewHTTPClientWithOpts(url, tc.relativeURLRoot, tc.caFile, "", false, 1, nil)
require.NoError(t, err)
client, err := NewGitlabNetClient("", "", secret, httpClient)
require.NoError(t, err)
......@@ -78,7 +77,6 @@
func testSuccessfulGet(t *testing.T, client *GitlabNetClient) {
t.Run("Successful get", func(t *testing.T) {
hook := testhelper.SetupLogger()
response, err := client.Get(context.Background(), "/hello")
require.NoError(t, err)
require.NotNil(t, response)
......@@ -88,18 +86,8 @@
responseBody, err := ioutil.ReadAll(response.Body)
require.NoError(t, err)
require.Equal(t, string(responseBody), "Hello")
require.True(t, testhelper.WaitForLogEvent(hook))
entries := hook.AllEntries()
require.Equal(t, 1, len(entries))
require.Equal(t, logrus.InfoLevel, entries[0].Level)
require.Contains(t, entries[0].Message, "method=GET")
require.Contains(t, entries[0].Message, "status=200")
require.Contains(t, entries[0].Message, "content_length_bytes=")
require.Contains(t, entries[0].Message, "Finished HTTP request")
require.Contains(t, entries[0].Message, "correlation_id=")
})
}
func testSuccessfulPost(t *testing.T, client *GitlabNetClient) {
t.Run("Successful Post", func(t *testing.T) {
......@@ -101,9 +89,8 @@
})
}
func testSuccessfulPost(t *testing.T, client *GitlabNetClient) {
t.Run("Successful Post", func(t *testing.T) {
hook := testhelper.SetupLogger()
data := map[string]string{"key": "value"}
response, err := client.Post(context.Background(), "/post_endpoint", data)
......@@ -115,18 +102,8 @@
responseBody, err := ioutil.ReadAll(response.Body)
require.NoError(t, err)
require.Equal(t, "Echo: {\"key\":\"value\"}", string(responseBody))
require.True(t, testhelper.WaitForLogEvent(hook))
entries := hook.AllEntries()
require.Equal(t, 1, len(entries))
require.Equal(t, logrus.InfoLevel, entries[0].Level)
require.Contains(t, entries[0].Message, "method=POST")
require.Contains(t, entries[0].Message, "status=200")
require.Contains(t, entries[0].Message, "content_length_bytes=")
require.Contains(t, entries[0].Message, "Finished HTTP request")
require.Contains(t, entries[0].Message, "correlation_id=")
})
}
func testMissing(t *testing.T, client *GitlabNetClient) {
t.Run("Missing error for GET", func(t *testing.T) {
......@@ -128,9 +105,8 @@
})
}
func testMissing(t *testing.T, client *GitlabNetClient) {
t.Run("Missing error for GET", func(t *testing.T) {
hook := testhelper.SetupLogger()
response, err := client.Get(context.Background(), "/missing")
require.EqualError(t, err, "Internal API error (404)")
require.Nil(t, response)
......@@ -134,15 +110,6 @@
response, err := client.Get(context.Background(), "/missing")
require.EqualError(t, err, "Internal API error (404)")
require.Nil(t, response)
require.True(t, testhelper.WaitForLogEvent(hook))
entries := hook.AllEntries()
require.Equal(t, 1, len(entries))
require.Equal(t, logrus.InfoLevel, entries[0].Level)
require.Contains(t, entries[0].Message, "method=GET")
require.Contains(t, entries[0].Message, "status=404")
require.Contains(t, entries[0].Message, "Internal API error")
require.Contains(t, entries[0].Message, "correlation_id=")
})
t.Run("Missing error for POST", func(t *testing.T) {
......@@ -146,7 +113,6 @@
})
t.Run("Missing error for POST", func(t *testing.T) {
hook := testhelper.SetupLogger()
response, err := client.Post(context.Background(), "/missing", map[string]string{})
require.EqualError(t, err, "Internal API error (404)")
require.Nil(t, response)
......@@ -150,15 +116,6 @@
response, err := client.Post(context.Background(), "/missing", map[string]string{})
require.EqualError(t, err, "Internal API error (404)")
require.Nil(t, response)
require.True(t, testhelper.WaitForLogEvent(hook))
entries := hook.AllEntries()
require.Equal(t, 1, len(entries))
require.Equal(t, logrus.InfoLevel, entries[0].Level)
require.Contains(t, entries[0].Message, "method=POST")
require.Contains(t, entries[0].Message, "status=404")
require.Contains(t, entries[0].Message, "Internal API error")
require.Contains(t, entries[0].Message, "correlation_id=")
})
}
......@@ -178,8 +135,6 @@
func testBrokenRequest(t *testing.T, client *GitlabNetClient) {
t.Run("Broken request for GET", func(t *testing.T) {
hook := testhelper.SetupLogger()
response, err := client.Get(context.Background(), "/broken")
require.EqualError(t, err, "Internal API unreachable")
require.Nil(t, response)
......@@ -183,15 +138,6 @@
response, err := client.Get(context.Background(), "/broken")
require.EqualError(t, err, "Internal API unreachable")
require.Nil(t, response)
require.True(t, testhelper.WaitForLogEvent(hook))
entries := hook.AllEntries()
require.Equal(t, 1, len(entries))
require.Equal(t, logrus.InfoLevel, entries[0].Level)
require.Contains(t, entries[0].Message, "method=GET")
require.NotContains(t, entries[0].Message, "status=")
require.Contains(t, entries[0].Message, "Internal API unreachable")
require.Contains(t, entries[0].Message, "correlation_id=")
})
t.Run("Broken request for POST", func(t *testing.T) {
......@@ -195,8 +141,6 @@
})
t.Run("Broken request for POST", func(t *testing.T) {
hook := testhelper.SetupLogger()
response, err := client.Post(context.Background(), "/broken", map[string]string{})
require.EqualError(t, err, "Internal API unreachable")
require.Nil(t, response)
......@@ -200,15 +144,6 @@
response, err := client.Post(context.Background(), "/broken", map[string]string{})
require.EqualError(t, err, "Internal API unreachable")
require.Nil(t, response)
require.True(t, testhelper.WaitForLogEvent(hook))
entries := hook.AllEntries()
require.Equal(t, 1, len(entries))
require.Equal(t, logrus.InfoLevel, entries[0].Level)
require.Contains(t, entries[0].Message, "method=POST")
require.NotContains(t, entries[0].Message, "status=")
require.Contains(t, entries[0].Message, "Internal API unreachable")
require.Contains(t, entries[0].Message, "correlation_id=")
})
}
......
......@@ -11,9 +11,7 @@
"strings"
"time"
"gitlab.com/gitlab-org/labkit/correlation"
log "github.com/sirupsen/logrus"
"gitlab.com/gitlab-org/labkit/log"
)
const (
......@@ -71,8 +69,8 @@
return path
}
func newRequest(ctx context.Context, method, host, path string, data interface{}) (*http.Request, string, error) {
func newRequest(ctx context.Context, method, host, path string, data interface{}) (*http.Request, error) {
var jsonReader io.Reader
if data != nil {
jsonData, err := json.Marshal(data)
if err != nil {
......@@ -75,8 +73,8 @@
var jsonReader io.Reader
if data != nil {
jsonData, err := json.Marshal(data)
if err != nil {
return nil, "", err
return nil, err
}
jsonReader = bytes.NewReader(jsonData)
......@@ -84,6 +82,6 @@
request, err := http.NewRequestWithContext(ctx, method, host+path, jsonReader)
if err != nil {
return nil, "", err
return nil, err
}
......@@ -88,8 +86,6 @@
}
correlationID := correlation.ExtractFromContext(ctx)
return request, correlationID, nil
return request, nil
}
func parseError(resp *http.Response) error {
......@@ -116,7 +112,7 @@
}
func (c *GitlabNetClient) DoRequest(ctx context.Context, method, path string, data interface{}) (*http.Response, error) {
request, correlationID, err := newRequest(ctx, method, c.httpClient.Host, path, data)
request, err := newRequest(ctx, method, c.httpClient.Host, path, data)
if err != nil {
return nil, err
}
......@@ -136,8 +132,7 @@
start := time.Now()
response, err := c.httpClient.Do(request)
fields := log.Fields{
"correlation_id": correlationID,
"method": method,
"url": request.URL.String(),
"duration_ms": time.Since(start) / time.Millisecond,
"method": method,
"url": request.URL.String(),
"duration_ms": time.Since(start) / time.Millisecond,
}
......@@ -143,5 +138,5 @@
}
logger := log.WithFields(fields)
logger := log.WithContextFields(ctx, fields)
if err != nil {
logger.WithError(err).Error("Internal API unreachable")
......
......@@ -5,6 +5,7 @@
"crypto/tls"
"crypto/x509"
"errors"
"fmt"
"io/ioutil"
"net"
"net/http"
......@@ -8,7 +9,8 @@
"io/ioutil"
"net"
"net/http"
"os"
"path/filepath"
"strings"
"time"
......@@ -11,6 +13,5 @@
"path/filepath"
"strings"
"time"
log "github.com/sirupsen/logrus"
"gitlab.com/gitlab-org/labkit/correlation"
......@@ -16,4 +17,6 @@
"gitlab.com/gitlab-org/labkit/correlation"
"gitlab.com/gitlab-org/labkit/log"
"gitlab.com/gitlab-org/labkit/tracing"
)
const (
......@@ -24,6 +27,10 @@
defaultReadTimeoutSeconds = 300
)
var (
ErrCafileNotFound = errors.New("cafile not found")
)
type HttpClient struct {
*http.Client
Host string
......@@ -48,6 +55,22 @@
}
}
func validateCaFile(filename string) error {
if filename == "" {
return nil
}
if _, err := os.Stat(filename); err != nil {
if os.IsNotExist(err) {
return fmt.Errorf("cannot find cafile '%s': %w", filename, ErrCafileNotFound)
}
return err
}
return nil
}
// Deprecated: use NewHTTPClientWithOpts - https://gitlab.com/gitlab-org/gitlab-shell/-/issues/484
func NewHTTPClient(gitlabURL, gitlabRelativeURLRoot, caFile, caPath string, selfSignedCert bool, readTimeoutSeconds uint64) *HttpClient {
c, err := NewHTTPClientWithOpts(gitlabURL, gitlabRelativeURLRoot, caFile, caPath, selfSignedCert, readTimeoutSeconds, nil)
......@@ -59,15 +82,6 @@
// NewHTTPClientWithOpts builds an HTTP client using the provided options
func NewHTTPClientWithOpts(gitlabURL, gitlabRelativeURLRoot, caFile, caPath string, selfSignedCert bool, readTimeoutSeconds uint64, opts []HTTPClientOpt) (*HttpClient, error) {
hcc := &httpClientCfg{
caFile: caFile,
caPath: caPath,
}
for _, opt := range opts {
opt(hcc)
}
var transport *http.Transport
var host string
var err error
......@@ -76,6 +90,20 @@
} else if strings.HasPrefix(gitlabURL, httpProtocol) {
transport, host = buildHttpTransport(gitlabURL)
} else if strings.HasPrefix(gitlabURL, httpsProtocol) {