Commit 4328651a authored by Georges Racinet's avatar Georges Racinet 🦑
Browse files

Fully native Mercurial: merge from heptapod branch

We're staying for now on the development version of HGitaly
that provides the Blob and Tree methods, but we'll switch to
a released version for Heptapod 0.20.0rc2

--HG--
branch : heptapod-0-21
Pipeline #20644 canceled with stages
in 21 minutes and 10 seconds
......@@ -940,6 +940,11 @@ def mercurial?
MERCURIAL_VCS_TYPES.include?(vcs_type)
end
def hgitaly1?
vcs_type == 'hg' && (Feature.disabled?(:hg_fully_native, self, default_enabled: :yaml) ||
Feature.enabled?(:hgitaly1_fallback, self, default_enabled: :yaml))
end
def has_git_repo?
# currently, even "Mercurial native" Projects actually use an
# auxiliary Git repo. This knowledge is essential for backups
......
......@@ -1247,6 +1247,8 @@ def initialize_raw_repository
if mercurial? && repo_type == Gitlab::GlRepository::PROJECT
repo.hg_project_for_perms = project
# do not attempt to simplify: only HgitalyRepository has the attribute writer
repo.is_hgitaly1 = true if project.hgitaly1?
end
repo
......
---
name: hg_fully_native
introduced_by_url:
rollout_issue_url:
milestone: 'Heptapod 0.21'
type: development
group: group::hgitaly
default_enabled: false
---
name: hgitaly1_fallback
introduced_by_url:
rollout_issue_url:
milestone: 'Heptapod 0.21'
type: development
group: group::hgitaly
default_enabled: false
# Mercurial projects in Heptapod
The primary objective of Heptapod is to add Mercurial support to GitLab.
## Version Control System (VCS) types
Each project has a `vcs_type` field to describe the VCS in use. Administrators
can select which of these are allowed in the application settings
(Admin Area > Settings > General > Visibility and access controls).
We are currently using two distinct VCS types for Mercurial projects, to support our
effort towards "native" handling of Mercurial.
### Mercurial VCS type
Value of the `vcs_type` field: `hg_git`.
This is the historical way to handle Mercurial repositories, exposing to the rest of
GitLab by maintaining a conversion to Git. Even though the server side Mercurial
repository is the single source of truth, this has many drawbacks.
The Rails (web) application accesses content for display through Gitaly, but also needs
to perform operations (such as merges) on the local file system. Hence the Gitaly server
*must* be local.
This does not prevent using also remote Gitaly servers, as long as they serve Git
projects only.
### Mercurial (native) VCS type
Value of the `vcs_type` field: `hg`.
Still experimental as of Heptapod 0.21, these are meant to eventually replace the `hg_git`
projects. Projects created right now under this mode should not need heavy and risky migrations
in the future.
The native Mercurial projects are exposed to the various GitLab components via the HGitaly
service. Although content is still converted to Git at this point, the auxiliary Git repository
will eventually become completely superfluous and fade away.
There are three stages of development:
+ HGitaly1: references and commits are served via HGitaly, but files and directories are still read
from the Git repository. This is the norm as of Heptapod 0.21
+ HGitaly2 (fully native, currently under evaluation via feature flags): All Mercurial content is
served via HGitaly. The auxiliary Git repository is useful only to be able to get back to the
HGitaly1 stage
+ HGitaly3 (development not started): all read and write operations, including creation and
maintenance tasks happen through HGitaly. The web application does not need to access
the repositories directly, and the HGitaly servers can be remote.
More insight on these different ways to handle Mercurial is avalaible
[on the Heptapod web site](https://heptapod.net/the-road-to-fully-native-mercurial-in-heptapod.md#the-road-to-fully-native-mercurial-in-heptapod)
## Managing the fully native mode
Warning: as of Heptapod 0.21, this is highly experimental, but you can get back to
the HGitaly1 stage easily.
In all cases, only native Mercurial projects are affected by this.
#### The feature flags
- `hg_fully_native`: switches on the fully native mode
- `hgitaly1_fallback`: make a given project still use the HGitaly1 mode, even
if the whole instance is using the fully native mode
#### For the whole instance
To have all "Mercurial (native)" projects use the fully native mode, do this under the
[Rails console]
```ruby
Feature.enable(:hg_fully_native)
```
To get back to the HGitaly1 mode of operation:
```ruby
Feature.disable(:hg_fully_native)
```
It is not possible to use `disable()` for a single project. To exclude a given project
from the fully native mode, do:
```ruby
project = Project.find(project_id) # project_id is the numeric ID,
Feature.enable(:hgitaly1_fallback, project)
```
Of course, you are free to use other means to retrieve a `Project` instance, using
the `ActiveRecord` API (the Object Relational Mapping in use).
#### For a given project
##### Introspection
To know whether a given project is running under the fully native mode, retrieve its
details using the [Projects API](../api/projects.md).
- if `vcs_type` is `hg` and `hgitaly1` is `false`, the project is running under the fully
native mode.
- if `hgitaly1` is `true`, the project is running the HGitaly1 mode of operation
- if `vcs_type` is not `hg`, this is not a native Mercurial repository, the value of
`hgitaly1` is meaningless.
##### Using the Rest API
This is perhaps the simplest. Use the [Features flags API](../api/features.md#set-or-create-a-feature)
with this data:
- `name`: `hg_fully_native`
- `value`: `true` or `false` whether you want to enable or disable
- `project`: full web path to the project
Note: disabling the fully native mode this way does not work if it is enabled for the
whole instance. Enable the `hgitaly_fallback` feature flag instead.
##### Under the [Rails console]:
```ruby
project = Project.find(project_id) # project_id is the numeric ID,
Feature.enable(:hg_fully_native, project)
```
This has no impact on non-native Mercurial projects nor on Git projects either
To get back to HGitaly1 mode of operation (assuming the feature flag is not
activated for the whole instance):
```ruby
project = Project.find(project_id) # project_id is the numeric ID,
Feature.disable(:hg_fully_native, project)
```
[Rails console]: operations/rails_console.md
......@@ -56,6 +56,7 @@ class Project < BasicProjectDetails
expose(:snippets_enabled) { |project, options| project.feature_available?(:snippets, options[:current_user]) }
expose :service_desk_enabled
expose :service_desk_address
expose :hgitaly1?, as: :hgitaly1
expose(:can_create_merge_request_in) do |project, options|
Ability.allowed?(options[:current_user], :create_merge_request_in, project)
......
......@@ -52,10 +52,14 @@ def find(repository, sha, path, limit: MAX_DATA_DISPLAY_SIZE)
def tree_entry(repository, sha, path, limit)
return unless path
git_sha = repository.gitsha_from_sha(sha)
# sha can also be a ref, so let's keep using it if it's not
# known as an actual Mercurial SHA
query_sha = git_sha.nil? ? sha : git_sha
query_sha = if repository.hgitaly1?
git_sha = repository.gitsha_from_sha(sha)
# sha can also be a ref, so let's keep using it if it's not
# known as an actual Mercurial SHA
git_sha.nil? ? sha : git_sha
else
sha
end
path = path.sub(%r{\A/*}, '')
path = '/' if path.empty?
......@@ -68,7 +72,7 @@ def tree_entry(repository, sha, path, limit)
# actually 0.
req_limit = limit == 0 ? 1 : limit
entry = Gitlab::GitalyClient::CommitService.new(repository).tree_entry(query_sha, path, req_limit)
entry = repository.gitaly_commit_client.tree_entry(query_sha, path, req_limit)
return unless entry
entry.data = "" if limit == 0
......@@ -95,6 +99,12 @@ def raw(repository, sha)
# to the caller to limit the number of blobs and blob_size_limit.
#
def batch(repository, blob_references, blob_size_limit: MAX_DATA_DISPLAY_SIZE)
unless repository.hgitaly1?
return blob_references.each_slice(BATCH_SIZE).flat_map do |refs|
repository.gitaly_blob_client.get_blobs(refs, blob_size_limit).to_a
end
end
# HGitaly1 transitional state: converting to Git commit SHAs
# rubocop:disable Style/MultilineBlockChain (temp code anyway)
blob_references.map do |commit_sha, path|
......@@ -223,10 +233,10 @@ def external_storage
alias_method :external_size, :lfs_size
# Perform transitional back conversion for given repository
# Transitional Hgitaly1 back conversion for given repository
#
# Heptapod specific, and transitional: will be useful only until
# we reach the HGitaly2 milestone
# all native repositories are fully native.
def hpd_back_convert!(repository)
converted = repository.sha_from_gitsha(@commit_id)
# commit_id can actually be a ref. In that case it will fail
......
......@@ -194,7 +194,7 @@ def get_messages(repository, commit_ids)
def initialize(repository, raw_commit, head = nil, lazy_load_parents: false)
raise "Nil as raw commit passed" unless raw_commit
@repository = repository.hpd_original
@repository = repository
@head = head
@lazy_load_parents = lazy_load_parents
......@@ -212,10 +212,10 @@ def init_commit(raw_commit)
end
end
# Perform transitional back conversion for given repository
# Transitional Hgitaly1 back conversion for given repository
#
# Heptapod specific, and transitional: will be useful only until
# we reach the HGitaly2 milestone
# all native repositories are fully native.
def hpd_back_convert!(repository)
@id = repository.sha_from_gitsha(@id)
self
......
......@@ -64,10 +64,8 @@ def initialize(storage, relative_path, gl_repository, gl_project_path)
@name = @relative_path.split("/").last
end
# Subclassed by HGitaly as temporary scaffolding, waiting for
# HGitaly2 milestone to be reached
def hpd_original
self
def hgitaly1?
false
end
def to_s
......
......@@ -22,15 +22,20 @@ def where(repository, sha, path = nil, recursive = false)
end
def tree_entries(repository, sha, path, recursive)
git_sha = repository.gitsha_from_sha(sha)
# sha can also be a ref, so let's keep using it if it's not
# known as an actual Mercurial SHA
sha = git_sha unless git_sha.nil?
wrapped_gitaly_errors do
repository.gitaly_commit_client.tree_entries(
repository, sha, path, recursive
).map do |tree|
tree.hpd_back_convert!(repository)
if repository.hgitaly1?
git_sha = repository.gitsha_from_sha(sha)
# sha can also be a ref, so let's keep using it if it's not
# known as an actual Mercurial SHA
sha = git_sha unless git_sha.nil?
repository.gitaly_commit_client.tree_entries(
repository, sha, path, recursive
).map do |tree|
tree.hpd_back_convert!(repository)
end
else
repository.gitaly_commit_client.tree_entries(repository, sha, path, recursive)
end
end
end
......@@ -73,10 +78,10 @@ def initialize(options)
end
end
# Perform transitional back conversion for given repository
# Transitional Hgitaly1 back conversion for given repository
#
# Heptapod specific, and transitional: will be useful only until
# we reach the HGitaly2 milestone
# all native repositories are fully native.
def hpd_back_convert!(repository)
converted = repository.sha_from_gitsha(@commit_id)
# commit_id can actually be a ref. In that case it will fail
......
......@@ -5,9 +5,16 @@ module GitalyClient
class BlobService
include Gitlab::EncodingHelper
def initialize(repository)
def initialize(repository, vcs: nil)
@gitaly_repo = repository.gitaly_repository
@is_hgitaly = repository.is_a?(Gitlab::Mercurial::HgitalyRepository)
@storage = if repository.hgitaly1?
repository.storage
else
Gitlab::GitalyClient.vcs_qualified_storage(
repository.storage, vcs)
end
@vcs = vcs
end
def get_blob(oid:, limit:)
......@@ -16,12 +23,12 @@ def get_blob(oid:, limit:)
oid: oid,
limit: limit
)
response = GitalyClient.call(@gitaly_repo.storage_name, :blob_service, :get_blob, request, timeout: GitalyClient.fast_timeout)
response = GitalyClient.call(@storage, :blob_service, :get_blob, request, timeout: GitalyClient.fast_timeout)
consume_blob_response(response)
end
def batch_lfs_pointers(blob_ids)
return [] if blob_ids.empty?
return [] if blob_ids.empty? || @vcs == 'hg'
request = Gitaly::GetLFSPointersRequest.new(
repository: @gitaly_repo,
......@@ -46,7 +53,7 @@ def get_blobs(revision_paths, limit = -1)
)
response = GitalyClient.call(
@gitaly_repo.storage_name,
@storage,
:blob_service,
:get_blobs,
request,
......@@ -69,7 +76,7 @@ def get_blob_types(revision_paths, limit = -1)
)
response = GitalyClient.call(
@gitaly_repo.storage_name,
@storage,
:blob_service,
:get_blobs,
request,
......@@ -79,7 +86,7 @@ def get_blob_types(revision_paths, limit = -1)
end
def get_new_lfs_pointers(revision, limit, not_in, dynamic_timeout = nil)
return [] if @is_hgitaly
return [] if @vcs == 'hg'
request = Gitaly::GetNewLFSPointersRequest.new(
repository: @gitaly_repo,
......@@ -111,7 +118,7 @@ def get_new_lfs_pointers(revision, limit, not_in, dynamic_timeout = nil)
end
def get_all_lfs_pointers
return [] if @is_hgitaly
return [] if @vcs == 'hg'
request = Gitaly::GetAllLFSPointersRequest.new(
repository: @gitaly_repo
......
......@@ -10,7 +10,8 @@ def initialize(repository, vcs: nil)
# This client class doesn't have a separate `@storage` instance
# variable, so we're cloning the incoming `repository`, with
# an explicit `storage`.
@repository = vcs.nil? ? repository : repository.vcs_storage_clone(vcs)
@repository = repository
@storage = Gitlab::GitalyClient.vcs_qualified_storage(repository.storage, vcs)
@git_storage = repository.storage
end
......@@ -20,6 +21,7 @@ def ls_files(revision)
revision: encode_binary(revision)
)
# TODO HGitaly2!
response = GitalyClient.call(@git_storage, :commit_service, :list_files, request, timeout: GitalyClient.medium_timeout)
response.flat_map do |msg|
msg.paths.map { |d| EncodingHelper.encode!(d.dup) }
......@@ -33,7 +35,7 @@ def ancestor?(ancestor_id, child_id)
child_id: child_id
)
GitalyClient.call(@repository.storage, :commit_service, :commit_is_ancestor, request, timeout: GitalyClient.fast_timeout).value
GitalyClient.call(@storage, :commit_service, :commit_is_ancestor, request, timeout: GitalyClient.fast_timeout).value
end
def diff(from, to, options = {})
......@@ -76,7 +78,8 @@ def diff_from_parent(commit, options = {})
def commit_deltas(commit)
request = Gitaly::CommitDeltaRequest.new(diff_from_parent_request_params(commit))
response = GitalyClient.call(@git_storage, :diff_service, :commit_delta, request, timeout: GitalyClient.fast_timeout)
storage = @repository.hgitaly1? ? @git_storage : @storage
response = GitalyClient.call(storage, :diff_service, :commit_delta, request, timeout: GitalyClient.fast_timeout)
response.flat_map { |msg| msg.deltas }
end
......@@ -94,7 +97,8 @@ def tree_entry(ref, path, limit = nil)
path: encode_binary(path),
limit: limit.to_i
)
response = GitalyClient.call(@git_storage, :commit_service, :tree_entry, request, timeout: GitalyClient.medium_timeout)
storage = @repository.hgitaly1? ? @git_storage : @storage
response = GitalyClient.call(storage, :commit_service, :tree_entry, request, timeout: GitalyClient.medium_timeout)
entry = nil
data = []
......@@ -120,7 +124,8 @@ def tree_entries(repository, revision, path, recursive)
recursive: recursive
)
response = GitalyClient.call(@git_storage, :commit_service, :get_tree_entries, request, timeout: GitalyClient.medium_timeout)
storage = @repository.hgitaly1? ? @git_storage : @storage
response = GitalyClient.call(storage, :commit_service, :get_tree_entries, request, timeout: GitalyClient.medium_timeout)
response.flat_map do |message|
message.entries.map do |gitaly_tree_entry|
......@@ -150,7 +155,7 @@ def commit_count(ref, options = {})
request.path = encode_binary(options[:path]) if options[:path].present?
request.max_count = options[:max_count] if options[:max_count].present?
GitalyClient.call(@repository.storage, :commit_service, :count_commits, request, timeout: GitalyClient.medium_timeout).count
GitalyClient.call(@storage, :commit_service, :count_commits, request, timeout: GitalyClient.medium_timeout).count
end
def diverging_commit_count(from, to, max_count:)
......@@ -160,7 +165,7 @@ def diverging_commit_count(from, to, max_count:)
to: encode_binary(to),
max_count: max_count
)
response = GitalyClient.call(@repository.storage, :commit_service, :count_diverging_commits, request, timeout: GitalyClient.medium_timeout)
response = GitalyClient.call(@storage, :commit_service, :count_diverging_commits, request, timeout: GitalyClient.medium_timeout)
[response.left_count, response.right_count]
end
......@@ -174,7 +179,8 @@ def list_last_commits_for_tree(revision, path, offset: 0, limit: 25, literal_pat
global_options: parse_global_options!(literal_pathspec: literal_pathspec)
)
response = GitalyClient.call(@git_storage, :commit_service, :list_last_commits_for_tree, request, timeout: GitalyClient.medium_timeout)
storage = @repository.hgitaly1? ? @git_storage : @storage
response = GitalyClient.call(storage, :commit_service, :list_last_commits_for_tree, request, timeout: GitalyClient.medium_timeout)
response.each_with_object({}) do |gitaly_response, hsh|
gitaly_response.commits.each do |commit_for_tree|
......@@ -191,7 +197,7 @@ def last_commit_for_path(revision, path, literal_pathspec: false)
global_options: parse_global_options!(literal_pathspec: literal_pathspec)
)
gitaly_commit = GitalyClient.call(@repository.storage, :commit_service, :last_commit_for_path, request, timeout: GitalyClient.fast_timeout).commit
gitaly_commit = GitalyClient.call(@storage, :commit_service, :last_commit_for_path, request, timeout: GitalyClient.fast_timeout).commit
return unless gitaly_commit
Gitlab::Git::Commit.new(@repository, gitaly_commit)
......@@ -204,18 +210,24 @@ def between(from, to)
to: to
)
response = GitalyClient.call(@repository.storage, :commit_service, :commits_between, request, timeout: GitalyClient.medium_timeout)
response = GitalyClient.call(@storage, :commit_service, :commits_between, request, timeout: GitalyClient.medium_timeout)
consume_commits_response(response)
end
def diff_stats(left_commit_sha, right_commit_sha)
if @repository.hgitaly1?
left_commit_sha = @repository.fallbacking_gitsha_from_sha(left_commit_sha)
right_commit_sha = @repository.fallbacking_gitsha_from_sha(right_commit_sha)
end
request = Gitaly::DiffStatsRequest.new(
repository: @gitaly_repo,
left_commit_id: @repository.fallbacking_gitsha_from_sha(left_commit_sha),
right_commit_id: @repository.fallbacking_gitsha_from_sha(right_commit_sha)
left_commit_id: left_commit_sha,
right_commit_id: right_commit_sha
)
response = GitalyClient.call(@git_storage, :diff_service, :diff_stats, request, timeout: GitalyClient.medium_timeout)
storage = @repository.hgitaly1? ? @git_storage : @storage
response = GitalyClient.call(storage, :diff_service, :diff_stats, request, timeout: GitalyClient.medium_timeout)
response.flat_map(&:stats)
end
......@@ -225,7 +237,7 @@ def find_changed_paths(commits)
commits: commits
)
response = GitalyClient.call(@repository.storage, :diff_service, :find_changed_paths, request, timeout: GitalyClient.medium_timeout)
response = GitalyClient.call(@storage, :diff_service, :find_changed_paths, request, timeout: GitalyClient.medium_timeout)
response.flat_map do |msg|
msg.paths.map do |path|
Gitlab::Git::ChangedPath.new(
......@@ -245,7 +257,7 @@ def find_all_commits(opts = {})
)
request.order = opts[:order].upcase if opts[:order].present?
response = GitalyClient.call(@repository.storage, :commit_service, :find_all_commits, request, timeout: GitalyClient.medium_timeout)
response = GitalyClient.call(@storage, :commit_service, :find_all_commits, request, timeout: GitalyClient.medium_timeout)
consume_commits_response(response)
end
......@@ -254,7 +266,7 @@ def list_commits_by_oid(oids)
request = Gitaly::ListCommitsByOidRequest.new(repository: @gitaly_repo, oid: oids)
response = GitalyClient.call(@repository.storage, :commit_service, :list_commits_by_oid, request, timeout: GitalyClient.medium_timeout)
response = GitalyClient.call(@storage, :commit_service, :list_commits_by_oid, request, timeout: GitalyClient.medium_timeout)
consume_commits_response(response)
rescue GRPC::NotFound # If no repository is found, happens mainly during testing
[]
......@@ -271,7 +283,7 @@ def commits_by_message(query, revision: '', path: '', limit: 1000, offset: 0, li
global_options: parse_global_options!(literal_pathspec: literal_pathspec)
)
response = GitalyClient.call(@repository.storage, :commit_service, :commits_by_message, request, timeout: GitalyClient.medium_timeout)
response = GitalyClient.call(@storage, :commit_service, :commits_by_message, request, timeout: GitalyClient.medium_timeout)
consume_commits_response(response)
end
......@@ -289,7 +301,7 @@ def raw_blame(revision, path)
path: encode_binary(path)
)
response = GitalyClient.call(@repository.storage, :commit_service, :raw_blame, request, timeout: GitalyClient.medium_timeout)
response = GitalyClient.call(@storage, :commit_service, :raw_blame, request, timeout: GitalyClient.medium_timeout)
response.reduce([]) { |memo, msg| memo << msg.data }.join
end
......@@ -325,7 +337,7 @@ def commit_stats(revision)
repository: @gitaly_repo,
revision: encode_binary(revision)
)
GitalyClient.call(@repository.storage, :commit_service, :commit_stats, request, timeout: GitalyClient.medium_timeout)
GitalyClient.call(@storage, :commit_service, :commit_stats, request, timeout: GitalyClient.medium_timeout)
end
def find_commits(options)
......@@ -349,7 +361,7 @@ def find_commits(options)
request.paths = encode_repeated(Array(options[:path])) if options[:path].present?
response = GitalyClient.call(@repository.storage, :commit_service, :find_commits, request, timeout: GitalyClient.medium_timeout)
response = GitalyClient.call(@storage, :commit_service, :find_commits, request, timeout: GitalyClient.medium_timeout)
consume_commits_response(response)
end
......@@ -366,7 +378,7 @@ def filter_shas_with_signatures(shas)
end
end
response = GitalyClient.call(@repository.storage, :commit_service, :filter_shas_with_signatures, enum, timeout: GitalyClient.fast_timeout)
response = GitalyClient.call(@storage, :commit_service, :filter_shas_with_signatures, enum, timeout: GitalyClient.fast_timeout)
response.flat_map do |msg|
msg.shas.map { |sha| EncodingHelper.encode!(sha) }
end
......@@ -393,7 +405,7 @@ def get_commit_signatures(commit_ids)
def get_commit_messages(commit_ids)
request = Gitaly::GetCommitMessagesRequest.new(repository: @gitaly_repo, commit_ids: commit_ids)
response = GitalyClient.call(@repository.storage, :commit_service, :get_commit_messages, request, timeout: GitalyClient.fast_timeout)
response = GitalyClient.call(@storage, :commit_service, :get_commit_messages, request, timeout: GitalyClient.fast_timeout)
messages = Hash.new { |h, k| h[k] = +''.b }
current_commit_id = nil
......@@ -411,7 +423,7 @@ def list_commits_by_ref_name(refs)
request = Gitaly::ListCommitsByRefNameRequest
.new(repository: @gitaly_repo, ref_names: refs.map { |ref| encode_binary(ref) })
response = GitalyClient.call(@repository.storage, :commit_service, :list_commits_by_ref_name, request, timeout: GitalyClient.medium_timeout)
response = GitalyClient.call(@storage, :commit_service, :list_commits_by_ref_name, request, timeout: GitalyClient.medium_timeout)
commit_refs = response.flat_map do |message|
message.commit_refs.map do |commit_ref|
......@@ -436,7 +448,8 @@ def call_commit_diff(request_params, options = {})
request_params.merge!(Gitlab::Git::DiffCollection.limits(options).to_h)
request = Gitaly::CommitDiffRequest.new(request_params)
response = GitalyClient.call(@git_storage, :diff_service, :commit_diff, request, timeout: GitalyClient.medium_timeout)