Commit 48847e59 authored by GitLab Bot's avatar GitLab Bot
Browse files

Add latest changes from gitlab-org/gitlab@master

parent a771420cfe05
......@@ -99,6 +99,8 @@ def prometheus_client
def configured?
kube_client.present? && available?
rescue Gitlab::UrlBlocker::BlockedUrlError
false
end
private
......
---
title: Fix job page not loading because kuberenetes/prometheus URL is blocked
merge_request: 24743
author:
type: fixed
---
title: Port `trigger` keyword in CI config to Core
merge_request: 24191
author:
type: fixed
......@@ -51,7 +51,8 @@ outbound connections for upstream and downstream pipeline dependencies.
## Creating multi-project pipelines from `.gitlab-ci.yml`
> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8.
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8.
> - [Moved](https://gitlab.com/gitlab-org/gitlab/issues/199224) to GitLab Core in 12.8.
### Triggering a downstream pipeline using a bridge job
......@@ -181,7 +182,8 @@ the ones defined in the upstream project will take precedence.
### Mirroring status from triggered pipeline
> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/11238) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.3.
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/11238) in [GitLab Premium](https://about.gitlab.com/pricing/) 12.3.
> - [Moved](https://gitlab.com/gitlab-org/gitlab/issues/199224) to GitLab Core in 12.8.
You can mirror the pipeline status from the triggered pipeline to the source
bridge job by using `strategy: depend`. For example:
......
......@@ -45,7 +45,7 @@ the child pipeline configuration.
## Examples
The simplest case is [triggering a child pipeline](yaml/README.md#trigger-premium) using a
The simplest case is [triggering a child pipeline](yaml/README.md#trigger) using a
local YAML file to define the pipeline configuration. In this case, the parent pipeline will
trigger the child pipeline, and continue without waiting:
......
......@@ -168,6 +168,9 @@ let's say you want to output `HELLO WORLD` for a `TEST` variable.
You can either set the variable directly in the `.gitlab-ci.yml`
file or through the UI.
NOTE: **Note:**
It is possible to [specify variables when running manual jobs](../pipelines.md#specifying-variables-when-running-manual-jobs).
#### Via `.gitlab-ci.yml`
To create a new custom `env_var` variable via [`.gitlab-ci.yml`](../yaml/README.md#variables), define their variable/value pair under
......
......@@ -113,7 +113,7 @@ The following table lists available parameters for jobs:
| [`retry`](#retry) | When and how many times a job can be auto-retried in case of a failure. |
| [`timeout`](#timeout) | Define a custom job-level timeout that takes precedence over the project-wide setting. |
| [`parallel`](#parallel) | How many instances of a job should be run in parallel. |
| [`trigger`](#trigger-premium) | Defines a downstream pipeline trigger. |
| [`trigger`](#trigger) | Defines a downstream pipeline trigger. |
| [`include`](#include) | Allows this job to include external YAML files. Also available: `include:local`, `include:file`, `include:template`, and `include:remote`. |
| [`extends`](#extends) | Configuration entries that this job is going to inherit from. |
| [`pages`](#pages) | Upload the result of a job to use with GitLab Pages. |
......@@ -2572,9 +2572,10 @@ Please be aware that semaphore_test_boosters reports usages statistics to the au
You can then navigate to the **Jobs** tab of a new pipeline build and see your RSpec
job split into three separate jobs.
### `trigger` **(PREMIUM)**
### `trigger`
> [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8.
> - [Introduced](https://gitlab.com/gitlab-org/gitlab/issues/8997) in [GitLab Premium](https://about.gitlab.com/pricing/) 11.8.
> - [Moved](https://gitlab.com/gitlab-org/gitlab/issues/199224) to GitLab Core in 12.8.
`trigger` allows you to define downstream pipeline trigger. When a job created
from `trigger` definition is started by GitLab, a downstream pipeline gets
......@@ -3892,7 +3893,7 @@ job_no_git_strategy:
Triggers can be used to force a rebuild of a specific branch, tag or commit,
with an API call when a pipeline gets created using a trigger token.
Not to be confused with [`trigger`](#trigger-premium).
Not to be confused with [`trigger`](#trigger).
[Read more in the triggers documentation.](../triggers/README.md)
......
......@@ -147,6 +147,7 @@ Complementary reads:
## Integration guides
- [Jira Connect app](integrations/jira_connect.md)
- [Security Scanners](integrations/secure.md)
## Testing guides
......
......@@ -53,8 +53,18 @@ As part of this script we also disable direct and background upload to avoid sit
We can simply run this script from the terminal:
Parameters:
| Attribute | Type | Required | Description |
| --------- | ---- | -------- | ----------- |
| `username` | string | yes | User name |
| `namespace_path` | string | yes | Namespace path |
| `project_path` | string | yes | Project name |
| `archive_path` | string | yes | Path to the exported project tarball you want to import |
| `measurement_enabled` | boolean | no | Measure execution time, number of SQL calls and GC count |
```shell
bundle exec rake "gitlab:import_export:import[root, root, testingprojectimport, /path/to/file.tar.gz]"
bundle exec rake "gitlab:import_export:import[root, root, testingprojectimport, /path/to/file.tar.gz, true]"
```
### Importing via the Rails console
......
......@@ -116,9 +116,9 @@ the scanner with all the libraries and tools it depends on.
### Image size
Depending on the CI infrastucture,
Depending on the CI infrastructure,
the CI may have to fetch the Docker image every time the job runs.
To make the scanning job run fast, and to avoid wasting bandwith,
To make the scanning job run fast, and to avoid wasting bandwidth,
it is important to make Docker images as small as possible,
ideally smaller than 50 MB.
......@@ -189,7 +189,7 @@ then `artifacts:reports:dependency_scanning` must be set to `depscan.json`.
### Exit code
Following the POSIX exit code standard, the scanner will exit with 0 for success and any number from 1 to 255 for anything else.
This also includes the case when vulnerabilities are found.
Success also includes the case when vulnerabilities are found.
### Logging
......@@ -275,7 +275,7 @@ It should not repeat the other fields of the vulnerability object.
In particular, the `description` should not repeat the `location` (what is affected)
or the `solution` (how to mitigate the risk).
There is a proposal to remove either the `name` or the `message`, to remove abmiguities.
There is a proposal to remove either the `name` or the `message`, to remove ambiguities.
See [issue #36779](https://gitlab.com/gitlab-org/gitlab/issues/36779).
#### Solution
......
......@@ -225,13 +225,21 @@ The following table describes details of your subscription for groups:
#### CI pipeline minutes
CI pipeline minutes are the execution time for your [pipelines](../ci/pipelines.md) on our shared runners. Each [GitLab.com tier](https://about.gitlab.com/pricing/) includes a monthly quota of CI pipeline minutes. The quota is applied per group, shared across all members of that group, its subgroups and nested projects. To view the usage, navigate to the group's page, then **Settings > Usage Quotas**.
CI pipeline minutes are the execution time for your [pipelines](../ci/pipelines.md) on GitLab's shared runners. Each [GitLab.com tier](https://about.gitlab.com/pricing/) includes a monthly quota of CI pipeline minutes.
Only pipeline minutes for our shared runners are restricted. If you have a specific runner setup for your projects, there is no limit to your build time on GitLab.com.
Quotas apply to:
The minutes limit only applies to private projects. The available quota is reset on the first of each calendar month at midnight UTC.
- Groups, where the minutes are shared across all members of the group, its subgroups, and nested projects. To view the group's usage, navigate to the group's page, then **Settings > Usage Quotas**.
If you reach your limit, you can [purchase additional CI minutes](#extra-shared-runners-pipeline-minutes), or upgrade your account to [Silver or Gold](https://about.gitlab.com/pricing/). Note, your own runners can still be used even if you reach your limits.
- Your personal account, where the minutes are available for your personal projects. To view and buy personal minutes, click your avatar, then **Settings > Pipeline quota**.
Only pipeline minutes for GitLab shared runners are restricted. If you have a specific runner set up for your projects, there is no limit to your build time on GitLab.com.
The minutes limit does not apply to public projects.
The available quota is reset on the first of each calendar month at midnight UTC.
If you reach your limit, you can [purchase additional CI minutes](#extra-shared-runners-pipeline-minutes), or upgrade your account to [Silver or Gold](https://about.gitlab.com/pricing/). Your own runners can still be used even if you reach your limits.
##### How pipeline quota usage is calculated
......@@ -285,14 +293,16 @@ For example:
##### Purchasing additional minutes
In order to purchase additional minutes, you should follow these steps:
To purchase additional minutes, follow these steps.
1. For group minutes, go to **Group > Settings > Pipelines quota**.
1. Go to **Group > Settings > Pipelines quota**. Once you are on that page, click on **Buy additional minutes**.
For personal project minutes, click your avatar, then **Settings > Pipeline quota**.
![Buy additional minutes](img/buy_btn.png)
1. Click **Buy additional minutes**.
1. Locate the subscription card that is linked to your group on GitLab.com,
click on **Buy more CI minutes**, and complete the details about the transaction.
click **Buy more CI minutes**, and complete the details about the transaction.
![Buy additional minutes](img/buy_minutes_card.png)
......
......@@ -359,7 +359,7 @@ The DAST job can emit various reports.
CAUTION: **Caution:**
The JSON report artifacts are not a public API of DAST and their format is expected to change in the future.
The DAST tool always emits a JSON report report file called `gl-dast-report.json` and sample reports can be found in the [DAST repository](https://gitlab.com/gitlab-org/security-products/dast/tree/master/test/end-to-end/expect).
The DAST tool always emits a JSON report report file called `gl-dast-report.json` and sample reports can be found in the [DAST repository](https://gitlab.com/gitlab-org/security-products/dast/-/tree/master/test/end-to-end/expect).
There are two formats of data in the JSON report that are used side by side: the proprietary ZAP format which will be eventually deprecated, and a "common" format which will be the default in the future.
......
......@@ -163,6 +163,24 @@ If you change your mind before your request is approved, just click the
![Withdraw access request button](img/withdraw_access_request_button.png)
## Changing the owner of a group
Ownership of a group means at least one of its members has
[Owner permission](../permissions.md#group-members-permissions). Groups must have at
least one owner.
Changing the owner of a group with only one owner is possible. To change the sole owner
of a group:
- As an administrator:
1. Go to the group's **{users}** **Members** tab.
1. Give a different member **Owner** permissions.
1. Refresh the page. You can now remove **Owner** permissions from the original owner.
- As the current group's owner:
1. Go to the group's **{users}** **Members** tab.
1. Give a different member **Owner** permissions.
1. Have the new owner sign in and remove **Owner** permissions from you.
## Add projects to a group
There are two different ways to add a new project to a group:
......
# frozen_string_literal: true
module Gitlab
module Ci
class Config
module Entry
##
# Entry that represents a CI/CD Bridge job that is responsible for
# defining a downstream project trigger.
#
class Bridge < ::Gitlab::Config::Entry::Node
include ::Gitlab::Config::Entry::Configurable
include ::Gitlab::Config::Entry::Attributable
include ::Gitlab::Config::Entry::Inheritable
ALLOWED_KEYS = %i[trigger stage allow_failure only except
when extends variables needs rules].freeze
validations do
validates :config, allowed_keys: ALLOWED_KEYS
validates :config, presence: true
validates :name, presence: true
validates :name, type: Symbol
validates :config, disallowed_keys: {
in: %i[only except when start_in],
message: 'key may not be used with `rules`'
},
if: :has_rules?
with_options allow_nil: true do
validates :when,
inclusion: { in: %w[on_success on_failure always],
message: 'should be on_success, on_failure or always' }
validates :extends, type: String
validates :rules, array_of_hashes: true
end
validate on: :composed do
unless trigger.present? || bridge_needs.present?
errors.add(:config, 'should contain either a trigger or a needs:pipeline')
end
end
validate on: :composed do
next unless bridge_needs.present?
next if bridge_needs.one?
errors.add(:config, 'should contain at most one bridge need')
end
end
entry :trigger, ::Gitlab::Ci::Config::Entry::Trigger,
description: 'CI/CD Bridge downstream trigger definition.',
inherit: false
entry :needs, ::Gitlab::Ci::Config::Entry::Needs,
description: 'CI/CD Bridge needs dependency definition.',
inherit: false,
metadata: { allowed_needs: %i[job bridge] }
entry :stage, ::Gitlab::Ci::Config::Entry::Stage,
description: 'Pipeline stage this job will be executed into.',
inherit: false
entry :only, ::Gitlab::Ci::Config::Entry::Policy,
description: 'Refs policy this job will be executed for.',
default: ::Gitlab::Ci::Config::Entry::Policy::DEFAULT_ONLY,
inherit: false
entry :except, ::Gitlab::Ci::Config::Entry::Policy,
description: 'Refs policy this job will be executed for.',
inherit: false
entry :rules, ::Gitlab::Ci::Config::Entry::Rules,
description: 'List of evaluable Rules to determine job inclusion.',
inherit: false,
metadata: {
allowed_when: %w[on_success on_failure always never manual delayed].freeze
}
entry :variables, ::Gitlab::Ci::Config::Entry::Variables,
description: 'Environment variables available for this job.',
inherit: false
helpers(*ALLOWED_KEYS)
attributes(*ALLOWED_KEYS)
def self.matching?(name, config)
!name.to_s.start_with?('.') &&
config.is_a?(Hash) &&
(config.key?(:trigger) || config.key?(:needs))
end
def self.visible?
true
end
def compose!(deps = nil)
super do
has_workflow_rules = deps&.workflow&.has_rules?
# If workflow:rules: or rules: are used
# they are considered not compatible
# with `only/except` defaults
#
# Context: https://gitlab.com/gitlab-org/gitlab/merge_requests/21742
if has_rules? || has_workflow_rules
# Remove only/except defaults
# defaults are not considered as defined
@entries.delete(:only) unless only_defined?
@entries.delete(:except) unless except_defined?
end
end
end
def has_rules?
@config&.key?(:rules)
end
def name
@metadata[:name]
end
def value
{ name: name,
trigger: (trigger_value if trigger_defined?),
needs: (needs_value if needs_defined?),
ignore: !!allow_failure,
stage: stage_value,
when: when_value,
extends: extends_value,
variables: (variables_value if variables_defined?),
rules: (rules_value if has_rules?),
only: only_value,
except: except_value }.compact
end
def bridge_needs
needs_value[:bridge] if needs_value
end
private
def overwrite_entry(deps, key, current_entry)
deps.default[key] unless current_entry.specified?
end
end
end
end
end
end
......@@ -36,7 +36,7 @@ def has_visible_job?
end
end
TYPES = [Entry::Hidden, Entry::Job].freeze
TYPES = [Entry::Hidden, Entry::Job, Entry::Bridge].freeze
private_constant :TYPES
......@@ -77,5 +77,3 @@ def compose!(deps = nil)
end
end
end
::Gitlab::Ci::Config::Entry::Jobs.prepend_if_ee('::EE::Gitlab::Ci::Config::Entry::Jobs')
# frozen_string_literal: true
module Gitlab
module Ci
class Config
module Entry
##
# Entry that represents a cross-project downstream trigger.
#
class Trigger < ::Gitlab::Config::Entry::Simplifiable
strategy :SimpleTrigger, if: -> (config) { config.is_a?(String) }
strategy :ComplexTrigger, if: -> (config) { config.is_a?(Hash) }
class SimpleTrigger < ::Gitlab::Config::Entry::Node
include ::Gitlab::Config::Entry::Validatable
validations { validates :config, presence: true }
def value
{ project: @config }
end
end
class ComplexTrigger < ::Gitlab::Config::Entry::Simplifiable
strategy :CrossProjectTrigger, if: -> (config) { !config.key?(:include) }
strategy :SameProjectTrigger, if: -> (config) do
::Feature.enabled?(:ci_parent_child_pipeline, default_enabled: true) &&
config.key?(:include)
end
class CrossProjectTrigger < ::Gitlab::Config::Entry::Node
include ::Gitlab::Config::Entry::Validatable
include ::Gitlab::Config::Entry::Attributable
ALLOWED_KEYS = %i[project branch strategy].freeze
attributes :project, :branch, :strategy
validations do
validates :config, presence: true
validates :config, allowed_keys: ALLOWED_KEYS
validates :project, presence: true
validates :branch, type: String, allow_nil: true
validates :strategy, type: String, inclusion: { in: %w[depend], message: 'should be depend' }, allow_nil: true
end
end
class SameProjectTrigger < ::Gitlab::Config::Entry::Node
include ::Gitlab::Config::Entry::Validatable
include ::Gitlab::Config::Entry::Attributable
include ::Gitlab::Config::Entry::Configurable
INCLUDE_MAX_SIZE = 3
ALLOWED_KEYS = %i[strategy include].freeze
attributes :strategy
validations do
validates :config, presence: true
validates :config, allowed_keys: ALLOWED_KEYS
validates :strategy, type: String, inclusion: { in: %w[depend], message: 'should be depend' }, allow_nil: true
end
entry :include, ::Gitlab::Ci::Config::Entry::Includes,
description: 'List of external YAML files to include.',
reserved: true,
metadata: { max_size: INCLUDE_MAX_SIZE }
def value
@config
end
end
class UnknownStrategy < ::Gitlab::Config::Entry::Node
def errors
if ::Feature.enabled?(:ci_parent_child_pipeline, default_enabled: true)
['config must specify either project or include']
else
['config must specify project']
end
end
end
end
class UnknownStrategy < ::Gitlab::Config::Entry::Node
def errors
["#{location} has to be either a string or a hash"]
end
end
end
end
end
end
end
......@@ -7,12 +7,12 @@
# 2. Performs Sidekiq job synchronously
#
# @example
# bundle exec rake "gitlab:import_export:import[root, root, imported_project, /path/to/file.tar.gz]"
# bundle exec rake "gitlab:import_export:import[root, root, imported_project, /path/to/file.tar.gz, true]"
#
namespace :gitlab do
namespace :import_export do
desc 'GitLab | Import/Export | EXPERIMENTAL | Import large project archives'
task :import, [:username, :namespace_path, :project_path, :archive_path] => :gitlab_environment do |_t, args|
task :import, [:username, :namespace_path, :project_path, :archive_path, :measurement_enabled] => :gitlab_environment do |_t, args|
# Load it here to avoid polluting Rake tasks with Sidekiq test warnings
require 'sidekiq/testing'
......@@ -26,7 +26,8 @@ namespace :gitlab do
namespace_path: args.namespace_path,
project_path: args.project_path,
username: args.username,
file_path: args.archive_path
file_path: args.archive_path,
measurement_enabled: args.measurement_enabled == 'true'
).import
end
end
......@@ -38,6 +39,7 @@ class GitlabProjectImport
@file_path = opts.fetch(:file_path)
@namespace = Namespace.find_by_full_path(opts.fetch(:namespace_path))
@current_user = User.find_by_username(opts.fetch(:username))
@measurement_enabled = opts.fetch(:measurement_enabled)
end
def import
......@@ -72,6 +74,54 @@ class GitlabProjectImport
RequestStore.clear!
end
def with_count_queries(&block)
count = 0
counter_f = ->(name, started, finished, unique_id, payload) {
unless payload[:name].in? %w[CACHE SCHEMA]
count += 1
end
}
ActiveSupport::Notifications.subscribed(counter_f, "sql.active_record", &block)
puts "Number of sql calls: #{count}"
end
def with_gc_counter
gc_counts_before = GC.stat.select { |k, v| k =~ /count/ }
yield
gc_counts_after = GC.stat.select { |k, v| k =~ /count/ }
stats = gc_counts_before.merge(gc_counts_after) { |k, vb, va| va - vb }
puts "Total GC count: #{stats[:count]}"
puts "Minor GC count: #{stats[:minor_gc_count]}"
puts "Major GC count: #{stats[:major_gc_count]}"
end
def with_measure_time
timing = Benchmark.realtime do
yield
end
time = Time.at(timing).utc.strftime("%H:%M:%S")
puts "Time to finish: #{time}"
end
def with_measuring
puts "Measuring enabled..."
with_gc_counter do
with_count_queries do
with_measure_time do
yield
end
end
end
end
def measurement_enabled?
@measurement_enabled != false
end
# We want to ensure that all Sidekiq jobs are executed
# synchronously as part of that process.
# This ensures that all expensive operations do not escape
......@@ -79,8 +129,13 @@ class GitlabProjectImport
def with_isolated_sidekiq_job
Sidekiq::Testing.fake! do
with_request_store do
# If you are attempting to import a large project into a development environment,
# you may see Gitaly throw an error about too many calls or invocations.
# This is due to a n+1 calls limit being set for development setups (not enforced in production)
# https://gitlab.com/gitlab-org/gitlab/-/merge_requests/24475#note_283090635
# For development setups, this code-path will be excluded from n+1 detection.
::Gitlab::GitalyClient.allow_n_plus_1_calls do
yield
measurement_enabled? ? with_measuring { yield } : yield
end
end
......
......@@ -16773,6 +16773,9 @@ msgstr ""
msgid "Security Reports|Oops, something doesn't seem right."
msgstr ""
msgid "Security Reports|Security reports can only be accessed by authorized users."
msgstr ""