Skip to content
Snippets Groups Projects
Commit 6f8899ba authored by Pierre-Yves David's avatar Pierre-Yves David :octopus:
Browse files

documentation: add extensive details about benchmarks configuration

This should make things clearer before we add another variants.
parent 6612a120
No related branches found
No related tags found
No related merge requests found
Pipeline #56163 passed
===============================
Details on benchmarks in Poulpe
===============================
A benchmark describe "something" we want to get performance data on. It is
defined in a "poulpe-benchmark-definition" file (`.pbd`). It will use code that
exists in a poulpe's "binary-environment" to benchmark that "something" on the
data present in a poulpe's "binary-environment".
The `.pbd` files includes multiple metadata and definition. Some are common to
all type of benchmarks, some are specific to each type of benchmarks.
Generic metata data
===================
section: "meta"
---------------
format
~~~~~~
The version of this benchmark definition. Currently known version are:
:0: The very first, quite volatile, version.
name
~~~~
identifier of the benchmark. This will be stored in result.
method
~~~~~~
The way this benchmark is expected to be run. It will also define most of the
other value we expect to find to configure this benchmark.
The current list of supported method is :
:simple-command: benchmarking the run of a single shell command.
Simple-command configuration
============================
section: "single-command"
-------------------------
command
~~~~~~~
The command to be run.
It can we extended with variants (see next section)
cwd
~~~
The directory to run the command in.
Value can be substituded for data-env variable using the
`"DATA-VARS:path.to.variable.in.the.data.env"` syntax.
For example `"DATA-VARS:mercurial.status.large.all"`
The default should be the root of the data-env.
This can be changed by variants (see next section).
variants
--------
Variants allow for slight variation of a benchmark. For example, they allow to change the
input data, of to turn some feature on and off.
Variants are organised within independant "dimensions".
Each dimension has a set of values, each value can modify the benchmark. Each
dimension must have a default value.
defining a variant
~~~~~~~~~~~~~~~~~~
Variants are defined under keys following this patterns :
simple-command.variants.dimensions.<dimension-name>.<variant-key>
The `dimension-name` and `variant-key` are used as identifier in the benchmark
result and for selecting variant at run time.
variants variables
~~~~~~~~~~~~~~~~~~
default
~~~~~~~
Exactly one variant must have it set to `true` per dimension.
When set and no variant have been explicitly selected for this dimension, this
variant will be used.
extend-command
~~~~~~~~~~~~~~
A string to append to the commend when this variable is selected.
cwd
~~~
overwrite the global `simple-command.cwd` variable. Behave the same.
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment