diff --git a/docs/benchmarks.rst b/docs/benchmarks.rst new file mode 100644 index 0000000000000000000000000000000000000000..6f8899bae69ec1391bf4f44b0dcab4a1e64de777_ZG9jcy9iZW5jaG1hcmtzLnJzdA== --- /dev/null +++ b/docs/benchmarks.rst @@ -0,0 +1,108 @@ +=============================== +Details on benchmarks in Poulpe +=============================== + +A benchmark describe "something" we want to get performance data on. It is +defined in a "poulpe-benchmark-definition" file (`.pbd`). It will use code that +exists in a poulpe's "binary-environment" to benchmark that "something" on the +data present in a poulpe's "binary-environment". + +The `.pbd` files includes multiple metadata and definition. Some are common to +all type of benchmarks, some are specific to each type of benchmarks. + +Generic metata data +=================== + +section: "meta" +--------------- + +format +~~~~~~ + +The version of this benchmark definition. Currently known version are: + +:0: The very first, quite volatile, version. + +name +~~~~ + +identifier of the benchmark. This will be stored in result. + +method +~~~~~~ + +The way this benchmark is expected to be run. It will also define most of the +other value we expect to find to configure this benchmark. + +The current list of supported method is : + +:simple-command: benchmarking the run of a single shell command. + +Simple-command configuration +============================ + +section: "single-command" +------------------------- + +command +~~~~~~~ + +The command to be run. + +It can we extended with variants (see next section) + +cwd +~~~ + +The directory to run the command in. + +Value can be substituded for data-env variable using the +`"DATA-VARS:path.to.variable.in.the.data.env"` syntax. + +For example `"DATA-VARS:mercurial.status.large.all"` + +The default should be the root of the data-env. + +This can be changed by variants (see next section). + +variants +-------- + +Variants allow for slight variation of a benchmark. For example, they allow to change the +input data, of to turn some feature on and off. + +Variants are organised within independant "dimensions". + +Each dimension has a set of values, each value can modify the benchmark. Each +dimension must have a default value. + +defining a variant +~~~~~~~~~~~~~~~~~~ + +Variants are defined under keys following this patterns : + + simple-command.variants.dimensions.<dimension-name>.<variant-key> + +The `dimension-name` and `variant-key` are used as identifier in the benchmark +result and for selecting variant at run time. + +variants variables +~~~~~~~~~~~~~~~~~~ + +default +~~~~~~~ + +Exactly one variant must have it set to `true` per dimension. + +When set and no variant have been explicitly selected for this dimension, this +variant will be used. + +extend-command +~~~~~~~~~~~~~~ + +A string to append to the commend when this variable is selected. + +cwd +~~~ + +overwrite the global `simple-command.cwd` variable. Behave the same.