Overall architecture discussion
Name
poulpe
Modules
Subcommands that can be executed separately under the same umbrella.
-
view
: allows to view a result file -
diff
: allows to diff results -
setup-bin-env
: sets up a binary environment according to abin-env.toml
file -
benchmark-collect
: collects all benchmark definitions -
setup-data-env
: prepares the data environment for a benchmark -
run-bench
: runs a benchmark in a given (pre-setup) environment
Concepts
Broad concepts with very little care for implementation details
Binary environment
Described by a poulpe-bin-env.toml
, is an abstraction over the instances of software environments (that is, everything that you can control about your environment that is not your hardware) to allow for reproducible or quicker setups. The file specifies some metadata yet to be defined and one of the types of environments:
- a shell binary environment which simply executes a shell script in a given directory
- a Nix (or other sandbox) environment
It will be used for all benchmarks in the entire run
command. We can imagine a future not far away where we have multi-run to compare between environments.
Data environment
Data in poulpe
is separated in tarballs. One may be tempted to use some kind of clever filesystem or object storage in machine that runs the benchmarks, but the filesystem itself is one of the variables that we need to account for and test.
Inside each tarball is a poulpe-data-env.toml
, which specifies the data source, metadata and transformations needed to setup for a benchmark. For now we will only have one type of source which is a repository, but other types might be specified later.
This ensures that the tarballs have a way of specifying metadata and benchmark traits.
Benchmark traits
Every data environment exposes traits to enable benchmarks to detect which environments are suitable for running. For example, testing hg update
between 10000 revisions doesn't make sense for a repository with less than 10000 revisions, so the benchmark will look if the trait is advertised in the poulpe-data-env.toml
file for the tarball in question.