Poulpe
A tool for benchmarking software. It is in very early stages and probably won't work for you yet.
Concepts
See docs/
for high-level documentation.
Installing for development
Poulpe itself requires Python 3.7+.
$ python -m venv .venv
$ source .venv/bin/activate
$ pip install -e .
Note This guide will use
$DATA_DIR
as a shortcut for where your data (bin-envs, data-envs, results) will reside. A config file will be available shortly
Creating a bin-env
You can use poulpe bin-env-util
to create and query binary environments.
The setup-one
command will run a script that takes care of setting up the
environment. There is currently only one script in this repository (which sets
up a Mercurial repository), feel free to adapt and create your own.
Example:
$ PYTHON_VERSION="/usr/bin/python3.6" MERCURIAL_VERSION="6.4rc0" \
MERCURIAL_FLAVOR="rhg" poulpe bin-env-util setup-one \
$DATA_DIR/bin-envs/mercurial-6.4rc0-rhg \
suites/hg/bin-env-setup/mercurial.poulpe-setup.sh
You can make sure the environment is setup correctly:
$ poulpe env-desc show $DATA_DIR/bin-envs/mercurial-6.4rc0-rhg/bin-env.poulpe
bin-env-vars:
hg:
changeset:
branch = stable
node = 05de4896508e8ec387b33eb30d8aab78d1c8e9e4
phase = public
rank = 50270
tag =
flavor = rhg
version = 6.4rc0+hg6.ada9a0245fd7
python:
implementation = cpython
version = 3.10.8
poulpe-environment:
environment-type = binary
format-version = 0
setup-method = script
ready = True
You can query the environment for information:
$ poulpe env-desc get \
$DATA_DIR/bin-envs/mercurial-6.4rc0-rhg/bin-env.poulpe \
bin-env-vars.hg.changeset.node
05de4896508e8ec387b33eb30d8aab78d1c8e9e4
You can run command your bin-env:
$ $DATA/bin-envs/mercurial-6.4rc0-rhg/bin-env.shell YOUR COMMAND
Including an interactive shell
$ $DATA/bin-envs/mercurial-6.4rc0-rhg/bin-env.shell bash
Creating a data-env
There are a few scripts to help setting up a data environment in
suites/hg/setup-data
. They are all specific to setting up an existing
Mercurial clone, feel free to create your own.
Example:
Warning: the
delta-find
setup (included infull
) can take a very long time.
$ mkdir -p $DATA_DIR/data-envs/my-repo-dirstate-v2
$ hg clone /path/to/my-repo \
$DATA_DIR/data-envs/my-repo-dirstate-v2/my-repo-2023-03-09-abcdefcd-main
[...]
$ ./suites/hg/setup-data/setup-full.sh my-repo-dirstate-v2 \
$DATA_DIR/data-envs/my-repo-dirstate-v2 \
my-repo-2023-03-09-abcdefcd-main
DELETING EXISTING DATA
create the base data-env
creating new file: "$DATA_DIR/data-envs/my-repo-dirstate-v2/data-env.poulpe"
register graph details
register the format details
setup for `hg status` benchmark
setup for `delta-find` benchmark
Running a benchmark
You can use poulpe run-util
to run a benchmark. There are many benchmarks in
suites/hg/benchmarks
that are specific to benchmarking Mercurial. Feel free
to create your own.
Example:
$ mkdir $DATA_DIR/results
$ poulpe run-util \
$DATA_DIR/bin-envs/mercurial-6.4rc0-rhg \
$DATA_DIR/data-envs/my-repo-dirstate-v2 \
suites/hg/benchmarks/status.pbd \
$DATA_DIR/results/status-6.4rc0-rhg.pbr
Run more from other bin-envs, then you can then compare your results:
$ poulpe diff-result \
$DATA_DIR/results/status-6.4rc0-rust.pbr \
$DATA_DIR/results/status-6.4rc0-rhg.pbr
0.0756 -> 0.0026: -0.0730 (0.03)
More info:
$ poulpe diff-result --full \
$DATA_DIR/results/status-0.pbr \
$DATA_DIR/results/status-1.pbr --full
bin-env-vars.hg.flavor: rust -> rhg
bin-env-vars.hg.version: -> 6.4rc0+hg6.ada9a0245fd7
result.time.max: 0.0815292945 -> 0.00340343696
result.time.mean: 0.07588542467500001 -> 0.002615101740790962
result.time.median: 0.075647947 -> 0.0026036729600000005
result.time.min: 0.0728140685 -> 0.0022557289600000003
result.time.standard-deviation: 0.0018908053340991621 -> 0.00016997380518093058
run.duration: 3.0690555572509766 -> 2.6277692317962646
run.timestamp: 1678381483.3755798 -> 1678381351.3969703