Skip to content
Snippets Groups Projects
Name Last commit Last update
.repos_cache
bin
docs
python-libs/poulpe
scm-perf
suites/hg
tests
web-ui
.hg-format-source
.hgignore
.hgtags
.nvmrc
README.md
pyproject.toml
setup.cfg

Poulpe

A tool for benchmarking software. It is in very early stages and probably won't work for you yet.

Concepts

See docs/ for high-level documentation.

Installing for development

Poulpe itself requires Python 3.7+.

$ python -m venv .venv
$ source .venv/bin/activate
$ pip install -e .

Note This guide will use $DATA_DIR as a shortcut for where your data (bin-envs, data-envs, results) will reside. A config file will be available shortly

Creating a bin-env

You can use poulpe bin-env-util to create and query binary environments.

The setup-one command will run a script that takes care of setting up the environment. There is currently only one script in this repository (which sets up a Mercurial repository), feel free to adapt and create your own.

Example:

$ PYTHON_VERSION="/usr/bin/python3.6" MERCURIAL_VERSION="6.4rc0" \
     MERCURIAL_FLAVOR="rhg" poulpe bin-env-util setup-one \
     $DATA_DIR/bin-envs/mercurial-6.4rc0-rhg \
     suites/hg/bin-env-setup/mercurial.poulpe-setup.sh

You can make sure the environment is setup correctly:

$ poulpe env-desc show $DATA_DIR/bin-envs/mercurial-6.4rc0-rhg/bin-env.poulpe
bin-env-vars:
  hg:
    changeset:
      branch = stable
      node = 05de4896508e8ec387b33eb30d8aab78d1c8e9e4
      phase = public
      rank = 50270
      tag =
    flavor = rhg
    version = 6.4rc0+hg6.ada9a0245fd7
  python:
    implementation = cpython
    version = 3.10.8
poulpe-environment:
  environment-type = binary
  format-version = 0
  setup-method = script
ready = True

You can query the environment for information:

$ poulpe env-desc get \
      $DATA_DIR/bin-envs/mercurial-6.4rc0-rhg/bin-env.poulpe \
      bin-env-vars.hg.changeset.node
05de4896508e8ec387b33eb30d8aab78d1c8e9e4

You can run command your bin-env:

$ $DATA/bin-envs/mercurial-6.4rc0-rhg/bin-env.shell YOUR COMMAND

Including an interactive shell

$ $DATA/bin-envs/mercurial-6.4rc0-rhg/bin-env.shell bash

Creating a data-env

There are a few scripts to help setting up a data environment in suites/hg/setup-data. They are all specific to setting up an existing Mercurial clone, feel free to create your own.

Example:

Warning: the delta-find setup (included in full) can take a very long time.

$ mkdir -p $DATA_DIR/data-envs/my-repo-dirstate-v2
$ hg clone /path/to/my-repo \
    $DATA_DIR/data-envs/my-repo-dirstate-v2/my-repo-2023-03-09-abcdefcd-main
[...]
$ ./suites/hg/setup-data/setup-full.sh my-repo-dirstate-v2 \
      $DATA_DIR/data-envs/my-repo-dirstate-v2 \
      my-repo-2023-03-09-abcdefcd-main
DELETING EXISTING DATA
create the base data-env
creating new file: "$DATA_DIR/data-envs/my-repo-dirstate-v2/data-env.poulpe"
register graph details
register the format details
setup for `hg status` benchmark
setup for `delta-find` benchmark

Running a benchmark

You can use poulpe run-util to run a benchmark. There are many benchmarks in suites/hg/benchmarks that are specific to benchmarking Mercurial. Feel free to create your own.

Example:

$ mkdir $DATA_DIR/results
$ poulpe run-util \
      $DATA_DIR/bin-envs/mercurial-6.4rc0-rhg \
      $DATA_DIR/data-envs/my-repo-dirstate-v2 \
      suites/hg/benchmarks/status.pbd \
      $DATA_DIR/results/status-6.4rc0-rhg.pbr

Run more from other bin-envs, then you can then compare your results:

$ poulpe diff-result \
      $DATA_DIR/results/status-6.4rc0-rust.pbr \
      $DATA_DIR/results/status-6.4rc0-rhg.pbr
0.0756 -> 0.0026: -0.0730 (0.03)

More info:

$ poulpe diff-result --full \
    $DATA_DIR/results/status-0.pbr \
    $DATA_DIR/results/status-1.pbr --full
bin-env-vars.hg.flavor: rust -> rhg
bin-env-vars.hg.version:  -> 6.4rc0+hg6.ada9a0245fd7
result.time.max: 0.0815292945 -> 0.00340343696
result.time.mean: 0.07588542467500001 -> 0.002615101740790962
result.time.median: 0.075647947 -> 0.0026036729600000005
result.time.min: 0.0728140685 -> 0.0022557289600000003
result.time.standard-deviation: 0.0018908053340991621 -> 0.00016997380518093058
run.duration: 3.0690555572509766 -> 2.6277692317962646
run.timestamp: 1678381483.3755798 -> 1678381351.3969703