Skip to content
Snippets Groups Projects
README.md 4.24 KiB
Newer Older
Poulpe
======

A tool for benchmarking software. It is in very early stages and probably won't work for you yet.

Concepts
See `docs/` for high-level documentation.

Installing for development
==========================

Poulpe itself requires Python 3.7+.

```console
$ python -m venv .venv
$ source .venv/bin/activate
> **Note** This guide will use `$DATA_DIR` as a shortcut for where your data
> (bin-envs, data-envs, results) will reside. A config file will be available
> shortly

Creating a bin-env
==================

You can use `poulpe bin-env-util` to create and query binary environments.

The `setup-one` command will run a script that takes care of setting up the
environment. There is currently only one script in this repository (which sets
up a Mercurial repository), feel free to adapt and create your own.
Simon Sapin's avatar
Simon Sapin committed

$ PYTHON_VERSION="/usr/bin/python3.6" MERCURIAL_VERSION="6.4rc0" \
     MERCURIAL_FLAVOR="rhg" poulpe bin-env-util setup-one \
     $DATA_DIR/bin-envs/mercurial-6.4rc0-rhg \
     suites/hg/bin-env-setup/mercurial.poulpe-setup.sh
```

You can make sure the environment is setup correctly:

```console
$ poulpe env-desc show $DATA_DIR/bin-envs/mercurial-6.4rc0-rhg/bin-env.poulpe
bin-env-vars:
  hg:
    changeset:
      branch = stable
      node = 05de4896508e8ec387b33eb30d8aab78d1c8e9e4
      phase = public
      rank = 50270
      tag =
    flavor = rhg
    version = 6.4rc0+hg6.ada9a0245fd7
  python:
    implementation = cpython
    version = 3.10.8
poulpe-environment:
  environment-type = binary
  format-version = 0
  setup-method = script
ready = True
```
Simon Sapin's avatar
Simon Sapin committed

You can query the environment for information:

```console
$ poulpe env-desc get \
      $DATA_DIR/bin-envs/mercurial-6.4rc0-rhg/bin-env.poulpe \
      bin-env-vars.hg.changeset.node
05de4896508e8ec387b33eb30d8aab78d1c8e9e4
```

You can run command your bin-env:
$ $DATA/bin-envs/mercurial-6.4rc0-rhg/bin-env.shell YOUR COMMAND
```

Including an interactive shell

```console
$ $DATA/bin-envs/mercurial-6.4rc0-rhg/bin-env.shell bash
```

Creating a data-env
===================

There are a few scripts to help setting up a data environment in
`suites/hg/setup-data`.  They are all specific to setting up an existing
Mercurial clone, feel free to create your own.

Example:

> **Warning**: the `delta-find` setup (included in `full`) can take a very long time.
Simon Sapin's avatar
Simon Sapin committed

```console
$ mkdir -p $DATA_DIR/data-envs/my-repo-dirstate-v2
$ hg clone /path/to/my-repo \
    $DATA_DIR/data-envs/my-repo-dirstate-v2/my-repo-2023-03-09-abcdefcd-main
$ ./suites/hg/setup-data/setup-full.sh my-repo-dirstate-v2 \
      $DATA_DIR/data-envs/my-repo-dirstate-v2 \
      my-repo-2023-03-09-abcdefcd-main
DELETING EXISTING DATA
create the base data-env
creating new file: "$DATA_DIR/data-envs/my-repo-dirstate-v2/data-env.poulpe"
register graph details
register the format details
setup for `hg status` benchmark
setup for `delta-find` benchmark
```

Running a benchmark
===================

You can use `poulpe run-util` to run a benchmark. There are many benchmarks in
`suites/hg/benchmarks` that are specific to benchmarking Mercurial. Feel free
to create your own.
$ poulpe run-util \
      $DATA_DIR/bin-envs/mercurial-6.4rc0-rhg \
      $DATA_DIR/data-envs/my-repo-dirstate-v2 \
      suites/hg/benchmarks/status.pbd \
      $DATA_DIR/results/status-6.4rc0-rhg.pbr
```

Run more from other bin-envs, then you can then compare your results:
$ poulpe diff-result \
      $DATA_DIR/results/status-6.4rc0-rust.pbr \
      $DATA_DIR/results/status-6.4rc0-rhg.pbr
0.0756 -> 0.0026: -0.0730 (0.03)
Simon Sapin's avatar
Simon Sapin committed
```

Simon Sapin's avatar
Simon Sapin committed

    $DATA_DIR/results/status-0.pbr \
    $DATA_DIR/results/status-1.pbr --full
bin-env-vars.hg.flavor: rust -> rhg
bin-env-vars.hg.version:  -> 6.4rc0+hg6.ada9a0245fd7
result.time.max: 0.0815292945 -> 0.00340343696
result.time.mean: 0.07588542467500001 -> 0.002615101740790962
result.time.median: 0.075647947 -> 0.0026036729600000005
result.time.min: 0.0728140685 -> 0.0022557289600000003
result.time.standard-deviation: 0.0018908053340991621 -> 0.00016997380518093058
run.duration: 3.0690555572509766 -> 2.6277692317962646
run.timestamp: 1678381483.3755798 -> 1678381351.3969703
```