Poulpe ====== A tool for benchmarking software. It is in very early stages and probably won't work for you yet. Concepts ======== See `docs/` for high-level documentation. Installing for development ========================== Poulpe itself requires Python 3.7+. ```console $ python -m venv .venv $ source .venv/bin/activate $ pip install -e . ``` > **Note** This guide will use `$DATA_DIR` as a shortcut for where your data > (bin-envs, data-envs, results) will reside. A config file will be available > shortly Creating a bin-env ================== You can use `poulpe bin-env-util` to create and query binary environments. The `setup-one` command will run a script that takes care of setting up the environment. There is currently only one script in this repository (which sets up a Mercurial repository), feel free to adapt and create your own. Example: ```console $ PYTHON_VERSION="/usr/bin/python3.6" MERCURIAL_VERSION="6.4rc0" \ MERCURIAL_FLAVOR="rhg" poulpe bin-env-util setup-one \ $DATA_DIR/bin-envs/mercurial-6.4rc0-rhg \ suites/hg/bin-env-setup/mercurial.poulpe-setup.sh ``` You can make sure the environment is setup correctly: ```console $ poulpe env-desc show $DATA_DIR/bin-envs/mercurial-6.4rc0-rhg/bin-env.poulpe bin-env-vars: hg: changeset: branch = stable node = 05de4896508e8ec387b33eb30d8aab78d1c8e9e4 phase = public rank = 50270 tag = flavor = rhg version = 6.4rc0+hg6.ada9a0245fd7 python: implementation = cpython version = 3.10.8 poulpe-environment: environment-type = binary format-version = 0 setup-method = script ready = True ``` You can query the environment for information: ```console $ poulpe env-desc get \ $DATA_DIR/bin-envs/mercurial-6.4rc0-rhg/bin-env.poulpe \ bin-env-vars.hg.changeset.node 05de4896508e8ec387b33eb30d8aab78d1c8e9e4 ``` You can run command your bin-env: ```console $ $DATA/bin-envs/mercurial-6.4rc0-rhg/bin-env.shell YOUR COMMAND ``` Including an interactive shell ```console $ $DATA/bin-envs/mercurial-6.4rc0-rhg/bin-env.shell bash ``` Creating a data-env =================== There are a few scripts to help setting up a data environment in `suites/hg/setup-data`. They are all specific to setting up an existing Mercurial clone, feel free to create your own. Example: > **Warning**: the `delta-find` setup (included in `full`) can take a very long time. ```console $ mkdir -p $DATA_DIR/data-envs/my-repo-dirstate-v2 $ hg clone /path/to/my-repo \ $DATA_DIR/data-envs/my-repo-dirstate-v2/my-repo-2023-03-09-abcdefcd-main [...] $ ./suites/hg/setup-data/setup-full.sh my-repo-dirstate-v2 \ $DATA_DIR/data-envs/my-repo-dirstate-v2 \ my-repo-2023-03-09-abcdefcd-main DELETING EXISTING DATA create the base data-env creating new file: "$DATA_DIR/data-envs/my-repo-dirstate-v2/data-env.poulpe" register graph details register the format details setup for `hg status` benchmark setup for `delta-find` benchmark ``` Running a benchmark =================== You can use `poulpe run-util` to run a benchmark. There are many benchmarks in `suites/hg/benchmarks` that are specific to benchmarking Mercurial. Feel free to create your own. Example: ```console $ mkdir $DATA_DIR/results $ poulpe run-util \ $DATA_DIR/bin-envs/mercurial-6.4rc0-rhg \ $DATA_DIR/data-envs/my-repo-dirstate-v2 \ suites/hg/benchmarks/status.pbd \ $DATA_DIR/results/status-6.4rc0-rhg.pbr ``` Run more from other bin-envs, then you can then compare your results: ```console $ poulpe diff-result \ $DATA_DIR/results/status-6.4rc0-rust.pbr \ $DATA_DIR/results/status-6.4rc0-rhg.pbr 0.0756 -> 0.0026: -0.0730 (0.03) ``` More info: ```console $ poulpe diff-result --full \ $DATA_DIR/results/status-0.pbr \ $DATA_DIR/results/status-1.pbr --full bin-env-vars.hg.flavor: rust -> rhg bin-env-vars.hg.version: -> 6.4rc0+hg6.ada9a0245fd7 result.time.max: 0.0815292945 -> 0.00340343696 result.time.mean: 0.07588542467500001 -> 0.002615101740790962 result.time.median: 0.075647947 -> 0.0026036729600000005 result.time.min: 0.0728140685 -> 0.0022557289600000003 result.time.standard-deviation: 0.0018908053340991621 -> 0.00016997380518093058 run.duration: 3.0690555572509766 -> 2.6277692317962646 run.timestamp: 1678381483.3755798 -> 1678381351.3969703 ```