# HG changeset patch # User Raphaël Gomès <rgomes@octobus.net> # Date 1678376357 -3600 # Thu Mar 09 16:39:17 2023 +0100 # Node ID efca24b4400e3e172dc8c67a4745c3cfdd3cb08f # Parent f36ade32cda4b06b44c70814dddec75389a93c0d readme: update README to actually be somewhat helpful diff --git a/README.md b/README.md --- a/README.md +++ b/README.md @@ -1,34 +1,132 @@ -Megafine +Poulpe +====== + +A tool for benchmarking software. It is in very early stages and probably won't work for you yet. + +Concepts ======== -A tool for benchmarking Mercurial. -The name is inspired from [hyperfine](https://github.com/sharkdp/hyperfine). +See `docs/` for high-level documentation. + +Installing for development +========================== + +```console +$ python -m venv .venv +$ source .env/bin/activate +$ pip install -e . +``` + +> **Note** This guide will use `$DATA_DIR` as a shortcut for where your data (bin-envs, data-envs, results) will reside. A config file will be available shortly + +Creating a bin-env +================== + +You can use `bin/bin-env-util` to create and query binary environments. +The `setup-one` command will run a script that takes care of setting up the environment. There is currently only one script in this repository (which sets up a Mercurial repository), feel free to adapt and create your own. + +Example: +```console +$ MERCURIAL_VERSION="6.4rc0" MERCURIAL_FLAVOR="rhg" ./bin/bin-env-util setup-one \ + $DATA_DIR/bin-envs/mercurial-6.4rc0-rhg \ + suites/hg/bin-env-setup/mercurial.poulpe-setup.sh +``` + +You can make sure the environment is setup correctly: + +```console +$ ./bin/env-desc show $DATA_DIR/bin-envs/mercurial-6.4rc0-rhg/bin-env.poulpe +bin-env-vars: + hg: + changeset: + branch = stable + node = 05de4896508e8ec387b33eb30d8aab78d1c8e9e4 + phase = public + rank = 50270 + tag = + flavor = rhg + version = 6.4rc0+hg6.ada9a0245fd7 + python: + implementation = cpython + version = 3.10.8 +poulpe-environment: + environment-type = binary + format-version = 0 + setup-method = script +ready = True +``` -Usage ------ +You can query the environment for information: + +```console +$ ./bin/env-desc get $DATA_DIR/bin-envs/mercurial-6.4rc0-rhg/bin-env.poulpe bin-env-vars.hg.changeset.node +05de4896508e8ec387b33eb30d8aab78d1c8e9e4 +``` + +You can also get an interactive shell within your bin-env (bash only): + +```console +$ source bin-envs/mercurial-6.4rc0-rhg/bin-env.shell +``` + +Creating a data-env +=================== + +XXX You need to add `/path/to/poulpe/bin` to your `$PATH`. This will be fixed shortly. + +There are a few scripts to help setting up a data environment in `suites/hg/setup-data`. +They are all specific to setting up an existing Mercurial clone, feel free to create your own. + +Example: + +> **Warning**: the `delta-find` setup (included in `full`) can take a very long time. -See `./run --help` and `./run <subcommand> --help` +```console +$ mkdir -p $DATA_DIR/data-envs/my-repo-dirstate-v2 +$ hg clone /path/to/my-repo $DATA_DIR/data-envs/my-repo-dirstate-v2/my-repo-2023-03-09-abcdefcd-main +[...] +$ ./suites/hg/setup-data/setup-full.sh my-repo-dirstate-v2 $DATA_DIR/data-envs/my-repo-dirstate-v2 my-repo-2023-03-09-abcdefcd-main +DELETING EXISTING DATA +create the base data-env +creating new file: "$DATA_DIR/data-envs/my-repo-dirstate-v2/data-env.poulpe" +register graph details +register the format details +setup for `hg status` benchmark +setup for `delta-find` benchmark +``` + +Running a benchmark +=================== + +You can use `bin/run-util` to run a benchmark. There are many benchmarks in `suites/hg/benchmarks` that are specific to benchmarking Mercurial. Feel free to create your own. + +Example: + +```console +$ mkdir results +$ ./bin/run-util $DATA_DIR/bin-envs/mercurial-6.4rc0-rhg $DATA_DIR/data-envs/my-repo-dirstate-v2 suites/hg/benchmarks/status.pbd $DATA_DIR/results/status-6.4rc0-rhg.pbr +``` + +Run more from other bin-envs, then you can then compare your results: -Example -------- - -``` -$ ./run time datasets/default.toml 'hg status' - Finished release [optimized] target(s) in 0.05s - Running `target/release/megafine time datasets/default.toml 'hg status'` -Running in format-bytes_2021-11-05_483a743760ac… 131 ms -Running in mercurial-devel_2021-11-05_1a420a13eeea… 159 ms +```console +$ ./bin/diff-result $DATA_DIR/results/status-6.4rc0-rust.pbr $DATA_DIR/results/status-6.4rc0-rhg.pbr +0.0756 -> 0.0026: -0.0730 (0.03) ``` - -Cleanup -------- +More info: -`hg clean -i tmp` is safe to run any time Megafine is not running. -It should usually do nothing, but might if a process was interrupted. - -`hg clean -i cache` is safe to run any time Megafine is not running, -at the cost of CPU time and/or network traffic -to rebuild parts as they become needed again. +```console +$ ./bin/diff-result $DATA_DIR/results/status-0.pbr $DATA_DIR/results/status-1.pbr --full +bin-env-vars.hg.flavor: rust -> rhg +bin-env-vars.hg.version: -> 6.4rc0+hg6.ada9a0245fd7 +result.time.max: 0.0815292945 -> 0.00340343696 +result.time.mean: 0.07588542467500001 -> 0.002615101740790962 +result.time.median: 0.075647947 -> 0.0026036729600000005 +result.time.min: 0.0728140685 -> 0.0022557289600000003 +result.time.standard-deviation: 0.0018908053340991621 -> 0.00016997380518093058 +run.duration: 3.0690555572509766 -> 2.6277692317962646 +run.timestamp: 1678381483.3755798 -> 1678381351.3969703 +```