Enable clone benchmarks for all reference repositories
This is a bit tricky because there are several constraints: * we must cleanup after clone (in setup/teardown) to limit disk space usage * we must limit benchmark duration (avoid to run too much time on big repositories) * we must have more stable results for small repositories The whole algorithm is in asv.benchmark.benchmark_timing(). To ensure setup/teardown is called before/after each clone we *must* have number = 1 (so timer.timeit() will be called with 1 as argument). Then it's important to have a proper sample_time set, because it will control how long the benchmark will run. timeit() is called `repeat` (default 10) times unless the whole benchmark (= one benchmark function with one set of parameter) takes more than `repeat` * 1.3 * `sample_time` where `sample_time` has a default value of 0.1 This can produce very short times, for instance for a clone duration of 20s this will only run one clone even if repeat is set 10... So use sample_time = (max_time_we_want_the_benchmark_to_run / (repeat * 1.3)) to control maximum whole benchmark duration and to ensure timeit() will be called close to `repeat` times.
Loading
Please register or sign in to comment