Skip to content
Snippets Groups Projects
Commit 33ba3b5df38e authored by Raphaël Gomès's avatar Raphaël Gomès
Browse files

Add functional .t test for all expected failures

Most failure cases have to be covered basically end-to-end,
so we might as well test the entire script.
Moreover, failures would be easy to pinpoint since we are
testing a single condition every time.
parent dda52a8b8613
No related branches found
No related tags found
No related merge requests found
========================================
Test for the `migrate_benchmarks` script
========================================
This test the invalid calls to the script.
Setup
-----
$ export PATH=$TESTDIR/../script/:${PATH}
$ export SOURCES_DIR=/tmp/test-migrate-benchmarks
$ export OUTPUT_DIR=/tmp/test-migrate-benchmarks-out
$ if [ -d "$SOURCES_DIR" ]; then rm -r $SOURCES_DIR 2> /dev/null; fi
$ if [ -d "$OUTPUT_DIR" ]; then rm -r $OUTPUT_DIR 2> /dev/null; fi
$ mkdir -p "$SOURCES_DIR/results/"
$ mkdir -p $OUTPUT_DIR
Missing sources parameter
-------------------------
$ migrate_benchmarks.py
usage: migrate_benchmarks.py [-h] -s SOURCES [SOURCES ...] -b BENCHMARK_FILE
-o OUTPUT_DIR [-v]
migrate_benchmarks.py: error: argument -s/--sources is required
[2]
Missing benchmark_file parameter
----------------------------
$ migrate_benchmarks.py -s $SOURCES_DIR/test-1.json -o $OUTPUT_DIR
usage: migrate_benchmarks.py [-h] -s SOURCES [SOURCES ...] -b BENCHMARK_FILE
-o OUTPUT_DIR [-v]
migrate_benchmarks.py: error: argument -b/--benchmark_file is required
[2]
Missing output_dir parameter
----------------------------
$ migrate_benchmarks.py -s $SOURCES_DIR/test-1.json -b $SOURCES_DIR/benchmarks.json
usage: migrate_benchmarks.py [-h] -s SOURCES [SOURCES ...] -b BENCHMARK_FILE
-o OUTPUT_DIR [-v]
migrate_benchmarks.py: error: argument -o/--output_dir is required
[2]
Test not in benchmarks.json
---------------------------
$ cat << EOF > $SOURCES_DIR/results/test-1.json
> {"results": {"testname": {}}}
> EOF
$ cat << EOF > $SOURCES_DIR/benchmarks.json
> {"version": 1}
> EOF
$ migrate_benchmarks.py -s $SOURCES_DIR/results/test-1.json -b $SOURCES_DIR/benchmarks.json -o $OUTPUT_DIR
---------------------
/tmp/test-migrate-benchmarks/results/test-1.json
---------------------
Test testname does not exist in benchmarks.json, skipping
Warning: This file was left untouched.
Ignores machine.json
--------------------
$ rm $SOURCES_DIR/results/*
$ touch $SOURCES_DIR/results/machine.json
$ cat << EOF > $SOURCES_DIR/benchmarks.json
> {"version": 1}
> EOF
$ migrate_benchmarks.py -s $SOURCES_DIR/results/ -b $SOURCES_DIR/benchmarks.json -o $OUTPUT_DIR
---------------------
/tmp/test-migrate-benchmarks/results/machine.json
---------------------
Ignoring non-test file
Test does not have parameters
-----------------------------
$ cat << EOF > $SOURCES_DIR/results/test-1.json
> {
> "results": {
> "testname": {
> "params": [
> ["whatever"]
> ]
> }
> }
> }
> EOF
$ cat << EOF > $SOURCES_DIR/benchmarks.json
> {"version": 1, "testname": {"version": "myhash", "param_names": [], "params": []}}
> EOF
$ migrate_benchmarks.py -s $SOURCES_DIR/results/test-1.json -b $SOURCES_DIR/benchmarks.json -o $OUTPUT_DIR
Test testname does not have any parameters, aborting.
[1]
Test already new format
-----------------------
$ cat << EOF > $SOURCES_DIR/results/test-1.json
> {
> "results": {
> "testname": {
> "params": [
> [
> "'mercurial-2018-08-01'",
> "'mozilla-central-2018-08-01'",
> "'netbeans-2018-08-01'",
> "'pypy-2018-08-01'"
> ],
> ["'zlib'"],
> ["True"],
> ["True"],
> ["True"],
> ["True"],
> ["True", "False"]
> ]
> }
> }
> }
> EOF
$ cat << EOF > $SOURCES_DIR/benchmarks.json
> {
> "version": 1,
> "testname": {
> "version": "myhash",
> "param_names": [
> "repo",
> "repo-format-compression",
> "repo-format-dotencode",
> "repo-format-fncache",
> "repo-format-generaldelta",
> "repo-format-plain-cl-delta",
> "repo-format-sparserevlog"
> ],
> "params": [
> [
> "'mercurial-2018-08-01'",
> "'mozilla-central-2018-08-01'",
> "'netbeans-2018-08-01'",
> "'pypy-2018-08-01'"
> ],
> ["'zlib'"],
> ["True"],
> ["True"],
> ["True"],
> ["True"],
> ["True", "False"]
> ]
> }
> }
> EOF
$ migrate_benchmarks.py -v -s $SOURCES_DIR/results/test-1.json -b $SOURCES_DIR/benchmarks.json -o $OUTPUT_DIR
---------------------
/tmp/test-migrate-benchmarks/results/test-1.json
---------------------
Test testname is already in the new format, skipping
Other already new format
------------------------
$ cat << EOF > $SOURCES_DIR/results/test-1.json
> {
> "results": {
> "testname": {
> "params": [
> [
> "'mercurial-2018-08-01'",
> "'mozilla-central-2018-08-01'",
> "'netbeans-2018-08-01'",
> "'pypy-2018-08-01'",
> "'mozilla-try-2019-03-01'"
> ],
> ["'zlib'"],
> ["True"],
> ["True"],
> ["True"],
> ["True"],
> ["True", "False"]
> ]
> }
> }
> }
> EOF
$ cat << EOF > $SOURCES_DIR/benchmarks.json
> {
> "version": 1,
> "testname": {
> "version": "myhash",
> "param_names": [
> "repo",
> "repo-format-compression",
> "repo-format-dotencode",
> "repo-format-fncache",
> "repo-format-generaldelta",
> "repo-format-plain-cl-delta",
> "repo-format-sparserevlog"
> ],
> "params": [
> [
> "'mercurial-2018-08-01'",
> "'mozilla-central-2018-08-01'",
> "'netbeans-2018-08-01'",
> "'pypy-2018-08-01'",
> "'mozilla-try-2019-03-01'"
> ],
> ["'zlib'"],
> ["True"],
> ["True"],
> ["True"],
> ["True"],
> ["True", "False"]
> ]
> }
> }
> EOF
$ migrate_benchmarks.py -v -s $SOURCES_DIR/results/test-1.json -b $SOURCES_DIR/benchmarks.json -o $OUTPUT_DIR
---------------------
/tmp/test-migrate-benchmarks/results/test-1.json
---------------------
Test testname is already in the new format, skipping
Irregular repo parameter
------------------------
$ cat << EOF > $SOURCES_DIR/results/test-1.json
> {
> "benchmark_version": {"testname": "otherhash"},
> "results": {
> "testname": {
> "params": [
> ["invalid"]
> ],
> "result": []
> }
> }
> }
> EOF
$ cat << EOF > $SOURCES_DIR/benchmarks.json
> {"version": 1, "testname": {
> "version": "myhash",
> "param_names": ["repo"],
> "params": ["doesn't matter"]}}
> EOF
$ migrate_benchmarks.py -s $SOURCES_DIR/results/test-1.json -b $SOURCES_DIR/benchmarks.json -o $OUTPUT_DIR
---------------------
/tmp/test-migrate-benchmarks/results/test-1.json
---------------------
Test version does not match benchmark.json version, removing
Found irregular `repos` parameter: frozenset([u'invalid']). Aborting.
[1]
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment