Skip to content
Snippets Groups Projects
Commit f3b7acb41a35 authored by Pierre-Yves David's avatar Pierre-Yves David :octopus:
Browse files

upgrade-reference: preserve existing `repo-prefix` if any

Before this change, running upgrade-reference could change the repository-prefix
when new version were adding new repository format key. This change is not
necessary (see inline comment) and has various anoying side effect (eg: requires
a upload/download unchanged file again).
parent 5dbdd3198784
No related branches found
No related tags found
No related merge requests found
...@@ -54,8 +54,8 @@ ...@@ -54,8 +54,8 @@
repo_id = benchrepo_data["reference-repo"]["id"] repo_id = benchrepo_data["reference-repo"]["id"]
hadprefix = True hadprefix = True
old_repo_prefix = benchrepo_data["reference-repo"].get("repo-prefix") repo_prefix = benchrepo_data["reference-repo"].get("repo-prefix")
if old_repo_prefix is None: if repo_prefix is None:
# Support old benchrepo files without repo-prefix # Support old benchrepo files without repo-prefix
hadprefix = False hadprefix = False
old_repo_prefix = repo_id old_repo_prefix = repo_id
...@@ -59,9 +59,11 @@ ...@@ -59,9 +59,11 @@
# Support old benchrepo files without repo-prefix # Support old benchrepo files without repo-prefix
hadprefix = False hadprefix = False
old_repo_prefix = repo_id old_repo_prefix = repo_id
else:
old_repo_prefix = repo_prefix
# Compute the new stats # Compute the new stats
print("Gathering the new benchrepo data.") print("Gathering the new benchrepo data.")
repo_id = benchrepo_data["reference-repo"]["id"] repo_id = benchrepo_data["reference-repo"]["id"]
format_data = get_format_data(repository_path) format_data = get_format_data(repository_path)
...@@ -62,8 +64,28 @@ ...@@ -62,8 +64,28 @@
# Compute the new stats # Compute the new stats
print("Gathering the new benchrepo data.") print("Gathering the new benchrepo data.")
repo_id = benchrepo_data["reference-repo"]["id"] repo_id = benchrepo_data["reference-repo"]["id"]
format_data = get_format_data(repository_path) format_data = get_format_data(repository_path)
reference_id = get_reference_hash(repo_id, format_data) # if we already have a prefix, we keep it for simplicity
#
# This is fine because the goal of the repository prefix is to be unique.
# This is why we compute is from the format-data, so that two different
# reference have different IDs.
#
# Since we aims for uniqueness this is fine to keep the same repository
# prefix. No new reference repositories should collide with this one.
# The only reason the format-data would changes for this reference is that
# a new format key is added. If that key is new, there cannot exist a
# repository with a variation of this key. In addition, repository created
# from that point will have the new key and cannot result in the same
# reference ID.
#
# (however, recreating the same repository from crash could result in two
# identical reference using different keys)
if not hadprefix:
reference_id = get_reference_hash(repo_id, format_data)
repo_prefix = "{0}-{1}".format(repo_id, reference_id)
# Update the associated repo-prefix
new_benchrepo["reference-repo"]["repo-prefix"] = repo_prefix
...@@ -69,5 +91,4 @@ ...@@ -69,5 +91,4 @@
repo_prefix = "{0}-{1}".format(repo_id, reference_id)
repo_details = "{0}.benchrepo".format(repo_prefix) repo_details = "{0}.benchrepo".format(repo_prefix)
repo_tarball = "{0}-reference.tar".format(repo_prefix) repo_tarball = "{0}-reference.tar".format(repo_prefix)
repo_main_with_hash = "{0}-reference".format(repo_prefix) repo_main_with_hash = "{0}-reference".format(repo_prefix)
...@@ -77,8 +98,6 @@ ...@@ -77,8 +98,6 @@
# drop legacy "repo-hash" attribute # drop legacy "repo-hash" attribute
new_benchrepo["reference-repo"].pop("repo-hash", None) new_benchrepo["reference-repo"].pop("repo-hash", None)
# Update the associated repo-prefix
new_benchrepo["reference-repo"]["repo-prefix"] = repo_prefix
# And write it # And write it
with open(benchrepo_path, "w") as new_benchrepo_file: with open(benchrepo_path, "w") as new_benchrepo_file:
......
...@@ -155,8 +155,50 @@ ...@@ -155,8 +155,50 @@
$ tar tf my-source-repo-*-partial-same.tar --wildcards '*/.hg/requires' $ tar tf my-source-repo-*-partial-same.tar --wildcards '*/.hg/requires'
my-source-repo-39241d24-partial-same/.hg/requires my-source-repo-39241d24-partial-same/.hg/requires
Check that repo-prefix is preserved
-----------------------------------
update the repo-prefix and associate tar
$ benchrepofile=`tar tf my-source-repo-*-reference.tar | grep '.benchrepo$'`
$ tar xf my-source-repo-*-reference.tar --touch
$ old_prefix=`grep repo-prefix $benchrepofile | grep -oE '[^ ]+$'`
$ echo $old_prefix
my-source-repo-39241d24
$ sed "s/repo-prefix: .*/repo-prefix: my-source-repo-1a2b3c4d/" $benchrepofile > my-source-repo-1a2b3c4d.benchrepo
$ mv my-source-repo-*-reference my-source-repo-1a2b3c4d-reference
$ rm my-source-repo-*-reference.tar
$ tar cf my-source-repo-1a2b3c4d-reference.tar my-source-repo-1a2b3c4d-reference my-source-repo-1a2b3c4d.benchrepo
$ rm my-source-repo-*-partial-same.tar
$ tar cf my-source-repo-1a2b3c4d-partial-same.tar my-source-repo-1a2b3c4d-reference
$ ls -1 my-source-repo-*-reference.tar my-source-repo-*-partial-*.tar
my-source-repo-1a2b3c4d-partial-same.tar
my-source-repo-1a2b3c4d-reference.tar
The upgrade script should not touch the repo-prefix
$ tar xf my-source-repo-1a2b3c4d-reference.tar --to-stdout my-source-repo-1a2b3c4d.benchrepo | grep repo-prefix
repo-prefix: my-source-repo-1a2b3c4d
$ upgrade-reference my-source-repo-1a2b3c4d-reference.tar ../p-config.yaml
Extracting the tar.
Reading the original benchrepo file.
Gathering the new benchrepo data.
Archiving the new benchrepo file.
Writing the new reference tar as my-source-repo-1a2b3c4d-reference.tar
Skipping copy of my-source-repo-1a2b3c4d-partial-missing-last-10.tar, not present in reference archive directory
Skipping copy of my-source-repo-1a2b3c4d-partial-same.tar, it already has the right file name
Fixing directory name inside my-source-repo-1a2b3c4d-partial-same.tar
Removing the temporary directory.
$ tar xf my-source-repo-1a2b3c4d-reference.tar --to-stdout my-source-repo-1a2b3c4d.benchrepo | grep repo-prefix
repo-prefix: my-source-repo-1a2b3c4d
We should still have the same unique reference tarball
$ ls my-source-repo-*-reference.tar
my-source-repo-1a2b3c4d-reference.tar
Check that missing format entry get updated Check that missing format entry get updated
------------------------------------------- -------------------------------------------
remove a format entry from the file remove a format entry from the file
...@@ -158,9 +200,9 @@ ...@@ -158,9 +200,9 @@
Check that missing format entry get updated Check that missing format entry get updated
------------------------------------------- -------------------------------------------
remove a format entry from the file remove a format entry from the file
$ benchrepofile=`tar tf my-source-repo-*-reference.tar | grep '.benchrepo$'` $ benchrepofile=`tar tf my-source-repo-1a2b3c4d-reference.tar | grep '.benchrepo'`
$ tar xf my-source-repo-*-reference.tar $benchrepofile $ tar xf my-source-repo-1a2b3c4d-reference.tar $benchrepofile
$ grep -v "fncache: true" $benchrepofile > tmp $ grep -v "fncache: true" $benchrepofile > tmp
$ mv tmp $benchrepofile $ mv tmp $benchrepofile
...@@ -165,7 +207,7 @@ ...@@ -165,7 +207,7 @@
$ grep -v "fncache: true" $benchrepofile > tmp $ grep -v "fncache: true" $benchrepofile > tmp
$ mv tmp $benchrepofile $ mv tmp $benchrepofile
$ tar --delete --file my-source-repo-*-reference.tar $benchrepofile $ tar --delete --file my-source-repo-1a2b3c4d-reference.tar $benchrepofile
$ tar --append --file my-source-repo-*-reference.tar $benchrepofile $ tar --append --file my-source-repo-1a2b3c4d-reference.tar $benchrepofile
The upgrade script should add the missing entry again The upgrade script should add the missing entry again
...@@ -169,5 +211,5 @@ ...@@ -169,5 +211,5 @@
The upgrade script should add the missing entry again The upgrade script should add the missing entry again
$ tar xf my-source-repo-*-reference.tar --to-stdout $benchrepofile | grep fnache $ tar xf my-source-repo-1a2b3c4d-reference.tar --to-stdout $benchrepofile | grep fnache
[1] [1]
...@@ -173,6 +215,6 @@ ...@@ -173,6 +215,6 @@
[1] [1]
$ upgrade-reference my-source-repo-*-reference.tar ../p-config.yaml $ upgrade-reference my-source-repo-1a2b3c4d-reference.tar ../p-config.yaml
Extracting the tar. Extracting the tar.
Reading the original benchrepo file. Reading the original benchrepo file.
Gathering the new benchrepo data. Gathering the new benchrepo data.
Archiving the new benchrepo file. Archiving the new benchrepo file.
...@@ -175,9 +217,9 @@ ...@@ -175,9 +217,9 @@
Extracting the tar. Extracting the tar.
Reading the original benchrepo file. Reading the original benchrepo file.
Gathering the new benchrepo data. Gathering the new benchrepo data.
Archiving the new benchrepo file. Archiving the new benchrepo file.
Writing the new reference tar as my-source-repo-*-reference.tar (glob) Writing the new reference tar as my-source-repo-1a2b3c4d-reference.tar
Skipping copy of my-source-repo-*-partial-missing-last-10.tar, not present in reference archive directory (glob) Skipping copy of my-source-repo-1a2b3c4d-partial-missing-last-10.tar, not present in reference archive directory
Skipping copy of my-source-repo-*-partial-same.tar, it already has the right file name (glob) Skipping copy of my-source-repo-1a2b3c4d-partial-same.tar, it already has the right file name
Directory name inside my-source-repo-*-partial-same.tar is correct (glob) Directory name inside my-source-repo-1a2b3c4d-partial-same.tar is correct
Removing the temporary directory. Removing the temporary directory.
...@@ -183,5 +225,5 @@ ...@@ -183,5 +225,5 @@
Removing the temporary directory. Removing the temporary directory.
$ tar xf my-source-repo-*-reference.tar --to-stdout $benchrepofile $ tar xf my-source-repo-1a2b3c4d-reference.tar --to-stdout $benchrepofile
reference-repo: reference-repo:
format-info: format-info:
compression: zlib compression: zlib
...@@ -192,5 +234,5 @@ ...@@ -192,5 +234,5 @@
plain-cl-delta: true plain-cl-delta: true
sparserevlog: true sparserevlog: true
id: my-source-repo id: my-source-repo
repo-prefix: my-source-repo-* (glob) repo-prefix: my-source-repo-1a2b3c4d
source: some-dir/my-source-repo source: some-dir/my-source-repo
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment