Commit ffac09da7a1 by Matt Harbison

lfs: avoid quadratic performance in processing server responses

This is also adapted from the Facebook repo[1].  Unlike there, we were already
reading the download stream in chunks and immediately writing it to disk, so we
basically avoided the problem on download.  There shouldn't be a lot of data to
read on upload, but it's better to get rid of this pattern.

[1] https://github.com/facebookexperimental/eden/commit/82df66ffe97e21f3ee73dfec093c87500fc1f6a7

Differential Revision: https://phab.mercurial-scm.org/D7882
parent 0ee0a3f6a99
......@@ -503,7 +503,6 @@ class _gitlfsremote(object):
for k, v in headers:
request.add_header(pycompat.strurl(k), pycompat.strurl(v))
response = b''
try:
with contextlib.closing(self.urlopener.open(request)) as res:
contentlength = res.info().get(b"content-length")
......@@ -520,11 +519,14 @@ class _gitlfsremote(object):
# blobstore
localstore.download(oid, res, contentlength)
else:
blocks = []
while True:
data = res.read(1048576)
if not data:
break
response += data
blocks.append(data)
response = b"".join(blocks)
if response:
ui.debug(b'lfs %s response: %s' % (action, response))
except util.urlerr.httperror as ex:
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment