Cannot clone Github HTTPS repositories
Created originally on Bitbucket by kankri
hg-git has stopped working with Github:
>hg clone git+https://github.com/jayKayEss/Flapper
destination directory: Flapper
** unknown exception encountered, please report by visiting
** http://mercurial.selenic.com/wiki/BugTracker
** Python 2.7.3 (default, Apr 10 2012, 23:24:47) [MSC v.1500 64 bit (AMD64)]
** Mercurial Distributed SCM (version 2.8.1)
** Extensions loaded: hggit
Traceback (most recent call last):
File "hg", line 42, in <module>
File "mercurial\dispatch.pyo", line 28, in run
File "mercurial\dispatch.pyo", line 69, in dispatch
File "mercurial\dispatch.pyo", line 133, in _runcatch
File "mercurial\dispatch.pyo", line 806, in _dispatch
File "mercurial\dispatch.pyo", line 585, in runcommand
File "mercurial\dispatch.pyo", line 897, in _runcommand
File "mercurial\dispatch.pyo", line 868, in checkargs
File "mercurial\dispatch.pyo", line 803, in <lambda>
File "mercurial\util.pyo", line 512, in check
File "mercurial\commands.pyo", line 1286, in clone
File "mercurial\hg.pyo", line 372, in clone
File "mercurial\localrepo.pyo", line 2431, in clone
File "hg-git\hggit\hgrepo.py", line 14, in pull
return self.githandler.fetch(remote.path, heads)
File "hggit\git_handler.py", line 202, in fetch
refs = self.fetch_pack(remote, heads)
File "hggit\git_handler.py", line 1019, in fetch_pack
ret = client.fetch_pack(path, determine_wants, graphwalker, f.write, progress.progress)
File "dulwich\client.pyo", line 1001, in fetch_pack
File "dulwich\client.pyo", line 924, in _discover_references
File "dulwich\client.pyo", line 902, in _http_request
dulwich.errors.NotGitRepository
It is unfortunately not easy to log what requests hg-git (or dulwich) is doing. However, it is extremely easy to see what Git is doing:
>set GIT_CURL_VERBOSE=1
>git clone https://github.com/jayKayEss/Flapper
...
> GET /jayKayEss/Flapper/info/refs?service=git-upload-pack HTTP/1.1
User-Agent: git/1.7.8.msysgit.0
Host: github.com
Accept: */*
Pragma: no-cache
< HTTP/1.1 200 OK
< Server: GitHub Babel 2.0
< Content-Type: application/x-git-upload-pack-advertisement
< Transfer-Encoding: chunked
< Expires: Fri, 01 Jan 1980 00:00:00 GMT
< Pragma: no-cache
< Cache-Control: no-cache, max-age=0, must-revalidate
< Vary: Accept-Encoding
...
Trying to get the same URL with curl:
>curl --head "https://github.com/jayKayEss/Flapper/info/refs?service=git-upload-pack"
HTTP/1.1 404 Not Found
Server: GitHub.com
Date: Wed, 16 Apr 2014 20:50:21 GMT
Content-Type: text/html; charset=utf-8
Status: 404 Not Found
Cache-Control: no-cache
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Content-Length: 226672
X-GitHub-Request-Id: 5870E8CF:2231:EA2F13:534EED0D
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
After some head scratching I tried better simulating the Git request:
>curl --head -A "git/1.7.8.msysgit.0" "https://github.com/jayKayEss/Flapper/info/refs?service=git-upload-pack"
HTTP/1.1 200 OK
Server: GitHub Babel 2.0
Content-Type: application/x-git-upload-pack-advertisement
Transfer-Encoding: chunked
Expires: Fri, 01 Jan 1980 00:00:00 GMT
Pragma: no-cache
Cache-Control: no-cache, max-age=0, must-revalidate
Vary: Accept-Encoding
So the User-Agent
header seems to make a difference!
Looking at the dulwich sources and simulating its request seems to confirm the theory:
>curl --head -A "dulwich/0.9.6" "https://github.com/jayKayEss/Flapper/info/refs?
service=git-upload-pack"
HTTP/1.1 404 Not Found
Server: GitHub.com
Date: Wed, 16 Apr 2014 20:57:32 GMT
Content-Type: text/html; charset=utf-8
Status: 404 Not Found
Cache-Control: no-cache
X-XSS-Protection: 1; mode=block
X-Frame-Options: deny
Content-Length: 226672
X-GitHub-Request-Id: 5870E8CF:0AB7:10EE616:534EEEBB
Strict-Transport-Security: max-age=31536000
X-Content-Type-Options: nosniff
This seems a bit brain-dead to me, but I guess this problem needs to be fixed in dulwich by setting a better User-Agent
header...
However:
-
It would be nice to have something like
GIT_CURL_VERBOSE
for hg-git (and dulwich) to debug issues like this.urllib2
supports logging requests with:http_logger = urllib2.HTTPHandler(debuglevel = 1) opener = urllib2.build_opener(http_logger)
-
It would be nice to be able to control settings like
User-Agent
from a configuration file.