Read about our upcoming Code of Conduct on this issue

manage_large_binaries_with_git_lfs.md 8.5 KB
Newer Older
Marin Jankovski's avatar
Marin Jankovski committed
1
2
# Git LFS

3
4
5
Managing large files such as audio, video and graphics files has always been one
of the shortcomings of Git. The general recommendation is to not have Git repositories
larger than 1GB to preserve performance.
Marin Jankovski's avatar
Marin Jankovski committed
6

James Ramsay's avatar
James Ramsay committed
7
8
9
10
11
![Git LFS tracking status](img/lfs-icon.png)

An LFS icon is shown on files tracked by Git LFS to denote if a file is stored
as a blob or as an LFS pointer.

Marin Jankovski's avatar
Marin Jankovski committed
12
13
## How it works

14
15
16
Git LFS client talks with the GitLab server over HTTPS. It uses HTTP Basic Authentication
to authorize client requests. Once the request is authorized, Git LFS client receives
instructions from where to fetch or where to push the large file.
Marin Jankovski's avatar
Marin Jankovski committed
17

18
## GitLab server configuration
Marin Jankovski's avatar
Marin Jankovski committed
19

20
Documentation for GitLab instance administrators is under [LFS administration doc](lfs_administration.md).
Marin Jankovski's avatar
Marin Jankovski committed
21

22
## Requirements
Marin Jankovski's avatar
Marin Jankovski committed
23

24
* Git LFS is supported in GitLab starting with version 8.2
25
* Git LFS must be enabled under project settings
26
* [Git LFS client](https://git-lfs.github.com) version 1.0.1 and up
Marin Jankovski's avatar
Marin Jankovski committed
27

Marin Jankovski's avatar
Marin Jankovski committed
28
## Known limitations
Marin Jankovski's avatar
Marin Jankovski committed
29

30
31
* Git LFS v1 original API is not supported since it was deprecated early in LFS
  development
Marin Jankovski's avatar
Marin Jankovski committed
32
* When SSH is set as a remote, Git LFS objects still go through HTTPS
33
* Any Git LFS request will ask for HTTPS credentials to be provided so a good Git
34
35
  credentials store is recommended
* Git LFS always assumes HTTPS so if you have GitLab server on HTTP you will have
36
  to add the URL to Git config manually (see [troubleshooting](#troubleshooting))
37
38
39
40
  
>**Note**: With 8.12 GitLab added LFS support to SSH. The Git LFS communication
 still goes over HTTP, but now the SSH client passes the correct credentials
 to the Git LFS client, so no action is required by the user.
Marin Jankovski's avatar
Marin Jankovski committed
41
42
43

## Using Git LFS

44
45
46
Lets take a look at the workflow when you need to check large files into your Git
repository with Git LFS. For example, if you want to upload a very large file and
check it into your Git repository:
Marin Jankovski's avatar
Marin Jankovski committed
47
48
49

```bash
git clone git@gitlab.example.com:group/project.git
50
git lfs install                       # initialize the Git LFS project
Marin Jankovski's avatar
Marin Jankovski committed
51
git lfs track "*.iso"                 # select the file extensions that you want to treat as large files
Marin Jankovski's avatar
Marin Jankovski committed
52
53
```

54
55
Once a certain file extension is marked for tracking as a LFS object you can use
Git as usual without having to redo the command to track a file with the same extension:
Marin Jankovski's avatar
Marin Jankovski committed
56
57

```bash
Marin Jankovski's avatar
Marin Jankovski committed
58
cp ~/tmp/debian.iso ./                # copy a large file into the current directory
Marin Jankovski's avatar
Marin Jankovski committed
59
git add .                             # add the large file to the project
Marin Jankovski's avatar
Marin Jankovski committed
60
61
62
63
git commit -am "Added Debian iso"     # commit the file meta data
git push origin master                # sync the git repo and large file to the GitLab server
```

64
>**Note**: Make sure that `.gitattributes` is tracked by git. Otherwise Git
65
66
67
68
69
 LFS will not be working properly for people cloning the project.
 ```bash
 git add .gitattributes
 ```

70
71
72
73
Cloning the repository works the same as before. Git automatically detects the
LFS-tracked files and clones them via HTTP. If you performed the git clone
command with a SSH URL, you have to enter your GitLab credentials for HTTP
authentication.
Marin Jankovski's avatar
Marin Jankovski committed
74
75
76
77
78

```bash
git clone git@gitlab.example.com:group/project.git
```

79
80
If you already cloned the repository and you want to get the latest LFS object
that are on the remote repository, eg. from branch `master`:
81
82
83
84

```bash
git lfs fetch master
```
Marin Jankovski's avatar
Marin Jankovski committed
85

86
87
88
89
90
91
92
93
94
95
96
97
98
99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146
147
148
149
150
151
## File Locking

The first thing to do before using File Locking is to tell Git LFS which
kind of files are lockable. The following command will store PNG files
in LFS and flag them as lockable:

```bash
git lfs track "*.png" --lockable
```

After executing the above command a file named `.gitattributes` will be
created or updated with the following content:

```bash
*.png filter=lfs diff=lfs merge=lfs -text lockable
```

You can also register a file type as lockable without using LFS
(In order to be able to lock/unlock a file you need a remote server that implements the  LFS File Locking API),
in order to do that you can edit the `.gitattributes` file manually:

```bash
*.pdf lockable
```

After a file type has been registered as lockable, Git LFS will make
them readonly on the file system automatically. This means you will
need to lock the file before editing it.

### Managing Locked Files

Once you're ready to edit your file you need to lock it first:

```bash
git lfs lock images/banner.png
Locked images/banner.png
```

This will register the file as locked in your name on the server:

```bash
git lfs locks
images/banner.png  joe   ID:123
```

Once you have pushed your changes, you can unlock the file so others can
also edit it:

```bash
git lfs unlock images/banner.png
```

You can also unlock by id:

```bash
git lfs unlock --id=123
```

If for some reason you need to unlock a file that was not locked by you,
you can use the `--force` flag as long as you have a `master` access on
the project:

```bash
git lfs unlock --id=123 --force
```

Marin Jankovski's avatar
Marin Jankovski committed
152
## Troubleshooting
Marin Jankovski's avatar
Marin Jankovski committed
153
154
155

### error: Repository or object not found

Marin Jankovski's avatar
Marin Jankovski committed
156
There are a couple of reasons why this error can occur:
Marin Jankovski's avatar
Marin Jankovski committed
157

158
* You don't have permissions to access certain LFS object
Marin Jankovski's avatar
Marin Jankovski committed
159

160
Check if you have permissions to push to the project or fetch from the project.
Marin Jankovski's avatar
Marin Jankovski committed
161

162
163
* Project is not allowed to access the LFS object

164
165
LFS object you are trying to push to the project or fetch from the project is not
available to the project anymore. Probably the object was removed from the server.
Marin Jankovski's avatar
Marin Jankovski committed
166

167
* Local git repository is using deprecated LFS API
Marin Jankovski's avatar
Marin Jankovski committed
168

169
### Invalid status for `<url>` : 501
Marin Jankovski's avatar
Marin Jankovski committed
170

171
172
173
174
175
176
177
178
179
Git LFS will log the failures into a log file.
To view this log file, while in project directory:

```bash
git lfs logs last
```

If the status `error 501` is shown, it is because:

180
181
182
* Git LFS is not enabled in project settings. Check your project settings and
  enable Git LFS.

183
184
185
186
* Git LFS support is not enabled on the GitLab server. Check with your GitLab
  administrator why Git LFS is not enabled on the server. See
  [LFS administration documentation](lfs_administration.md) for instructions
  on how to enable LFS support.
187

188
189
190
191
192
* Git LFS client version is not supported by GitLab server. Check your Git LFS
  version with `git lfs version`. Check the Git config of the project for traces
  of deprecated API with `git lfs -l`. If `batch = false` is set in the config,
  remove the line and try to update your Git LFS client. Only version 1.0.1 and
  newer are supported.
Marin Jankovski's avatar
Marin Jankovski committed
193
194
195

### getsockopt: connection refused

196
197
198
199
If you push a LFS object to a project and you receive an error similar to:
`Post <URL>/info/lfs/objects/batch: dial tcp IP: getsockopt: connection refused`,
the LFS client is trying to reach GitLab through HTTPS. However, your GitLab
instance is being served on HTTP.
Marin Jankovski's avatar
Marin Jankovski committed
200

201
202
This behaviour is caused by Git LFS using HTTPS connections by default when a
`lfsurl` is not set in the Git config.
Marin Jankovski's avatar
Marin Jankovski committed
203

Marin Jankovski's avatar
Marin Jankovski committed
204
To prevent this from happening, set the lfs url in project Git config:
Marin Jankovski's avatar
Marin Jankovski committed
205
206

```bash
207
git config --add lfs.url "http://gitlab.example.com/group/project.git/info/lfs"
Marin Jankovski's avatar
Marin Jankovski committed
208
209
210
211
```

### Credentials are always required when pushing an object

212
213
214
215
>**Note**: With 8.12 GitLab added LFS support to SSH. The Git LFS communication
 still goes over HTTP, but now the SSH client passes the correct credentials
 to the Git LFS client, so no action is required by the user.

216
217
Given that Git LFS uses HTTP Basic Authentication to authenticate the user pushing
the LFS object on every push for every object, user HTTPS credentials are required.
Marin Jankovski's avatar
Marin Jankovski committed
218

219
220
By default, Git has support for remembering the credentials for each repository
you use. This is described in [Git credentials man pages](https://git-scm.com/docs/gitcredentials).
Marin Jankovski's avatar
Marin Jankovski committed
221

222
223
For example, you can tell Git to remember the password for a period of time in
which you expect to push the objects:
Marin Jankovski's avatar
Marin Jankovski committed
224
225
226
227
228

```bash
git config --global credential.helper 'cache --timeout=3600'
```

229
230
This will remember the credentials for an hour after which Git operations will
require re-authentication.
Marin Jankovski's avatar
Marin Jankovski committed
231

232
233
If you are using OS X you can use `osxkeychain` to store and encrypt your credentials.
For Windows, you can use `wincred` or Microsoft's [Git Credential Manager for Windows](https://github.com/Microsoft/Git-Credential-Manager-for-Windows/releases).
Marin Jankovski's avatar
Marin Jankovski committed
234

235
More details about various methods of storing the user credentials can be found
236
on [Git Credential Storage documentation](https://git-scm.com/book/en/v2/Git-Tools-Credential-Storage).
237
238
239
240
241
242
243

### LFS objects are missing on push

GitLab checks files to detect LFS pointers on push. If LFS pointers are detected, GitLab tries to verify that those files already exist in LFS on GitLab.

Verify that LFS in installed locally and consider a manual push with `git lfs push --all`.

244
If you are storing LFS files outside of GitLab you can disable LFS on the project by settting `lfs_enabled: false` with the [projects api](../../api/projects.md#edit-project).