0

I am trying to push my repository that has a lot of .png, .mp4, .h5 files, that I can ignore.

the directory structure:

.gitignore
CV PROJECT
├───.vscode
├───Docker
├───src
│   ├───Game
│   ├───Main
│   │   └───__pycache__  
│   └───Video
│       ├───Images       
│       │   ├───test    
│       │   │   ├───boost # dir that has .png files 
│       │   │   ├───click # dir that has .png files 
│       │   │   └───upgrade # dir that has .png files 
│       │   └───train  
│       │       ├───boost # dir that has .png files 
│       │       ├───click # dir that, has .png files 
│       │       └───upgrade # dir that has .png files 
│       ├───Models # dir that has .h5 files  
│       └───Videos
│           ├───boost # dir that has .mp4 files
│           ├───click # dir that has .mp4 files
│           └───upgrade # dir that has .mp4 files
└───Tutorial
    └───.ipynb_checkpoints

tl;dr what does my .gitignore need to look like?

the commands:

git config --global http.postBuffer 2048M
git config --global http.maxRequestBuffer 1024M
git config --global core.compression 9

git config --global ssh.postBuffer 2048M
git config --global ssh.maxRequestBuffer 1024M

git config --global pack.windowMemory 256m 
git config --global pack.packSizeLimit 256m

# git hangs here
git push --verbose -f origin master

the output:

git hangs here:

Total 133 (delta 29), reused 0 (delta 0), pack-reused 0
POST git-receive-pack (569667108 bytes)

so i tried to ignore them using this .gitignore:

*.mp4
*.h5


Models/*
Videos/*
test/*
train/*

boost/*
upgrade/*
click/*

and got this output:

remote: Resolving deltas: 100% (20/20), completed with 1 local object.
remote: warning: File src/Video/Videos/boost/b7.mp4 is 57.86 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File src/Video/Videos/click/c7.mp4 is 73.08 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File src/Video/Videos/upgrade/u6.mp4 is 56.50 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
remote: error: Trace: 3a803801864c0608c9e57dd7cf9d3ee3eaca6a180713b3c7119e8b0414c776ee
remote: error: See http://git.io/iEPt8g for more information.
remote: error: File src/Video/medmodel.h5 is 112.78 MB; this exceeds GitHub's file size limit of 100.00 MB
To https://github.com/romhayh/special-mario.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'https://github.com/romhayh/special-mario.git'

after this i converted the .gitignore to .gitattributes (to not use git lfs track) and got the same error message:

remote: Resolving deltas: 100% (29/29), completed with 1 local object.
remote: warning: File src/Video/Videos/boost/b7.mp4 is 57.86 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File src/Video/Videos/click/c7.mp4 is 73.08 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: warning: File src/Video/Videos/upgrade/u6.mp4 is 56.50 MB; this is larger than GitHub's recommended maximum file size of 50.00 MB
remote: error: GH001: Large files detected. You may want to try Git Large File Storage - https://git-lfs.github.com.
remote: error: Trace: 9add1b6eba287591037b40a5a483a1b7bb9d2746853150a2146bdf83b8e58b9c
remote: error: See http://git.io/iEPt8g for more information.
remote: error: File src/Video/Models/medmodel.h5 is 112.78 MB; this exceeds GitHub's file size limit of 100.00 MB
remote: error: File src/Video/medmodel.h5 is 112.78 MB; this exceeds GitHub's file size limit of 100.00 MB
romhayh
  • 25
  • 12

1 Answers1

3

A good real-world analogy is tricky here, but suppose you have a cement delivery service. You delivered, to some customer, 1000 cubic yards of cement, which have now set. Your customer calls you up and says they did not want that much cement: they only wanted ten cubic yards. So you have your trucks deliver them another 10 yards of cement. Did you improve their situation, or just make it worse?

Now, the reason for the above analogy is simple enough: Git commits only ever add new stuff to the repository. If you put a giant file into some commit, that giant file is in that commit forever. If someone didn't want that giant file, and you make a new commit that lacks the giant file, well, now you have two commits: one with the giant file, and one without. The second commit depends on, and therefore requires, the first commit.

No matter what else you do, if you keep adding on new commits, you're just making everything worse. What you need is not more cement (er, commits). You need some kind of jackhammer and cement-removal service (something to remove commits from your repository).

That's not all, though! While we already see that listing a file in .gitignore has no effect on any existing commit—no existing commit can ever be changed, at all; you just have to get them hauled away and stop using them entirely to get rid of them—it also has no good effect on new commits, at least not yet. If a file is tracked, it has no effect on that file in future commits either.

So:

  • Your first job is to remove some commit(s). We generally do this with git reset. The git reset command is destructive! Be very careful here: you probably should work on a clone of your repository, rather than the original.

  • Then, having removed the bad commit(s), you may need to explicitly remove the large files, so that they are no longer tracked. A file is tracked, in Git, if the file is in Git's index. How do you know if a file is in Git's index? Well, you know because it's tracked. This is kind of a circular problem, as you can see. :-) But if you definitely want it not-tracked, you can run:

    git rm --cached src/Video/Videos/boost/b7.mp4

    for instance (for the file named src/Video/Videos/boost/b7.mp4). This will remove the file from Git's index, if it is present in Git's index (is tracked), or will give you an error about the file not being in Git if the file isn't present in Git's index (is untracked). So either way, you can now be sure that the file is untracked.

When a file is untracked, git status usually whines about it. The .gitignore listing makes git status shut up about it. That's often the main feature of .gitignore: making git status behave. The other feature is that, as long as some file currently is untracked, listing that file in .gitignore means that git add --all or git add * won't add it to Git's index, which would make it tracked.

The tracked-ness of some file is not exactly something you set once and forget. Because commits are unchangeable, once you've made some commit and that commit does have the file in it, any time you use git checkout to extract that commit, the file goes into Git's index and is thus once again tracked. You can use git rm --cached to un-track the file, so that it does not go into new commits you make. But you cannot remove it from existing commits, ever: no existing commit can ever change.

What you must do is remove the bad commits and then not commit the files again

You can put the files into Git-LFS, if you want to do that. (Git-LFS has its own weirdness and quirks. I only bring this up because you did, with the tag and with a mention of LFS in your question.) But you have to get rid of the existing commits that have the big files in them. Only you can say which commits those are, because those commits are in your repository, which nobody else has.

This is why I recommend that you clone your existing repository first, before trying to remove any of the bad commits. You may, however, find some other tools and/or methods more useful here. See, e.g., How to remove/delete a large file from commit history in Git repository?

torek
  • 448,244
  • 59
  • 642
  • 775