Debug git lost changes

Question

git has lost my 3 last commits with no trace in the log. When I do git log synthesize.py the last reported change is commit id 71ef61c, but this change does not match what is in the file. The content of the file is as though the last 3 commits never happened.

The lost changes were originally on another branch (I think).

$ git log synthesize.py
commit 71ef61c
Author: Axel Bregnsbo <axbr@demant.com>
Date:   Thu Nov 5 10:53:12 2020 +0100

    Moved comment from DC setup.tcl to Triage synthesize.py.

commit 6cc8397
Author: Axel Bregnsbo <axbr@demant.com>
Date:   Thu Nov 5 10:18:59 2020 +0100

    Create symlink to latest synth log-file that has constant name.

Based on comment input from Don Branson and matt, I got this surprising result:

$ git rev-parse HEAD
c66c96b1336848803d55fc002942c42c07a701e7

$ git log --oneline
c66c96b Added dim support isters              <====== missing 3 changes
b279877 extra analysis job exit
0d32763 added possiat the end of the synth
...
ef75ccf added remove all uand error_check     <====== this commit threw out my changes!!!
71ef61c Moved comment fromge synthesize.py.   <====== the last of my 3 changes
7b2ffae added check_unused ports procedure
6cc8397 Create symli that has constant name.

When I diff the troublesome ef75ccf commit, git admits that is has changed synthesize.py, and a diff shows the deleted code. Why does commit ef75ccf not show up in git log synthesize.py ?

$ git diff 71ef61c ef75ccf --name-only
flow/ralgen/lego2-to-ipxact
flow/triage/tasks/synthesize.py     <======= 

My working directory is in sync with index and master and the remote repository. I have done git pull git push and git status and none of the commands report any local change or any un-updated stuff.

Git lost change describes the same problem, but was closed due to too few details provided.

Additional info answering questions in the comment field:

$ git log --graph 
* commit c66c96b
| Author: yyyy
| Date:   Tue Nov 10 11:32:20 2020 +0100
| 
|     Added dim support for both types of registers
| 
*   commit 8789184
|\  Merge: f1043b3 74288c7
| | Author: xxxx
| | Date:   Fri Nov 6 09:48:29 2020 +0100
| | 
| |     Merge branch 'develop' of gitlab.kitenet.com:higgs/tools into develop
| |
| *   commit ef75ccf                           <======== removes all my changes
| |\  Merge: 7b2ffae 71ef61c
| | | Author: xxxx
| | | Date:   Thu Nov 5 11:14:14 2020 +0100
| | | 
| | |     added remove all unconnected procedure and error_check
| | | 
| | * commit 71ef61c                            <========  my 3rd change
| | | Author: Axel Bregnsbo <axbr@demant.com>
| | | Date:   Thu Nov 5 10:53:12 2020 +0100
| | | 
| | |     Moved comment from DC setup.tcl to Triage synthesize.py.
| | | 
| | * commit 6cc8397
| | | Author: Axel Bregnsbo <axbr@demant.com>
| | | Date:   Thu Nov 5 10:18:59 2020 +0100
| | | 
| | |     Create symlink to latest synth log-file that has constant name.
| | |   

$ uname -a
Linux kbnuxsrv220 4.9.0-8-amd64 #1 SMP Debian 4.9.144-3 (2019-02-02) x86_64 x86_64 x86_64 GNU/Linux

$ git --version
git version 2.18.0.rc1.1.g6f333ff.dirty

$ git diff --cached
$ git status -s
$ rm synthesize.py
$ git status -s
 D synthesize.py

$ git checkout synthesize.py   
<file still missing changes>

Answer 1

The git status you've done shows there are no changes between your file and HEAD (c66c96b1336848803d55fc002942c42c07a701e7). There are changes leading up to 71ef61c, and those changes seem to be on another branch, otherwise git log synthesize.py would show a more recent change that reflects what's in your workspace.

You could always use gitk to get a visual.

Having said that, it's possible there are two of these files. Use find * -name synthesize.py to find out. Just allowing for the possibility that Matt's on the right track, but I lean toward it being multiple branches.

Answer 2

TL;DR

You want git log --full-history (and later, perhaps -m as well).

Long

When using git log with path names—as in git log synthesize.py—you're telling Git to lie to you. These lies are meant to be helpful, but if you are not careful, they will be harmful instead.

Specifically, you are telling Git not to tell you everything about all commits. This part makes sense, especially when you realize that each Git commit has a full snapshot of every file, so that synthesize.py probably appears in every commit from the point at which it was first created, onward. Showing all of those commits is not helpful. We might like Git not to show many of the commits that do contain this file, and only show those commits that:

• don't have the file at all, right before and/or after commits that do have the file; or
• do have the file, right before and/or after commits that also have the file but have a different version of that file.

These commits add, remove, or modify the file, as compared to the commit just before or after them. This is important to us, in a way that two adjacent commits that have the file, but make no change to it, aren't.

Unfortunately, git log synthesize.py doesn't show you what I just suggested we might like. Instead, it shows something else—something easier for Git to do, and something that we might still like, or like even more. I predict, however, that it's something you don't like. Well, this is not exactly a prediction: it's something I have seen before and that you are griping about:

Why does commit ef75ccf not show up in git log synthesize.py?

The answer to this question is that Git is performing its default kind of History Simplification. But that just leaves the question: what is this History Simplification, and how can we control it?

The git log documentation has a whole section on this. This documentation has improved this year (2020), so if you have not read it for a while, it's worth another look. The key is that the default history simplification will omit commits, or even entire "branches",¹ that didn't contribute to the way the file looks right now. If you're looking for a file that has disappeared, or lines that have disappeared in some file, you are likely to miss where this happened because the file that isn't there now, or the lines that aren't there now, were in one of these commits or "branches" that didn't contribute in the end, so git log simplifies them away, as if they had never even occurred!

The big hammer to find the commits is --full-history, which tells Git that while it's simplifying away some history for display, don't ever cut off any "branch" wholesale such that it never even gets looked-at internally. This will cause the commit to turn up. However, if you're using some of the fancier (but more subtle) history searching tools such as the so-called "pickaxe" (e.g., -S and -G) tools, even --full-history may not be sufficient: add -m as well. If --full-history is too much, consider some of the other options described in the documentation.

The keys to understanding this are complicated. The two items to keep in mind are these:

1. git log is performing a commit graph traversal. Each commit in the commit graph stores the hash IDs of its immediate parent commits. Most commits have one parent, which forms those commits into a simple backwards-looking chain. When git log is working on one of these commits in the chain, it must visit the (single) parent of that commit. Some commits, however, are merge commits. When git log is working on one of these commits, it has a choice: it could follow just one parent, or it could follow all parents. When doing History Simplification by file contents—as in git log synthesize.py, for instance—it will by default follow a parent that has the same copy of the file. So any merge commit in which some programmer threw out some line(s) that they should have kept, git log will follow the wrong chain.

2. As git log performs this traversal, it will normally run a git diff between the (single) parent and the commit itself. That's the source for determining not only which file(s) changed, but also what those changes were. If you're using -S or -G to inspect the diff result, git log must perform such a diff. But merges are problematic: it's not possible to diff a child commit against all of its parents at the same time.² This is where the -m flag comes in: this tells git log to split the merge (virtually, not physically). If the merge commit has, say, six parents, this produces six individual diffs, one for each parent. Those six diffs each get tested via the -S or -G inspection.

So this is both why and how you must redirect the history simplification at times. First, you must make sure git log traverses the correct "legs" of any merge, so that git log will march through the commits on the "branch" (see footnote 1) that you care about. Second, if the lines you're searching for, with a fancy search, might have disappeared in a merge, and you're using -S to count their occurrences or -G to find them, you must use -m to split the merge so that they can be found.

---

¹I put "branches" in quotes here because I am referring to what I have taken to calling a "DAGlet". That is, we want Git to look at the "leg" of merge where the last commit in that leg has a copy of the file that doesn't match the final version of the file in that merge. See also What exactly do we mean by "branch"?

²Note that Git's diff engine is capable of producing a combined diff, but such a diff is almost never what we want here. What we need is an n-way diff that doesn't take the combining shortcuts, and the existing Git diff engine just is not prepared to do this.

Answer 3

I would try git reflog.

This shows a slightly different view of the world. It shows you where you've been and what movements you've made between commits and/or branches. You may or may not be able to directly recover from this output, but at least it would give you your missing commit IDs so you could cherry-pick or whatever you prefer.

Also, if it's available gitk --all is an easy way to visualise the state of your tree but it isn't always installed with Git depending on platform and version.