What Actually Happens on `git push`

Part 2 of 8 in the Beneath the Porcelain series: what you can build, break, and repair using nothing but git plumbing.

Part 1 pushed into a bare repo nine times and treated git push as a black box: run it, watch a hook fire, move on. That is fine until something in a CI pipeline breaks in a way that only makes sense if you know what is actually happening on the wire. "Everything up-to-date" when you were certain you had new commits. A push rejected with "stale info" on a shared runner that pushed successfully thirty seconds ago. A ref that fetches as all zeroes. All three are explainable, in a few minutes, once you have watched a push happen at the protocol level instead of at the command level.

Git ships the tool for this: set GIT_TRACE_PACKET=1 and every pkt-line — the atomic unit of git's wire protocol — gets printed to stderr as it is sent or received. It is noisy and exactly precise, which is what you want when a CI failure needs a real explanation instead of a guess.

Ref Advertisement, Then Objects

A push is two phases: figure out what the other side already has, then send only what it does not. Pushing a single new branch with tracing on shows both:

$ GIT_TRACE_PACKET=1 git push origin main

packet: receive-pack> 0000000000000000000000000000000000000000 capabilities^{}\0report-status report-status-v2 delete-refs side-band-64k quiet atomic ofs-delta object-format=sha1 agent=git/2.50.1-Darwin
packet: receive-pack> 0000
packet:         push< 0000000000000000000000000000000000000000 capabilities^{}\0report-status report-status-v2 delete-refs side-band-64k quiet atomic ofs-delta object-format=sha1 agent=git/2.50.1-Darwin
packet:         push< 0000
packet:         push> 0000000000000000000000000000000000000000 a8ebdd64cfed18df1958bfe8a683b3b3d41a937d refs/heads/main\0 report-status-v2 side-band-64k quiet object-format=sha1 agent=git/2.50.1-Darwin
packet:         push> 0000
packet: receive-pack< 0000000000000000000000000000000000000000 a8ebdd64cfed18df1958bfe8a683b3b3d41a937d refs/heads/main\0 report-status-v2 side-band-64k quiet object-format=sha1 agent=git/2.50.1-Darwin
packet: receive-pack< 0000
packet: receive-pack> unpack ok
packet: receive-pack> ok refs/heads/main
packet: receive-pack> 0000

Read the < and > as "received" and "sent" from the perspective of the label in front of them (receive-pack is the remote process; push is your local one). The shape is:

The remote advertises its capabilities and current refs. The first 0000000000000000000000000000000000000000 capabilities^{} line is the remote saying "I support these features," using the all-zero SHA as a sentinel because there is no ref to advertise data about yet on a brand-new repo.
The client states the ref update it wants. push> sends <old-sha> <new-sha> <refname>: the old SHA it believes the remote has (0000... here, since this is a new branch), the new SHA it wants the remote to have, and which ref. This single line is the entire content of what you are asking the remote to do.
The objects themselves travel as a packfile, sent as a side-band stream immediately after the ref negotiation. It is not shown as pkt-line text in the trace above because from this point the payload is binary packfile data, not text commands.
The remote reports status per ref. unpack ok means the packfile applied cleanly; ok refs/heads/main means that specific ref update succeeded. A multi-ref push gets one status line per ref, and it is entirely possible for some to succeed and others to fail in the same push.

Every CI failure this part covers is a variation on step 2 going wrong: the client's belief about the old SHA, or the remote's actual current SHA, not matching what the other side expects.

"Everything up-to-date" Is Not a Bug Report

Push again with no new commits and git says almost nothing:

$ git push origin main
Everything up-to-date

This confuses people who expect a push to always talk to the network. It does talk to the network — the ref advertisement in step 1 above still happens — but the client compares the remote's advertised SHA for refs/heads/main against its own local SHA for the same branch, finds them equal, and stops before ever building a packfile. There is nothing to negotiate and nothing to send.

In CI this shows up as a deploy step that appears to run successfully but changes nothing, usually because an earlier step already pushed the same commit (a re-run of a partially-failed pipeline, a duplicate trigger, a rebase that produced an identical tree). It is not evidence the push failed silently; it is git correctly determining there was no work to do. If you need proof a specific commit landed, check the SHA the remote actually has (git ls-remote origin main), not the exit message of the push that happened to run before it.

The Force-Push Race on a Shared Runner

Two branches, both cut from the same base, both pushed by different CI jobs running concurrently — a common shape on any runner pool without strict serialization. The first push succeeds normally:

$ git push origin main   # from job A
   a8ebdd6..3c7e24a  main -> main

The second job's local view of main is now stale — its own old-sha for the ref-update line is the original base commit, not 3c7e24a — and git's own pre-push fast-forward check catches this locally, before contacting the server at all:

$ git push origin main   # from job B, moments later
 ! [rejected]        main -> main (fetch first)
error: failed to push some refs to '...'
hint: Updates were rejected because the remote contains work that you do not
hint: have locally.

This is the correct, boring outcome, and it is why routine CI jobs should never reach for --force to make a rejection like this go away. --force skips the fast-forward check and overwrites the ref outright — job B's push would silently discard job A's commit. The right response to "fetch first" is exactly what it says: fetch, then decide whether to merge, rebase, or abandon your branch.

Where teams get hurt is when someone does reach for --force-with-lease — the more careful cousin of --force, which refuses to overwrite a ref unless the remote's current value still matches what your local remote-tracking ref last recorded. Run it with a stale tracking ref and it fails the same way, for a subtly different reason:

$ git push --force-with-lease origin main
 ! [rejected]        main -> main (stale info)
error: failed to push some refs to '...'

"Stale info" specifically means your lease — your last-known value of the remote ref — no longer matches reality, because someone else moved the ref since your last fetch. That is --force-with-lease doing exactly its job: refusing to blindly overwrite work it does not know about. Fetch and it succeeds:

$ git fetch origin
$ git push --force-with-lease origin main
 + 3c7e24a...6664eab main -> main (forced update)

The lesson for CI specifically: any job that force-pushes a ref other jobs might also touch needs to fetch immediately before force-pushing, every time, with no gap for another job to slip in between the fetch and the push. On a busy shared runner that gap is exactly where these races live.

Corrupted Refs, and What They Actually Look Like

A ref is, at the loose-ref level, just a file containing a 40 (or 64, for SHA-256 repos) character hex string, sitting under refs/heads/ or refs/tags/. Anything that writes a partial file there — a disk full during a ref update, a crashed process, a bad manual edit — produces a ref that is not a valid pointer, and the failure mode depends on exactly how it is broken.

A ref file with garbage content that is not valid hex:

$ echo "notasha" > /srv/site.git/refs/heads/main
$ git ls-remote origin
0000000000000000000000000000000000000000	refs/heads/main

Git treats an unparseable ref as if it points at nothing — the all-zero SHA again, the same sentinel from the ref advertisement in step 1. A client that then tries to fetch that ref gets a clear failure rather than silent corruption:

$ git fetch origin
fatal: git upload-pack: not our ref 0000000000000000000000000000000000000000

A ref file with a well-formed-looking SHA that simply does not exist in the object database is more dangerous, because it advertises fine:

$ echo -n "deadbeefdeadbeefdeadbeefdeadbeefdeadbeef" > /srv/site.git/refs/heads/main
$ git ls-remote origin
deadbeefdeadbeefdeadbeefdeadbeefdeadbeef	HEAD
deadbeefdeadbeefdeadbeefdeadbeefdeadbeef	refs/heads/main

It looks like a normal SHA right up until something tries to actually fetch it:

$ git fetch origin
fatal: git upload-pack: not our ref deadbeefdeadbeefdeadbeefdeadbeefdeadbeef

git fsck on the repository itself is far more specific about what is wrong, because it is checking the ref file's syntax directly rather than trying to resolve it as an object reference:

$ git fsck
warning: refs/heads/main: refMissingNewline: misses LF at the end
error: refs/heads/main: invalid sha1 pointer deadbeefdeadbeefdeadbeefdeadbeefdeadbeef
error: HEAD: invalid sha1 pointer deadbeefdeadbeefdeadbeefdeadbeefdeadbeef
dangling commit 6664eab01ec034d79b301054c91a97f8dcbdb5bf
dangling commit 3c7e24a0c621942ab191cefc7f3f3b110f391604

That last part is the fix, hiding in plain sight: git fsck found the actual commits, still sitting in the object database as "dangling" (unreachable from any ref, but not deleted), because a corrupted ref file does not delete the objects it used to point at. Repairing the ref is a single update-ref call to whichever dangling SHA is correct:

$ git update-ref refs/heads/main 6664eab01ec034d79b301054c91a97f8dcbdb5bf
$ git fsck
dangling commit 3c7e24a0c621942ab191cefc7f3f3b110f391604

On a CI runner that reports a mysteriously empty or missing branch after what looked like a successful pipeline run, this is the actual triage order: check ls-remote for an all-zero or suspicious-looking SHA before assuming history was actually lost, then fsck for dangling commits before reaching for a backup.

Shallow Clones Change the Negotiation, Not Just the History

CI runners routinely clone with --depth 1 to save time on repositories with long histories, and it works fine for a build that only reads files. It stops working the moment that same shallow clone tries to push somewhere that has never seen its full history:

$ git clone --depth 1 file:///srv/site.git shallow-clone
$ cd shallow-clone && echo "change" >> f.txt && git commit -am "shallow commit"
$ git push origin main
 ! [remote rejected] main -> main (shallow update not allowed)
error: failed to push some refs to '...'

"Shallow update not allowed" is the negotiation from earlier in this part hitting a wall specific to shallow history: the client's ref-update line still says "here is my old SHA, here is my new SHA," but the client cannot prove ancestry beyond its shallow boundary, and a fresh remote has no way to verify the new commit is actually reachable from history it can trust. Note that this is not universal — pushing that same shallow clone's new commit to a remote that already has the full ancestor history (because something else already pushed it there in full) succeeds without complaint, since the remote can independently verify the ancestry itself. It is specifically pushing shallow history to a remote that has never seen it that triggers the rejection.

The fix is to stop being shallow before pushing:

$ git fetch --unshallow
$ git rev-parse --is-shallow-repository
false
$ git push origin main
 * [new branch]      main -> main

--unshallow fetches the full ancestry the shallow clone was missing and converts the repository back into an ordinary, complete one, at the cost of the network transfer --depth 1 was meant to avoid in the first place. Any pipeline stage that both clones shallow (for speed) and later needs to push (to redistribute a rewritten history, tag a release, or push a generated commit back) needs this fetch in between, or it needs to avoid --depth on that job entirely and accept the slower, complete clone instead.

Takeaways

Every push is a two-phase negotiation, not a single atomic upload. The client and server first agree on what the client believes the remote has for each ref; only then does a packfile move. Every failure mode in this post is a mismatch surfacing during that negotiation, not a transfer error.

"Up-to-date," "rejected," and "stale info" are three different, precise statements, not synonyms for "push failed." Up-to-date means no negotiation was needed at all. Rejected (fetch first) means your local fast-forward check caught a real divergence before contacting the server. Stale info means --force-with-lease's remembered value of the remote ref is out of date. Reading which one you got tells you what actually happened.

A ref is a file, and files can be corrupted independently of the objects they point to. git fsck's "dangling commit" output after a ref corruption is usually your data, still present, just unreachable. Recovery is often a single update-ref away, not a restore-from-backup situation.

A shallow clone can read history it does not have, but it cannot always push into a place that needs to trust it. --depth 1 is a fine default for a build stage that only reads files; any stage in the same pipeline that also needs to push should either avoid the flag or run git fetch --unshallow first.

Next: Part 3: Hooks as a Poor Man's CI. Turning the pre-receive phase of the negotiation you just traced into an actual quality gate.

Beneath the Porcelain is an eight-part series by Ajitem Sahasrabuddhe on the git internals that quietly hold together small production systems. Every command shown was run against git 2.50.1 on macOS before being written up; output may differ slightly on other platforms or versions.