Worktrees in Production: Blue-Green Deploys from One Clone
Two live checkouts, one bare repo, and a symlink flip driven by a `post-merge` hook, including the classic BSD/GNU `mv` bug that breaks the naive version of this trick.
Worktrees in Production: Blue-Green Deploys from One Clone
Part 4 of 8 in the Beneath the Porcelain series: what you can build, break, and repair using nothing but git plumbing.
Part 1's deploy hook checks out main into the same directory every time. That is fine until the new version is broken and the webserver is already serving it. Rolling back means checking out the previous commit into that same directory and hoping nothing read a half-written file in between, because for the instant rsync is mid-sync, the live directory is neither the old version nor the new one.
Blue-green deployment solves this by never deploying into the directory that is currently live. Instead, you keep two complete copies — call them blue and green — and a pointer that says which one is currently serving traffic. A deploy writes the other one, fully, at its own pace, and only once it is completely in place does the pointer move. Rollback is just moving the pointer back. The trick that makes this cheap instead of doubling your disk and your git clone time is git worktree: one bare repo, two independent checkouts, sharing a single object database.
Two Worktrees From One Bare Repo
Start from the bare repo and a normal clone the way Part 1 did, then add two more checkouts as worktrees instead of full clones:
$ git clone /srv/site.git /srv/control
$ git -C /srv/control worktree add --detach /srv/blue main
$ git -C /srv/control worktree add --detach /srv/green main
$ git -C /srv/control worktree list
/srv/control 860f607 [main]
/srv/blue 860f607 (detached HEAD)
/srv/green 860f607 (detached HEAD)That --detach is not optional convenience — it is required. git worktree refuses to check the same branch out in two places at once:
$ git worktree add /srv/blue main
fatal: 'main' is already used by worktree at '/srv/control'/srv/control already has main checked out, so blue and green have to sit in detached HEAD, pinned to a specific commit rather than following a branch. That is exactly the right posture for a deploy target anyway: you do not want blue silently moving forward every time someone updates main, you want it to sit at precisely the commit it was told to deploy until told otherwise. Each worktree shares /srv/control/.git's object database (worktrees under git worktree add link back to it via a .git file rather than a full .git directory), so adding a second and third checkout costs disk space proportional to the working tree, not the whole repository history a second time.
A plain symlink decides which one is live:
$ ln -s /srv/blue /srv/live
$ cat /srv/live/index.html
v1Whatever serves this site — nginx, a static file server, a reverse proxy — points at /srv/live, never at /srv/blue or /srv/green directly.
The post-merge Hook That Flips the Symlink
Part 1 drove its deploy from post-receive, on the bare repo itself. This time the deploy logic lives in /srv/control, a normal non-bare clone, and fires from post-merge — the hook git runs after a git merge completes, which includes every git pull that is not a no-op. Something (a webhook receiver, a post-receive hook on the bare repo doing git -C /srv/control pull, or — as Part 8 will show — the bare repo's own hook driving this directly) triggers a pull in /srv/control; post-merge reacts to it landing.
#!/usr/bin/env bash
set -euo pipefail
SRV="/srv"
LIVE_LINK="$SRV/live"
NEW_SHA=$(git rev-parse HEAD)
current_target=$(readlink "$LIVE_LINK")
if [ "$current_target" = "$SRV/blue" ]; then
idle="green"
else
idle="blue"
fi
echo "post-merge: deploying $NEW_SHA into idle slot '$idle'"
git -C "$SRV/$idle" checkout -q --detach "$NEW_SHA"
ln -sfn "$SRV/$idle" "$LIVE_LINK"
echo "post-merge: live now points at '$idle' ($NEW_SHA)"The logic is deliberately simple: look at which slot live currently points to, deploy into the other one, then move the pointer. git -C "$SRV/$idle" checkout -q --detach "$NEW_SHA" writes the new commit into the idle worktree in full before anything about the live pointer changes — the currently-serving slot is never touched.
Push a new commit and pull it into /srv/control:
$ git push origin main # v2
$ git -C /srv/control pull origin main
post-merge: deploying 7b6a87c...into idle slot 'green'
post-merge: live now points at 'green' (7b6a87c...)$ readlink /srv/live
/srv/green
$ cat /srv/live/index.html
v2Push again and the pointer swings back:
$ git push origin main # v3
$ git -C /srv/control pull origin main
post-merge: deploying 888f174...into idle slot 'blue'
post-merge: live now points at 'blue' (888f174...)Rollback, if v3 turns out to be broken, is not a git revert and a redeploy. It is ln -sfn /srv/green /srv/live, pointed back at the slot that was serving v2 a moment ago and was never touched by the v3 deploy. The rollback is instant because the old version was never torn down; it was just no longer pointed at.
Refusing to Flip on a Bad Deploy
Everything so far still flips traffic onto whatever landed in the idle slot, sight unseen. Combining this with Part 3's pattern — checkout, then run something real against the result, then act on the exit code — closes that gap: deploy into the idle slot exactly as before, but smoke-test it before moving the symlink, and simply stop if the check fails.
#!/usr/bin/env bash
set -euo pipefail
SRV="/srv"
LIVE_LINK="$SRV/live"
NEW_SHA=$(git rev-parse HEAD)
current_target=$(readlink "$LIVE_LINK")
if [ "$current_target" = "$SRV/blue" ]; then idle="green"; else idle="blue"; fi
echo "post-merge: deploying $NEW_SHA into idle slot '$idle'"
git -C "$SRV/$idle" checkout -q --detach "$NEW_SHA"
echo "post-merge: smoke-testing idle slot '$idle' before flipping traffic"
if ! node --check "$SRV/$idle/app.js"; then
echo "post-merge: REFUSED to flip. $idle failed its smoke test; live still on $current_target" >&2
exit 1
fi
ln -sfn "$SRV/$idle" "$LIVE_LINK"
echo "post-merge: smoke test passed. live now points at '$idle' ($NEW_SHA)"A broken deploy gets fully checked out into the idle slot — parked there for inspection — but the flip never happens:
$ git -C /srv/control pull origin main
post-merge: deploying 1d62d27...into idle slot 'green'
post-merge: smoke-testing idle slot 'green' before flipping traffic
/srv/green/app.js:1
console.log('v2'
^^^^
SyntaxError: missing ) after argument list
post-merge: REFUSED to flip. green failed its smoke test; live still on /srv/blue
$ readlink /srv/live
/srv/blueThere is a real subtlety worth catching before it surprises you in production: git pull's own exit code does not reflect the hook's failure. The merge itself succeeded — post-merge is an informational hook, not a gate like pre-receive, and its exit status has no power to undo a merge that already happened or to make git pull itself report failure:
$ git -C /srv/control pull origin main; echo "pull exit: $?"
...
post-merge: REFUSED to flip. green failed its smoke test; live still on /srv/blue
pull exit: 0Anything driving this pull — a cron job, a webhook receiver, Part 8's capstone — has to check the hook's own output or a side effect it leaves behind (a marker file, an alert, the live symlink's target) rather than trusting the exit code of the command that triggered it. main in /srv/control does still advance to the new commit even when the flip is refused, which is correct: the code is not broken from git's point of view, only from the smoke test's, and the next successful deploy will still try to promote it into the other idle slot on the next pull.
Worktrees Remember Where They Are, Even When the Directory Is Gone
git worktree add registers metadata under the main repository's .git/worktrees/, separate from the actual checkout directory it points at. Delete a worktree's directory the ordinary way — rm -rf, not git worktree remove — and that metadata does not know the directory is gone until something asks:
$ rm -rf /srv/green
$ git -C /srv/control worktree list
/srv/control 888f174 [main]
/srv/blue 888f174 (detached HEAD)
/srv/green 3de810d (detached HEAD) prunableprunable is git's way of flagging a worktree entry whose directory no longer exists, without actually removing the registration on your behalf — a safety margin against a transient mount failure or a directory that is merely unmounted, not truly gone. Left in that state, trying to recreate a worktree at the same path fails outright, and the error is specific enough to be worth recognizing on sight rather than debugging from scratch:
$ git worktree add --detach /srv/green main
fatal: '/srv/green' is a missing but already registered worktree;
use 'add -f' to override, or 'prune' or 'remove' to cleargit worktree prune clears the stale registration, and the same add command that just failed succeeds immediately afterward:
$ git worktree prune
$ git worktree add --detach /srv/green main
Preparing worktree (detached HEAD 888f174)Any deploy tooling around blue-green worktrees that might legitimately need to recreate a slot — after a disk wipe, a container rebuild, or a deliberate reset — should run git worktree prune as a matter of course before attempting add, rather than assuming a missing directory means a clean slate.
The mv Bug That Breaks the Obvious Version of This
A lot of blue-green tutorials write the symlink swap as "write to a temp symlink, then atomically rename it into place," on the theory that a bare ln -sfn has a brief window where the link does not exist. That instinct is reasonable — and the naive implementation of it is broken on both GNU and BSD systems, for a reason that only shows up the first time you actually try it against a link that points at a directory:
$ readlink /srv/live
/srv/blue
$ ln -sfn /srv/green /srv/.live.tmp
$ mv -f /srv/.live.tmp /srv/live
$ readlink /srv/live
/srv/blueThe link did not move. Here is what actually happened:
$ ls /srv/blue | grep live
.live.tmp -> /srv/greenmv followed /srv/live — because it is a symlink pointing at a directory — straight into that directory, and moved .live.tmp inside /srv/blue instead of replacing the live link itself. This is standard, documented mv/cp behavior: when the destination resolves to a directory, the source gets moved into it. GNU coreutils has a -T (--no-target-directory) flag specifically to opt out of this, but it is GNU-only; there is no equivalent flag on the BSD mv that ships with macOS, which is exactly the version this got tested against.
The fix is to skip the temp-file dance entirely and call ln -sfn directly on the link name, which is what the hook above actually does:
$ ln -sfn /srv/green /srv/live
$ readlink /srv/live
/srv/greenln -f treats an existing symlink at that path as a file to replace, not a directory to enter — the directory-following behavior is specific to mv and cp, not ln. This is the standard idiom for exactly this reason: it is portable across GNU and BSD, and it does not have the "looks fine until the link happens to point at a directory" failure mode. If your deploy tooling was written by copying a snippet that does mv a temp symlink into place, it is worth checking which side of this bug it landed on.
Takeaways
git worktree turns "two environments" into "two directories sharing one object database," not "two clones." Blue and green cost the disk space of two working trees, not two full repository histories. That is the difference that makes blue-green deploys cheap enough to use for something as small as a single VM serving a personal site.
Detached HEAD is the right state for a deploy target, not a workaround. A worktree following a branch moves every time someone pushes to it. A worktree pinned to a specific SHA stays exactly where a deploy last put it, which is the property you actually want for something serving production traffic.
The symlink swap has to happen with ln -sfn on the link itself, not a temp-file-plus-mv. mv and cp follow a destination symlink into the directory it points at; ln -f replaces the symlink instead. It is a one-line difference that determines whether your blue-green flip actually flips.
post-merge informs; it does not gate. Its exit code never propagates back to git pull, so a smoke test that lives there can refuse to flip the symlink, but it cannot make the pull itself look like it failed. Anything orchestrating this deploy needs to watch the hook's actual output or a side effect, not the exit status of the command that triggered it.
A worktree's registration and its directory are two separate things, and only one of them disappears with rm -rf. A worktree deleted by hand instead of with git worktree remove leaves a prunable entry behind that blocks recreating a worktree at the same path until git worktree prune clears it.
Next: Part 5: Custom Merge and Filter Drivers for Generated Files. Teaching git that a lockfile and a build-generated JSON manifest do not need to conflict just because two branches both touched them.
Beneath the Porcelain is an eight-part series by Ajitem Sahasrabuddhe on the git internals that quietly hold together small production systems. Every command shown was run against git 2.50.1 on macOS before being written up; output may differ slightly on other platforms or versions.