Sync said unchanged, and the new project stayed empty

Adding a project to a repo GitBasedDocs had already synced showed an empty project until someone pushed. The check that makes sync cheap was keyed on the branch head, and creating a project does not move the head.

The cheapest thing GitBasedDocs sync does is ask GitHub one question: has the branch moved? One call to git/refs/heads/main returns a sha. If it matches lastSyncedSha, the run logs "unchanged" and stops. That check is why a timed recheck every ten minutes costs one API call.

Before using the app day to day, I read lib/sync/sync.ts next to the project code, and the two did not agree about what "nothing changed" means.

The bug

Creating a project inserts a row into projects. It does not touch the repo connection. So the next run goes like this:

1. Ask for the head. Same sha as last time.
2. Log "unchanged" and stop.
3. Never look at the tree, so never see the new project's files.

The project page says "No pages yet. Push Markdown to the repo and it shows up here after the next sync." The Markdown is already in the repo. Pressing Sync now gives the same answer, because the button calls the same function.

Changing a project's folder has the same problem, and so does restoring an archived one. None of those moves the branch head.

The fix was already in the codebase for a neighbouring case. Saving a connection that points at a different repo or branch clears the stored head:

...(retargeted
  ? { lastSyncedSha: null, lastCheckedAt: null, lastError: null }
  : {}),

Project changes needed the same treatment.

Forgetting the head, twice

export async function forgetSyncedHead() {
  const db = await getDb()
  await db.update(repoConnections).set({ lastSyncedSha: null })
}

export async function resync(): Promise<SyncResult> {
  await forgetSyncedHead()
  if (inFlight) {
    await inFlight.catch(() => null)
    await forgetSyncedHead()
  }
  return runSync("manual")
}

The first forgetSyncedHead is the durable one. If the process restarts before the run, the timed recheck still finds no stored head and reads the tree.

The second one handles a sync that was already running when the admin clicked Create. That run read the project list before the new row existed, and when it finishes it writes the head back. Without waiting for it and clearing again, the new project would be skipped by the exact race this function exists to fix.

The create and update routes call it inside after(), so the admin gets their response straight away and the pages appear a few seconds later.

A second bug underneath

While I was in that loop, the skip for unchanged files caught my eye:

const unchangedBlob =
  prior && prior.blobSha === entry.sha && prior.status === "active"
if (unchangedBlob) continue

A page's slug comes from its path relative to the project folder. docs/acme/guides/auth.md under a project at docs/acme is guides/auth. Move that project to docs and the same file should become acme/guides/auth.

The file did not change, so its blob sha did not change, so the row was skipped and kept its old slug. The sidebar is built from slugs, so it showed the old nesting, and links written against the new layout broke.

The skip now also compares the slug:

prior.slug === pageSlug(project.repoPath, entry.path)

Moving a folder rewrites those rows without fetching a single blob again.

Checking it without GitHub

lib/sync/resync.check.ts runs full syncs on a scratch SQLite file. It replaces globalThis.fetch with a fake GitHub that serves a head sha, a tree and blobs from an in-memory object.

The check writes the old behaviour down on purpose:

const beta = await addProject("beta", "beta")
assert.equal((await runSync("manual")).status, "unchanged")
assert.deepEqual(await slugsOf(beta), [], "the old behaviour this fix is for")
assert.equal((await resync()).status, "synced")
assert.deepEqual(await slugsOf(beta), ["b"])

Then it moves a project to the repo root and checks the slug changed.

One test did not survive. I wrote a case for "resync while a run is in flight", then noticed the in-flight run started after the project was inserted, so it passed with or without the fix. A check that cannot fail is worse than no check, because it reads as coverage. It came out, and the card says the race is not covered.