Two GitHub calls per changed page against a token allowed 5,000 an hour meant a big vault's first sync would fail partway. Fixing that turned up a worse bug: a file that failed to download was marked as synced along with everything else.
When GitBasedDocs started showing who changed each page, sync gained a second GitHub call per changed file: the blob, then the newest commit that touched it. Both ran one after the other. For a vault of a few dozen notes that is nothing.
A fine-grained token gets 5,000 calls an hour. At two calls per page, a vault of about 2,500 notes cannot finish its first sync inside that hour. I went looking for how a run fails partway, and found that it did not fail at all. It succeeded, wrongly.
The worse bug
When a blob fetch failed, the loop did this:
} catch (e) {
// Keep the last good copy, flag it, try again next run.
errors.push(`${entry.path}: ${note(e)}`)
if (prior) await markStale(prior.id)
continue
}
The comment promises a retry. The end of the run broke that promise:
await markConnection("connected", errors.length ? errors[0] : null, headSha)
The head sha was stored whether or not every file made it. The next run asked GitHub for the head, got the same sha, and stopped with "unchanged". A new file that failed never got a row. An existing one stayed flagged stale. Neither was fetched again until somebody pushed.
On a rate limited first sync, that is hundreds of pages missing with nothing to bring them back.
One rule: the head moves when every file is in
A run is now incomplete if any of these happened:
- a blob fetch failed
- GitHub answered 403 or 429
- commit lookups ran past this run's budget
- asset downloads stopped early
await markConnection(
rateLimited ? "error" : "connected",
errors.length ? errors[0] : null,
incomplete ? undefined : headSha,
)
undefined leaves the stored head alone. That sounds expensive until you trace the next run. It sees a head that differs from the stored one, reads the tree (one call), and skips every page whose blob sha already matches. Only the files still missing cost anything.
A rate limited run also leaves the connection in "error", which is what raises the admin banner.
Stop asking once GitHub says no
lastCommit used to swallow every error and return null, which is right for an optional detail and wrong for a rate limit: the loop kept calling an API that had already refused it. It now rethrows 403 and 429, and every fetch checks a flag before it starts.
Pages download four at a time with a small pool instead of a queue library:
async function eachLimit<T>(items: T[], limit: number, fn: (item: T) => Promise<void>) {
let next = 0
const worker = async () => {
while (next < items.length) await fn(items[next++])
}
await Promise.all(Array.from({ length: Math.min(limit, items.length) }, worker))
}
A budget for the optional part
Who changed a page is nice to have. The page itself is not optional. So commit lookups get 300 per run, and the lookup function returns three different things:
// undefined: not looked up this run. null: looked up, nothing found.
const commitFor = async (path: string): Promise<LastCommit | null | undefined> => {
if (rateLimited || commitBudget <= 0) {
incomplete = true
return undefined
}
...
}
Past the budget, a page syncs without author details and the run counts as incomplete. The next run finds unchanged pages with no commit recorded and spends its budget filling them in.
There is a known ceiling, written into the code: a file whose lookup genuinely finds no commit gets asked again on every run that reads the tree. The budget bounds it. If it ever shows up in the call count, the fix is a "looked up" column.
Proving it with a fake GitHub
The sync check already swapped fetch for an in-memory GitHub. It gained a counter and a switch:
if (++calls.blobs > limitBlobsAfter) return new Response("slow down", { status: 429 })
With the switch at 5 and 20 changed pages, the first run keeps at most five pages and leaves the old head in place. The second run, with the limit lifted, makes exactly 20 - got blob calls and then stores the head.
A project with 310 changed pages makes exactly 300 commit lookups on the first run and 10 on the second. A third run reports "unchanged".
The numbers are the point. "The retry works" is a sentence. "Fifteen calls, not twenty" is a check that fails the day someone breaks the skip.
