How GitBasedDocs keeps pages fresh from a private repo with nothing installed in it: one ref call to check for changes, one tree call on a change, and blob fetches only for files whose sha moved.
The rule for GitBasedDocs was that the content repo stays clean. No .github/workflows folder, no build step, no deploy key. The only thing I allow on the GitHub side is an optional webhook. The app pulls everything else itself, with a fine-grained token that has Contents and Metadata read-only on that one repo.
Three things can start a sync: the webhook on push, the Sync now button in admin, and a timer. All three call the same runSync(trigger) in lib/sync/sync.ts, and every run writes a row to sync_logs with the trigger, head sha, counts and duration.
One cheap call first
Most syncs find nothing. The timer fires every 10 minutes by default, and on a quiet day none of those ticks has anything to do. So the first step is one request for the branch ref:
const ref = (await gh(
`/repos/${owner}/${repo}/git/refs/heads/${branch}`,
)) as { object: { sha: string } }
headSha = ref.object.sha
If that sha matches lastSyncedSha on the connection row, the run logs unchanged and stops. When I measured it, an unchanged recheck made exactly one GitHub call in 388 ms.
One tree, then only the blobs that moved
When the sha has moved, the next call is GET /git/trees/{headSha}?recursive=1. That returns every path in the repo with its blob sha in one response.
Each entry has to be a blob, sit under docsRoot if one is set, and stay out of dot folders:
export function isHiddenPath(path: string): boolean {
return path.split("/").some((segment) => segment.startsWith("."))
}
This came in as a fix. Obsidian moves deleted notes into .trash, and those should never come back as pages. .obsidian and .github are tooling, so they go too.
Then the entry has to belong to a project. A project is a folder path, and projectFor picks the longest matching path. That way a project nested inside another project's folder still claims its own files, and an empty path means the project owns the whole repo.
The diff itself is a comparison of blob shas. The run loads the existing doc_pages rows for active projects and skips any path whose stored blobSha equals the tree's sha. Git already computed a content hash for every file, so I never compare bodies myself.
For each page that did change, the run fetches the blob, decodes the base64, parses frontmatter with gray-matter, strips Obsidian %% comments %%, and upserts the row with title, slug, order, draft flag and a 2,000 character search excerpt. It also asks the commits API for the last commit that touched that path, which is what the "Updated by" line on each page shows. That lookup is optional: if it fails, the page syncs without it.
Paths that vanished from the tree are marked deleted rather than removed, and kept 30 days.
When one file fails
A sync should never blank a page because one request failed. If a blob fetch throws, the run keeps the last good copy, marks the row stale and moves on to the next file. The next run tries again because the stored sha still differs.
Images named by their hash
Images and PDFs under project folders go through lib/sync/assets.ts. Each file is stored on disk under its git blob sha:
// Files are named by git blob sha, which is already a content hash: one copy
// per distinct file however many paths or projects point at it.
If the same logo sits in five project folders, it is fetched and stored once. A write goes to a .tmp file first and is renamed into place, so a crash can't leave half a file under a name later reads would trust. At the end of the run, any file on disk that no row points at is deleted.
Assets arrived after the page sync, which created a small trap. An install already at the repo head would stop on unchanged and never download its images. The fix was a migration that clears the stored sha once:
UPDATE `repo_connections` SET `last_synced_sha` = NULL;
One run at a time
Two pushes in quick succession, or a webhook landing while someone presses Sync now, must not run two syncs over the same rows. The lock is a module-level promise:
export function runSync(trigger: SyncTrigger = "manual"): Promise<SyncResult> {
if (inFlight) return inFlight
inFlight = doSync(trigger).finally(() => {
inFlight = null
})
return inFlight
}
A second caller gets the run already in progress. A click on Sync now during a run joins it instead of starting another. The comment above it says the ceiling out loud: this holds for one app instance, and the day it runs on two nodes it becomes a database advisory lock.
The timer
instrumentation.ts starts the recheck once when the server boots. The interval comes from SYNC_INTERVAL_MINUTES, clamped to 5 to 15, default 10. The first tick runs 5 seconds after boot, to catch pushes that arrived while the app was down. After one restart the first scheduled run picked up a push no webhook had delivered: cron 6b7ef5d +1 ~1.
If GitHub answers 403 or 429, the run reports rateLimited and the timer backs off: retry after 1 minute, then 5, then back to the normal interval. The connection stays in the error state meanwhile, which raises the admin banner. At the default interval the recheck writes 144 log rows a day, so each tick also deletes sync logs older than 30 days.
A full run of my docs repo came in at 5 GitHub calls and 3.4 seconds.
