Rendering
Content pipeline
How Papervine syncs a Git repo of MDX into object storage and serves it, including static assets.
How Papervine syncs a Git repo of MDX into object storage and serves it, including static assets.
Papervine renders from a Git repo, but it never reaches into that repo on the hot path. Content flows through a sync step that copies a repo into Papervine’s own object storage; the render plane then reads only from storage. This page describes that flow, where things land, and how a push (or a manual re-sync) keeps it fresh.
A sync runs when a site is first connected, when a push webhook arrives, or when an
operator clicks Re-sync. All three paths call the same session-less core,
runSync (src/lib/sync-runner.ts) — the webhook’s authorization is its signature, not
a user session.
The sync resolves the branch head, then walks to the tree of the docs subdirectory
and lists it recursively (src/lib/sync.ts, the Git tree API). This is a handful of
requests no matter how large the repo is, and it never reads the rest of a monorepo —
only the files under the docs path are enumerated, each with its content-addressed
blob SHA.
Each blob’s SHA is compared to a per-site manifest (sites/{id}/.manifest.json,
path → blob SHA, written by the previous sync). Only files whose content changed — or
are new — are fetched; files that vanished from the repo are swept from storage. An
unchanged re-sync moves nothing. The decision logic is pure and unit-tested
(src/lib/sync-plan.ts).
Changed files are downloaded and uploaded in one bounded-concurrency pool that overlaps
fetch and upload, with Content-Type inferred per extension. Public repos read file
bytes from the raw.githubusercontent.com CDN; private repos read them through the
authenticated blobs API, with the installation token (or PAT) authorizing the request.
Splitting on visibility keeps a large public repo off GitHub’s REST rate limit, which a
burst of unauthenticated API calls would trip immediately.
The config is parsed against the schema. Because config is a compatibility layer, parsing warns rather than throws — see config.
The site row is stamped with lastSyncedCommitSha, and a deployment row records the
outcome (including how many files actually moved). The render plane keys its cache on
that sha (below), so the new content is served immediately.
Because sync cost scales with the diff rather than repo size, the heavy step is the
first sync of a site; subsequent pushes typically move only the handful of files that
changed. The whole-repo tarball reader (src/lib/tar.ts) is kept as a fallback for the
rare docs tree so large it exceeds the tree API’s single-response cap.
The long-term goal is to precompile MDX → serializable bundles at sync time so the render plane only executes them — no live compilation on the hot path. Today the synced files are compiled on request (reusing the same pipeline). The decision that matters for contributors is compile-on-sync, not compile-on-request — see renderer internals.
Each site owns a prefix in the bucket, sites/{id}/…, holding its docs.json, MDX, and
assets. The render plane reads a site only through the s3Source content source (behind
the ContentSource abstraction); the old request-time GitHub fallback is gone. A site
that has not synced has nothing to show — it 404s rather than reaching back to the repo
live. GitHub is touched only at sync time.
This is why local development seeds its sites into object storage too: docs render exactly the same way locally as in production.
The render path reads config and pages through the Next Data Cache (with a TTL safety net).
Rather than relying on cache invalidation to serve fresh content after a sync, the
content cache key is stamped with the synced head sha (lastSyncedCommitSha, read
live per request). A new sync writes new keys and therefore serves fresh content with no
revalidation needed; old versions age out via the TTL.
Stamping the key on the sha is what makes the push webhook correct. The webhook runs its sync in a deferred callback, where cache-tag revalidation does not propagate to the Data Cache — so a tag bust alone would serve stale docs until the TTL self-healed. Version-keying sidesteps that entirely. See gotchas.
A registered GitHub App delivers push events to POST /api/github/webhook. The route
verifies the X-Hub-Signature-256 HMAC over the raw body, maps the push to the site(s)
on that repo + branch, and runs the sync in a deferred callback so GitHub gets a fast
202. It is idempotent across redeliveries: a head already in lastSyncedCommitSha is
skipped.
The Re-sync button in the dashboard runs the same runSync synchronously. Because
there is no per-site lock yet, it refuses while a sync is already in flight (guarding
the common case of a mid-build re-sync); the button surfaces “A sync is already in
progress.”
The GitHub App is optional. With no App configured, the webhook rejects every (unsignable)
delivery and the connect form falls back to a pasted fine-grained PAT. Public repos stay
zero-config on any deployment; private repos add the App (or a PAT), whose token flows
through one seam (ghHeaders(token?) in src/lib/github.ts) and is encrypted at rest.
There is no sync queue or lock yet, so two concurrent syncs on the same site (e.g. webhook + manual, or two quick pushes) write to the same storage prefix and a reader can briefly see a torn tree. The manual-path guard and the idempotent sha-skip cover the common cases; a per-site advisory lock or job queue is the real fix.
Docs reference assets by absolute path from the repo root — , and
the logo / favicon paths in docs.json. Those files live alongside the MDX, outside the
app, so they are not served from Next’s public/. src/middleware.ts rewrites any request
whose path ends in a static-asset extension to an asset handler that streams the file with
the correct Content-Type and a path-traversal guard. The /img/… URL shape never changes
regardless of where the bytes actually live.
Assets are served by proxying from object storage: a request like
{slug}.papervine.io/img/x.png is rewritten to /api/tenant-asset/{slug}/…, whose handler
reads the object server-side (the bucket stays private, no CORS) and streams it back with a
cache header so the host CDN edge-caches it.
Proxying works transparently on custom domains because the page’s hostname and the asset’s
hostname are independent — both resolve to our origin as same-origin /img/…, whether the
docs live on {slug}.papervine.io or a tenant’s own docs.example.com, so there is never a
CORS step. Serving assets directly from a dedicated bucket-backed host is a possible future
optimization, not required at launch.
Content images render through Next’s next/image, so readers get format negotiation
(AVIF/WebP), responsive srcset, and lazy-loading — without layout shift as each image loads.
next/image needs an image’s intrinsic width and height, and a markdown image ()
carries none. Papervine captures them at sync time, not per request: as the sync copies
each raster image into object storage it measures the pixel dimensions from the file header and
records them in a .dimensions.json manifest alongside the content. Because sync is
incremental, only changed images are re-measured; untouched ones keep their recorded
dimensions and removed ones are dropped. The render path reads this manifest through the same
cache as everything else and hands each image its dimensions.
The image renderer degrades in tiers, so an image never breaks or jumps:
srcset.Animated GIFs stay a plain image so their animation survives, and SVGs and external-host images stay plain too — the optimizer only handles images served from this origin. Everything still gets lazy-loading regardless.
Both image syntaxes optimize identically: a markdown image () and a literal HTML
<img src="…"> (the form many docs use, often inside a <Frame>) take the same path. Authors
don’t have to choose one to get optimization.
Images are always served through the slug-keyed asset route (/api/tenant-asset/{slug}/…),
which identifies the site from the path rather than the host. This matters because the image
optimizer fetches the original server-side, without the tenant’s host — so a host-dependent URL
would fail to resolve. The slug-keyed route resolves the same on a subdomain, a custom domain,
or the apex, so optimization works everywhere.
A docs site renders as a persistent shell — the navbar, tabs, sidebar, and assistant — wrapping a per-page article. The shell is a layout that stays mounted as you move between pages of the same site, so only the article re-renders: the sidebar keeps its scroll and expanded groups, and a navigation swaps just the content (with an instant skeleton while it loads) rather than rebuilding the whole page.
docs.json is parsed leniently during sync.