Flicker-free A/B tests on Cloudflare Workers

This site’s home page is running an A/B test right now. Half the visitors get a hero that lists skills, the other half one that leads with outcomes, and the metric is whether they reach out. The results are public on /experiments. Here’s how it’s built, and what I’d carry into a store.

Why client-side tests flicker

Most testing tools work in the browser: the page renders the control, then a script picks the variant and rewrites the DOM. If the script is slow, the visitor sees A and then B. The usual fix, an anti-flicker snippet that hides the page until the tool is ready, trades the flicker for a blank screen and a worse LCP in both arms.

The flicker can also bias the result: the variant is the version that moves, and people react to movement. Rendering the variant in the HTML removes the problem instead of hiding it.

Bucketing on the server

The home page isn’t prerendered. The Worker assigns the visitor and renders the chosen copy:

export function assign(request: Request, url: URL, cookies: AstroCookies): Assignment {
  const secure = url.protocol === 'https:';
  const forced = url.searchParams.get('variant');
  if (isVariant(forced)) return { variant: forced, track: false, forced: true };
  if (isBot(request)) return { variant: VARIANT_KEYS[0], track: false, forced: false };

  visitorId(cookies, secure);
  const name = cookieVariant(heroTest.id);
  const saved = cookies.get(name)?.value;
  if (isVariant(saved)) return { variant: saved, track: true, forced: false };

  const variant = VARIANT_KEYS[Math.floor(Math.random() * VARIANT_KEYS.length)];
  cookies.set(name, variant, { path: '/', maxAge: YEAR, httpOnly: true, sameSite: 'lax', secure });
  return { variant, track: true, forced: false };
}

Two cookies do the work: a random visitor ID and the assigned variant, both HttpOnly, SameSite=Lax, kept for a year. No script on the page can read or change the variant, so the tracking endpoint never has to believe what the browser claims it saw. It reads the cookies.

The page also sends cache-control: private, no-store. Without it, a CDN can cache one variant and serve it to everyone.

Bots and ?variant=b previews come back with track: false. They still get a variant (previews are how I QA B), but nothing counts them, and the endpoint checks the user agent again. The Spanish home page shares the assignment but is never tracked, because the test is about the English copy.

Count the exposure when it’s seen

Assignment isn’t exposure. Someone can get a cookie and leave before the hero paints. Counting them dilutes the effect with people who never saw the change, so the exposure fires from the browser once 60% of the hero is on screen:

const hero = document.querySelector('.hero__lede');
if (hero && 'IntersectionObserver' in window) {
  const io = new IntersectionObserver(([en]) => {
    if (en.isIntersecting) { beacon('exposure'); io.disconnect(); }
  }, { threshold: 0.6 });
  io.observe(hero);
} else beacon('exposure');

beacon() uses navigator.sendBeacon and falls back to fetch with keepalive, so the event survives the tab closing. The body is only { kind }; visitor and variant come from the cookies, server side. Clients that never run JavaScript never send one.

Conversions are contact intent from any page: copying the email, opening Contact or sending the form. A sessionStorage flag skips repeat beacons, but the real guard is the table.

One row per visitor

D1 is SQLite, and the primary key turns a second exposure or conversion from the same visitor into a no-op:

CREATE TABLE IF NOT EXISTS xp_events (
  experiment TEXT NOT NULL,
  visitor    TEXT NOT NULL,
  kind       TEXT NOT NULL CHECK (kind IN ('exposure', 'conversion')),
  variant    TEXT NOT NULL,
  created_at INTEGER NOT NULL,
  PRIMARY KEY (experiment, visitor, kind)
);

-- A conversion only counts for visitors who were exposed, attributed to the variant they saw
INSERT OR IGNORE INTO xp_events (experiment, visitor, kind, variant, created_at)
SELECT ?1, ?2, 'conversion', variant, ?3 FROM xp_events
WHERE experiment = ?1 AND visitor = ?2 AND kind = 'exposure';

The conversion insert copies the variant from the visitor’s exposure row instead of trusting the current cookie, and with no exposure row the SELECT returns nothing and nothing is written. Someone who lands straight on the contact page never saw either hero, so they don’t count. Retries and replayed beacons hit INSERT OR IGNORE and disappear. Results are a GROUP BY variant, kind, and a new test is a new experiment id, so its counts start from zero.

Reading the results

GET /api/xp returns the counts plus three numbers, built on the same normal CDF in src/lib/stats.ts that the calculators on /experiments use:

• the p-value of a two-sided, pooled two-proportion z-test,
• the probability that B’s true rate beats A’s, from a normal approximation of the difference,
• a status that stays at “collecting” until each arm has 100 exposures.

That 100-visitor floor only keeps the panel from celebrating noise. It is not a sample size. At a 2% conversion rate, detecting a 20% relative lift with 80% power takes about 21,000 visitors per arm, and a 50% lift about 3,800. A portfolio won’t get there, so the reading here is directional. And a panel that refreshes every 15 seconds is a window, not a stopping rule: stopping the first time p dips under 0.05 inflates false positives.

Before reading any lift, I check the split:

export function srm(visitorsA: number, visitorsB: number, shareA = 0.5) {
  const total = visitorsA + visitorsB;
  const eA = total * shareA, eB = total * (1 - shareA);
  const chi = (visitorsA - eA) ** 2 / eA + (visitorsB - eB) ** 2 / eB;
  const p = 2 * (1 - phi(Math.sqrt(chi)));
  return { chi, p, mismatch: p < 0.01, share: visitorsA / total };
}

A sample ratio mismatch means assignment or counting is broken, so the lift means nothing. 10,000 against 10,400 looks harmless and gives p ≈ 0.005. The usual causes are caching, redirects that drop cookies and bots that slip past the filter. Exposure-on-view adds one of its own: if B’s hero is taller or sits lower, the 60% threshold is reached at a different rate. Keep variants comparable in layout, or observe an element both share.

Takeaways

• Bucket on the server with HttpOnly cookies, render the variant in the HTML and mark the page private, no-store.
• Count exposures when the element is seen, and let the primary key enforce one row per visitor, with conversions attributed through INSERT OR IGNORE … SELECT.
• Check sample ratio mismatch before reading the lift, and size the test with a power calculation, not a minimum count.