7 min read

Test What Works - A/B Testing in Framer (GA4-Ready, 2025 Edition)

Guessing is expensive. Testing is standard. With Framer’s built-in A/B tests, you can validate real changes (headlines, layouts, CTAs) against real goals - and pair the numbers with qualitative research to explain the “why.” (NN/g). Quick CTA: Roofing owner - want disciplined tests that ship winners fast? Claim your spot. You own the site from day one; we go live in 7 days; ROI signal in 21 days.

By: Lumioforge

Table of Contents

• What A/B testing is (and isn’t)

• What to test first (high-impact, low-risk)

• Framer setup (variants, conversions)

• Wire up GA4 (events, conversions, reports)

• Sample size, MDE, power - practical planning

• Run rules - SRM checks, stop criteria, ship fast

• Example wins - before → after

• Starter plan you can follow today

• Resources

A/B testing in 60 seconds

A/B testing is a controlled experiment: split comparable users across variants, measure a pre-defined success metric, and keep the better version. It’s quantitative (is A better than B?) and works best alongside qualitative methods (usability tests, surveys) to understand cause. (NN/g 101).

Key principles

• One hypothesis at a time. Keep the change small and specific to avoid confounders.

• Pre-register metrics. Primary KPI (e.g., “Get Free Estimate” submits), plus guardrails (bounce rate, form errors).

• Decide the bar up front. Minimum Detectable Effect (MDE), power (often 80%), alpha (often 0.05). Use a calculator to plan (Optimizely, Evan Miller).

What to test first

1. Hero headline + subhead - clarity beats clever.

2. Primary CTA - label (“Get Free Roofing Estimate” vs “Contact Us”), placement, repetition.

3. Form friction - step count, phone-first vs email-first, autofill, error copy.

4. Social proof - star-rating snippet, “jobs near you,” review count.

5. Offer framing - “Launch in 7 days. You own the site.” vs generic pitch.

These hit the most traffic, so even modest lifts compound. (NN/g).

Soft CTA: Want a ranked, high-impact backlog for your roofing site? Claim your spot.

Framer setup that avoids false wins

Create the experiment (no code):

1. In Framer, right-click the page → New A/B Test.

2. Make edits to Variant B (add up to five variants).

3. In Configure, set your conversion step (tracked link, form submit, or page view with a tracking ID that’s consistent across variants).

4. Start and publish; Framer splits traffic evenly (e.g., 50/50). Guide: How to run an A/B test on your Framer site.

How Framer calls winners: Framer uses a Bayesian model and shows Probability of Being Best; when it exceeds ~90% with enough traffic, you can declare and ship the winner (see guide above).

Important note (attribution window): Framer’s A/B tests are privacy-friendly. For funnels spanning multiple days, wire events into Google Analytics 4 (GA4) so longer windows and cross-channel attribution are captured.

Wire up GA4 (events, conversions, reports)

Add GA4 to Framer:

• Paste your GA4 Measurement ID in Site Settings → Google Analytics Measurement ID and publish; or install via Google Tag Manager (GTM) using Custom Code fields (head/body). Guides: Set up Google Analytics, Add Google Tag Manager.

Name events consistently:

• Prefer GA4 recommended events (e.g., generate_lead for quote requests) to unlock reports automatically; otherwise create custom events. (GA4 event reference).

Mark conversions (key events):

• In GA4, mark your lead event (e.g., generate_lead) as a Key event (conversion), then validate in Realtime/DebugView. Docs: Set up a key event, About events.

Tip: Keep the same tracking ID or form endpoint across variants so GA4 aggregates correctly, and use UTM parameters on paid traffic to segment experiment performance by channel.

Sample size, MDE, power - practical planning

Estimate sample size & duration before launch. Smaller MDEs need more visitors; lower traffic means test bigger changes first. Use: Optimizely’s calculator, Evan Miller’s tools, or ABTestGuide.

Rules of thumb: pick an MDE that moves the business (+10–20% relative on lead submits), target 80–90% power, 5% alpha, and include at least one full weekly cycle.

Run rules - reduce bias, ship winners fast

• SRM check: If observed traffic splits differ a lot from intent (e.g., 60/40 on a 50/50 test), treat results as invalid and fix bucketing/eligibility before reading outcomes. (SRM explainer).

• No mid-run edits: Don’t change variants while the test runs; stop, edit, restart (see Framer guide).

• Stop criteria: Pre-define “run until N users/variant or 2 weeks, whichever is later” and wait for Framer’s “probability best” to stabilize.

• Rollout: Ship the winner with a small holdback (e.g., 5–10%) for a week; monitor regressions via GA4.

Example wins - before → after

• CTA label clarity

  • Before: “Submit”

  • After: “Get Free Roofing Estimate” (matches searcher intent; higher clarity)

• Form friction

  • Before: 8 fields, single step

  • After: 2-step, phone-first, inline errors; GA4 generate_lead fires on thank-you

• Social proof placement

  • Before: Reviews buried on subpage

  • After: “4.9 ★ (327 local reviews)” under hero CTA + badges

• Offer framing

  • Before: “Redesign available”

  • After: “Launch in 7 days. You own the site. Hosting included.”

Starter plan you can follow today

1. Define success + guardrails Primary KPI = quote requests; guardrails = bounce, form error rate.

2. Instrument GA4 Add Measurement ID (or GTM), create/mark generate_lead as a key event; validate in DebugView. Guides: Set up Google Analytics, GA4 key event.

3. Prioritize the backlog Start with headline/CTA → form friction → social proof.

4. Launch Framer test New A/B Test → set conversion step → start and publish; watch for SRM and let Bayesian “probability best” converge. Guide: Run an A/B test in Framer.

5. Promote winners fast Replace control, publish, keep a small holdback; document learning and queue the sequel test.

Closing CTA: Want us to set up Framer experiments, wire GA4 properly, and ship winners without breaking UX? Claim your spot. Live in 7 days - ROI signal in 21 days - and you own the site from day one.

Resources

• Framer - A/B tests (setup, variants, Bayesian winner, privacy model): How to run an A/B test on your Framer site

• Framer - Analytics integrations (GA4 & GTM): Set up Google Analytics, Add Google Tag Manager

• Google Analytics 4 (events, key events/conversions): Recommended events, Set up a key event, About events

• A/B testing primers & calculators: NN/g: A/B Testing 101, Optimizely sample size, Evan Miller calculators, ABTestGuide

• SRM (Sample-Ratio Mismatch): Statsig SRM guide