Measurement
How Fulcrum measures the number behind your guarantee.
A controlled comparison with a cumulative measurement.
The holdout lift: measured against a randomized control group, continuously measured.
The method
Measuring with a constant control group.
When Fulcrum reports a guaranteed value, it isn't a count of clicks and orders - it's the measured difference in revenue per user between shoppers who get Fulcrum's personalization and a small, randomized group who don't. We call this the holdout lift, and it's the number your guarantee is based on.
The mechanism
A randomized holdout, running for your whole contract.
-
Split
Your traffic is divided at random. Most shoppers get Fulcrum's personalization; a fixed 10% holdout never sees it. The split stays locked for the life of your contract.
-
Measure
Both groups are tracked continuously across your whole site: conversion rate, average order value, and revenue per user, collected from day one onward.
-
Compare
At the end of the 12-month term we calculate the difference in revenue per user between the two groups and a refund is issued if the revenue generated by Fulcrum is less than the fees you paid.
Timing
Ongoing control group measurement.
To prove the results are real, we hold back 10% of your traffic as an untouched control group, they never see Fulcrum's personalization, while the other 90% (the variant group) does.
The 45-day pilot is a separate window from this 12-month guarantee term - see how the pilot works .
Significance
How we approach the data.
Before we validate any result, we wait until we've collected enough orders and traffic to trust the data, early numbers can be noisy. Then we run two independent statistical checks, and both have to confirm the result before we call it real. This keeps the validation honest and resistant to false alarms.
Scale
What this means for higher- and lower-traffic sites.
Higher traffic fills both sides of the split faster, which narrows the range around the measured lift, so the guarantee can carry a tighter margin. Lower traffic widens the range, so the same calculation yields a more conservative number. Every account uses the exact same method; what differs is how tight the range is, not the formula.
No separate minimum floor. No fallback metric. Every qualifying client gets a guaranteed number calculated the same way - just with a wider or narrower margin depending on traffic.
Common questions
Questions about the method.
Is this based on clicks and page views?
No. Engagement data like clicks and page views informs how we tune your algorithms internally, but it never factors into your guaranteed figure. Only the holdout comparison does.
Why don't you show me results sooner?
We can show you directional trending in conversion rate, AOV, and revenue per user throughout your contract - but the guaranteed figure itself is only calculated once, at the end of the 12-month term to identify if there is any refund required by Fulcrum.
What if my traffic is on the lower end?
You still get a guaranteed number, calculated the same way as every other client. It will simply be more conservative, reflecting the wider confidence interval that comes with a smaller sample.
How is this different from a standard A/B test?
The mechanics are similar - a randomized control group and a statistical significance threshold - but the holdout runs continuously for your full contract term rather than a fixed test window, and the result is used to calculate a dollar-denominated guarantee, not just a directional read.