More than most people send. Telling a 6% reply rate from 8% takes about 2,554 emails per version. Put in your own reply rate and volume to see how many you need and how long that takes, or check whether a test you ran actually won. Free, no signup, nothing leaves your browser.
Example numbers. Replace them with yours.
Your numbers
Confidence95% is the usual bar.
PowerThe chance of catching a real lift that size. 80% is the usual bar.
Emails you need
2,554
per version, to tell 6.0% from 8.0% at 95% confidence and 80% power.
At your volume this test takes about 11.4 weeks.
In 65 sending days (about a quarter), this volume can catch a lift of 31% or more: 6.0% to 7.9%. Changes smaller than that will look like noise. A week here is five sending days, with every send going to the test.
A test can only pick between the emails you wrote.
Monty writes each email for the one person reading it, sends it from your own mailbox and answers the replies.
Start with your listHow it is worked out
The calculator uses the standard sample size formula for comparing two proportions, with a two-sided test:
n per version = ( z₁ × √(2 × p̄ × (1 − p̄)) + z₂ × √(p1 × (1 − p1) + p2 × (1 − p2)) )² ÷ (p2 − p1)²
A worked example. These are example numbers, not industry figures. A team with a 6% reply rate wants to know if a new version reaches 8%. At 95% confidence and 80% power that needs 2,554 emails per version, 5,108 in all. Sending 30 a day from 3 mailboxes, that is 57 sending days, about 11.4 weeks. If they judge on positive replies instead, and those run at 1%, finding 1.5% needs 7,750 per version: 173 sending days at the same volume, which is longer than most offers stay the same.
Did it win? The second tab runs a two-proportion z-test: z = (pB − pA) ÷ √(p̂ × (1 − p̂) × (1/nA + 1/nB)), where p̂ is the reply rate of both versions together. Version A gets 25 replies from 500 (5.0%), version B 35 from 500 (7.0%). That reads as a 40% lift, but z = 1.33 and p = 0.183: not a win at any usual confidence. Confirming a gap that size would take about 2,213 emails per version.
The first term uses the average of the two rates (the pooled form), the same spread the z-test uses, so the sample delivers the power you ask for. Calculators that use the baseline rate there instead give smaller numbers for large lifts. There is no continuity correction, which would only make the numbers larger. The duration assumes every send goes to the test.
Usually thousands per version, not hundreds. To tell a 6% reply rate from 8% at 95% confidence and 80% power, each version needs about 2,554 emails. To tell 1% positive replies from 1.5%, each version needs about 7,750. The lower the reply rate and the smaller the change, the more emails it takes.
They stop too early. With a few hundred sends a version, a gap of a few replies is well within luck. In our example, 25 replies from 500 against 35 from 500 looks like a 40% lift, but a gap that big turns up by chance about 18% of the time (p = 0.183).
Test bigger changes. A new offer, a different audience or a different reason to write can move the reply rate enough to show up in a test you can afford. A new subject line or one changed sentence usually cannot. If even a big change would take months, pick the version you believe in and spend the volume on the people you are writing to.
Both use the standard formula for comparing two proportions. Woodpecker's, like Evan Miller's, uses the baseline rate's spread for the "no difference" part; this one uses the average of the two rates, which is what the significance test itself uses. For a 50% lift on a 1.5% baseline that gives 5,134 per version here against 4,414 there. The gap grows with the size of the lift. Use the larger number if you want the test to deliver the power you asked for.
Whichever you would act on. Positive replies are the better measure of whether an email works, but they are rarer, so a test on them needs many more emails. Out-of-office and auto-replies are not replies; take them out of both versions first.
No. The calculator runs in your browser and sends nothing to a server. Once you change a number, the inputs go into the page address so you can share the result; nothing else keeps them.
Before you book
The price
$1,399 a month. No setup fee.
Prices shown in USD. Billed in the same currency at checkout.
If it does not work
Cancel inside 30 days and your payment comes back in full. No usage deduction, no form to fill in.
How you leave
Monthly, and you cancel from inside the product. No contract, no notice period, no minimum term — the tools this replaces mostly ask for a quarter or a year up front.