Why should you use this calculator while planning all your tests?
Because you should only start tests that have a good chance of providing you with statistically valid test results within an actionable timeframe, and many otherwise interesting A/B tests have no chance of ever achieving this. This calculator helps you to easily and simply validate the business case of running the tests you are envisioning.
How to use this calculator?
Simply enter the parameters of the test you are planning in the input boxes. Indicate the number of variations for each element tested as a multivariate element on the page(s) (up to 4 different elements) and the total number of variations is computed automatically. The Baseline CR is required and the last 2 parameters are optional but strongly recommended. The calculator updates immediately with the planning data you need to use.
Here is how you read and use the results. Each percentage in red indicates a possible performance level of your best variation against the control version. Some key levels are indicated here, but any level in between is possible of course. Then, for each level indicated, you will get the number of Unique Visitors you need to complete the test while reaching the Minimum Sample Size required for statistically valid results. You also get the number of days you will have to wait for the test to finish.
If these concepts are unclear, please read my post How long to run an A/B test for? and come back here afterwards 🙂
Why is it critical to do this before you start any test?
Because as you’ll quickly realise by analyzing your own numbers, many tests that you would want to run simply won’t complete within an actionable timeframe, and are thus a waste of time and resources. Or they would complete, but only if the performance of the best variation is above +100%, which does happen, but is very, very rare and a sign your website is really under-optimized to start with.
You need to make the most of your testing resources, and only run tests that have good chances to complete and will improve your revenue by the most.
How to incorporate testing quotas?
A common question I have from smaller clients who are using their own A/B testing tools on limited plans is: how should we use this calculator while accounting for our testing quota? You can do it very easily: simply input your monthly quota instead of your Unique Visitors number if your quota is less than your monthly UVs. This will give you the delays while taking into account your testing quota limitations.
Please note: for the mathematical equations and parameters used in this calculator, please look them up here. Also, using this form is 100% secure and confidential, as all computations happen in your browser and no data is transmitted to me or anywhere else. This means you cannot save it though; if you need to share the results, please take a screenshot.