Why should you use this calculator to validate your running or ended test results?
First, because the vast majority of the testing tools make it hard to compute the actual minimum sample size you need to validate your running or ended tests. With this calculator, you just have to input your raw test results and the minimum sample size you need to reach is computed automatically, along with how many Unique Visitors you still need to test to reach it.
Then, if you complete the start date field, it also gives you the estimated date when you’ll reach this minimum sample size. It also gives you the impact on your monthly revenue, if you fill it.
It also test your results for statistical significance against 2 different statistical measures, so you can see if your raw results really mean something (i.e. are statistically significant) or if they are just random data that happens to give the impression of a substantial difference between your control’s performance and your variation’s performance.
When should you use this calculator?
During your tests, use it to get a feel of how your test is going. You’ll get an immediate snapshot of whether or not you’re getting close to significance and how many days you’re going to have to wait to get valid results. If after a week, you’re still needing to wait for 100 days to reach a minimum sample size at 80% power level, then you either have not enough traffic to test meaningfully or your variation’s performance is too low. You’re better off stopping and running another test.
When you’ve stopped a test, use it to validate that your results are statistically significant and that you’ve indeed reached the correct minimum sample size calculated from your test data.
When you get test results from your agency or consultant, ask for the raw test data (just 4 numbers, easy for them) and run it through this calculator. You may be surprised to find that the test result sent by your agency does not reach the required minimum sample size or is statistically significant. In short, the results they sent you do not mean anything.