[fusion_builder_container hundred_percent=”yes” overflow=”visible”][fusion_builder_row][fusion_builder_column type=”1_1″ background_position=”left top” background_color=”” border_size=”” border_color=”” border_style=”solid” spacing=”yes” background_image=”” background_repeat=”no-repeat” padding=”” margin_top=”0px” margin_bottom=”0px” class=”” id=”” animation_type=”” animation_speed=”0.3″ animation_direction=”left” hide_on_mobile=”no” center_content=”no” min_height=”none”][fusion_alert type=”general” accent_color=”” background_color=”” border_size=”1px” icon=”fa-refresh” box_shadow=”yes” animation_type=”0″ animation_direction=”down” animation_speed=”0.1″ class=”” id=””]Last updated: November 24, 2014.[/fusion_alert]
Optimizely new Free plan: 50,000 unique visitors a month included
Optimizely just announced a pricing change. They got rid of their 3-4 plan choices and instead are offering just 2 plans:
- Starter Plan: free.
- Enterprise Plan: call for a quote.
Details of the new pricing here and their KB article on how it compares to legacy plans here. All in all, this is great as the Starter plan includes 50,000 tested visitors per month for free. You can now do some serious A/B testing using their Starter plan, but if you want to do so without needing more visitors and upgrading to an Enterprise plan, you should keep in mind the following points.
Your main difficulty: minimum sample size with only 50,000 UVs monthly
The main challenge you’re going to face is to reach the necessary minimum sample size for your tests with only 50,000 users tested available. This number is small, and to reach the required minimum sample sizes you will likely need to:
- keep the number of new variations tested against your current version to 1: the more variations you test, the more UVs you need to complete your test
- run tests on the page where you’ll likely see the biggest impact on your current Conversion Rate: landing pages, your Homepage, your Pricing page, etc… The higher the relative performance of your variation compared to the control, the less UVs you need.
You can use my free online A/B testing planning strategy calculator to see how big a jump in performance you need to achieve statistical minimum sample size in your tests. You need to calculate it yourself because it is based on your current Conversion Rate, so these statistics will vary from page to page.
You’ll probably see that under 10% or 15% relative CR increase (for example, if your current CR is 2%, the variation you test should achieve 2.2%/2.3% minimum), then you’ll need more than 50,000 users to achieve the minimum sample size needed. Which leads to my 2nd point:
Use traffic allocation tested at your advantage
The key variable you need to use here is the percentage of your total traffic that you include in your test. Your goal with this setting should be twofold.
1. Avoid pauses in testing. If you have a website with 200,000 UVs monthly, then the 50,000 UVs included in the Starter plan will run out after 1 week of testing if you test on 100% of your traffic. You don’t want to run the same test for 1 week, pause for 3 weeks, and restart it for 1 week again, etc. until you reach minimum sample size. That will bias your results. You should instead run your test on 25% of your traffic all the time.
2. Run tests for at least 1 full business cycle, ideally 2. The other aspect of this 50,000 limitation is that, especially if you are testing Landing pages with huge potential for improvement, you may see statistically valid results in 3 days, which also minimum sample size. But you should not stop your test until it has run through at least a weekly cycle. So if you have quite a lot of traffic, adjust the percentage of traffic tested so that your test with only 50,000 UVs run at least through an entire weekly cycle.
Main limitation: no “Multi-page funnel experiments”
But this isn’t a key limitation, as this is quite an advanced testing format. Importantly, you can still run A/B tests on multiple pages of your website (using URL targeting) as long as you change the same element(s) on all the pages tested. For example, you can test a new header element site-wide or test a new checkout funnel item as long as it appears throughout the whole funnel.
What you cannot do is what is called “Multi-page funnel experiments” in Optimizely, and which is defined as not only running a test on multiple pages, but also changing different elements across the pages tested. You can read this Optimizely Knowledge Base article if you want to get more details on this type of experiment.
The main limitation in Optimizely’s new free plan remains the low number of UVs included, but that’s fair and par for the course for what is now a freemium pricing model.
There are little excuses not to experiment, at least, with this testing platform and set up a first test for free.[/fusion_text][/fusion_builder_column][/fusion_builder_row][/fusion_builder_container]