A/B Testing can now be set on project level
If you used our platform before, you know you can set A/B testing percentages on our Customer Journeys and Campaigns. The way this approach works is when a visitor is on your website and is about to see a touchpoint (because the audience targeting is met), a randomizer script will determine which A/B testing label the visitor will get. The change of which label the visitor gets, is the percentage you set in the Customer Journey or Campaign of the to be shown touchpoint.
After the visitor/profile got their A/B testing label, they keep this label forever, and the label is one of the following two: Show or Hide. Profiles with the label Show always see Datatrics’ touchpoints, the profiles with the label Hide never do. The only exception is when you create Campaigns with a 100% A/B testing setting.
It doesn’t matter if a touchpoint is shown, or it should have been shown to someone with a Hide label, we always send an event to ourselves and Google Analytics at this time. This is so you can see the performance of a touchpoint. You only want to compare people who saw a touchpoint to people in the same context, the ones who should have seen it, not the total group of people with the label Hide.
It’s great for comparisons, but there is a problem with doing this on Customer Journey and Campaign level. As mentioned above, profiles keep their A/B testing label. So if you create a Campaign for new visitors to show them a touchpoint to greet them, and you set the A/B testing percentage to 90%, 90% of all new visitors will get the Show label. But if you also created a Customer Journey with an A/B testing setting of 50%, by the time profiles see a touchpoint of this journey, they already have a label. And 90% of them got the Show label.
So what have we changed?
We moved the A/B testing settings to project level, so the situation explained above doesn’t happen anymore. When a profile is about to see a touchpoint, the change of which label they get is determined by one project level setting. The way our A/B testing works doesn’t change, but it will be a lot clearer and you won’t see weird distributions like the situation above (90% Show, with a setting of 50%).
We know a lot of our customers use different A/B testing settings for a phased rollout, like different domains or languages. This is why we included an extra setting to be able to change the percentage based on URL. You can still have different A/B testing percentages, but make sure the groups of profiles don’t overlap and you enter the same situation as explained earlier.
We are certain this will make the A/B testing experience with Datatrics a lot clearer. You can find the new A/B testing setting over here. Existing users can still keep their setup for A/B testing settings on the Campaign and Customer Journey level. From the moment you start using A/B testing settings on project level, this is irreversible. From April 1st 2021, all projects will automatically be transitioned to the new project level settings, so make sure you start using it in the coming months. We will provide you with more information by that date.
If you have any questions, feel free to contact us.