A/B Tests are a great way to accurately measure the impact that Chameleon is having and compare different versions of a product tour.

Chameleon makes it very easy to run a Control Group (Chameleon vs. nothing) or A/B Test (two different Chameleon Tours). We recommend starting with a Control Group test and iterating until your Tour is effective, before testing Tour variations. 

Note: A/B Testing is most relevant at scale and is available only for Growth plans. If you feel like you could get value from using this feature, send us a message.

There are two ways to deliver Chameleon Tours and the testing we describe here is relevant for Announcement Tours only. 

Once you have selected the Announcement option, you will be able to turn on Control Group testing and set the proportion of users that should be part of the control group -- as in the image below, where a random 20% of the target audience users will not see this Chameleon Tour.

Every user identified by Chameleon is automatically assigned a random number ("Testing ID" property) with a value between 0 and 100. This value is persistent for each user and can be leveraged for targeting and experimentation.

To select a control group, Chameleon randomly chooses a number between 0-100 as the start of the range for the group. The size of the group will determine the end range for the group (including going back to 0 after 100.) The corresponding users within this range will form the group.

E.g. if you choose a 20% control group for a Tour, the Testing IDs for the group could be 87.3 to 7.3. If you do this for another Tour, the group may be 12.9-32.9.

If you later increase the size of the control group, then the range is simply increased, from the same starting value.

An event will be logged whenever a user is first identified to be within the Target Audience, and thus is liable to see this Tour. This event* -- "Chameleon Experiment entered" -- will be available within all your connected analytics integrations.

Note: This event does not take into account URL Rules or other conditions for the Experience, only when a user first matches the user Segment's conditions.

💡 Within Mixpanel, the event name is "Experiment Started" to better match Mixpanel's experimentation analysis framework.

For this event the following properties will also be logged:

The "Group" property values are either:

Other events (e.g. "Tour started") will also be logged as normal once a user starts interacting with the Chameleon experience. You will see all these within your analytics platform (e.g. Amplitude, Heap, Mixpanel, Google Analytics) and can use this to further analyze the conversion or relative impact of your Chameleon experiment*. 

📩 It is within our plans to deliver and improve reporting within the Chameleon Dashboard to measure success and goal conversion, let us know what you'd like to see here.

Chameleon also lets you manually show an Experience to a random proportion of your Audience, so that you can show two different Tours to different groups or group multiple tests to the same random sample. 

To create a Test Group, simply add an extra "sampling filter" filter to your Segment that selects a random sample of users of the desired size. This means you can still target users based on other conditions, such as user properties, events, data sources, etc. 

To add the sampling filter:

In the above example, users that have a Testing ID value between 50 and 100 would be targeted. This would constitute 50% of the users within the group defined by the other segment filters. 

To target 10% of users, you could use either:

You could also use multiple filters to target another 10% group (e.g. Testing ID more than 10, and Testing ID less than 20). 

You can use the same filter configuration in another Segment (by re-creating the filter) to target the same user group. This means you can run multiple A/B tests on the same user group. 

To test two variations of a Tour:

The results from these experiments can be analyzed through the Chameleon Dashboard or within any connected analytics integrations.

You will be able to use the Tour name to segregate the data, as each Chameleon event contains additional attributes, such as Tour name, User ID, URL, etc.