A/B testing is commonly used in several forms of advertising predominantly in online marketing and web development. It is often called split-testing because sample of users is usually split in two halves or variants A and B.
Variants A (baseline) and B are compared which differ one from another in small aspect that might impact user’s behaviour. Point of A/B testing is finding the predominant version that will result in higher conversion rates.
Say hello to Bestly
Bestly offers an easy way to add A/B testing and staged rollouts to native mobile applications.
The easiest way to add Bestly into your iOS project is using CocoaPods.
Simply create new Podfile or add to existing one
pod 'Bestly' and run
pod install in your terminal.
Then you need to add the following import to your
Sign into Bestly and create a new app. Paste the following line to your
application:didFinishLaunchingWithOptions: and replace
APP_KEY with key that was generated on the website.
Creating your first controlled experiment
You can run experiment with two A, B or three A, B, C variants.
In this example I will show you the basic usage with two variants.
Let’s say that we want to A/B test a button with different title. We’ve got two variants:
- A: Hi!
- B: Hello!
Paste this code and replace
EXPERIMENT_ID with Experiment ID that you can find on the website.
Place the following code to the
UIButton action method so you can see if variation’s goal was reached.
After playing around a little bit on the iPhone simulator we can see that the conversion rate is higher for the button with the title Hello. However to get accurate results, the number of users needs to be bigger than 100 per variation.
Conversion rate and change over baseline
For each variation conversion rate is calculated as:
Where x is number of conversions and n is number of people that have viewed variation.
Formula for calculating change over baseline is basically substraction of conversion B and A divided by conversion value of A.
The response for this blog post was immense. I was lucky enough to get on the the front page of Hacker News resulting in almost 3000+ visitors from 70+ countries in one day—not too bad.
I also received a very special email. It was from James, the founder of Bestly. He thanked me for writing the post and informed me that a new version of Bestly is coming out.
- There’s a new version of Bestly SDK on GitHub
- You don’t need to use
- Design changes on the dashboard
- Other things such as aliases
You can download the project here.
You can send any event you want to track and it wil be tracked for every experiment. ↩