A/B testing iOS apps like a pro

A/B testing is commonly used in several forms of advertising predominantly in online marketing and web development. It is often called split-testing because sample of users is usually split in two halves or variants A and B.

Variants A (baseline) and B are compared which differ one from another in small aspect that might impact user’s behaviour. Point of A/B testing is finding the predominant version that will result in higher conversion rates.

Say hello to Bestly

Bestly offers an easy way to add A/B testing and staged rollouts to native mobile applications.

The easiest way to add Bestly into your iOS project is using CocoaPods. Simply create new Podfile or add to existing one pod 'Bestly' and run pod install in your terminal.

Then you need to add the following import to your .pch or AppDelegate.m

#import <Bestly/Bestly.h>

Sign into Bestly and create a new app. Paste the following line to your application:didFinishLaunchingWithOptions: and replace APP_KEY with key that was generated on the website.

[Bestly setupWithKey:@"APP_KEY"];

Creating your first controlled experiment

You can run experiment with two A, B or three A, B, C variants.

In this example I will show you the basic usage with two variants.

/**
 * Call this whenever you'd like to run an experiment.
 * @param experimentID The experiment ID corresposding to the
 * experiment that you'd like to run.
 * @param a Experiment variation A. This is the baseline
 * variation. Please note that in any sort of failure
 * (e.g. lack of internet connectivity),
 * this will be the variation that is run.
 * @param b Experiment variation B.
 */
+ (void)runExperimentWithID:(NSString *)experimentID
                          A:(void(^)(void))a
                          B:(void(^)(void))b;

Let’s say that we want to A/B test a button with different title. We’ve got two variants:

• A: Hi!
• B: Hello!

Paste this code and replace EXPERIMENT_ID with Experiment ID that you can find on the website.

[Bestly runExperimentWithID:@"EXPERIMENT_ID" A:^{
    // Insert your testing logic here...
    [self.testButton setTitle:@"Hi!" forState:UIControlStateNormal];
} B:^{
    [self.testButton setTitle:@"Hello!" forState:UIControlStateNormal];
}];

Place the following code to the UIButton action method so you can see if variation’s goal was reached.

- (IBAction)testButtonTapped:(id)sender {
    [Bestly goalReachedForExperimentID:@"EXPERIMENT_ID"];
}

After playing around a little bit on the iPhone simulator we can see that the conversion rate is higher for the button with the title Hello. However to get accurate results, the number of users needs to be bigger than 100 per variation.

Conversion rate and change over baseline

For each variation conversion rate is calculated as:

Where x is number of conversions and n is number of people that have viewed variation.

Formula for calculating change over baseline is basically substraction of conversion B and A divided by conversion value of A.

Update

The response for this blog post was immense. I was lucky enough to get on the the front page of Hacker News resulting in almost 3000+ visitors from 70+ countries in one day—not too bad.

I also received a very special email. It was from James, the founder of Bestly. He thanked me for writing the post and informed me that a new version of Bestly is coming out.

What changed

1. There’s a new version of Bestly SDK on GitHub
2. You don’t need to use [Bestly goalReachedForExperimentID:@""]; anymore¹.
3. Design changes on the dashboard
4. Other things such as aliases

You can download the project here.

1. You can send any event you want to track and it wil be tracked for every experiment. ↩