So we were asked recently “What’s your experience with A/B testing on iOS?” After quashing our instinctive retort of “Hah! They’ll be ice skating in hell the day there’s enough time and budget to ship ONE version of anything properly, never mind TWO!” we, well, weren’t left with much of anything. Ring any bells there, dear fellow contractors? Yeah, thought so. So let’s go looking for what tools are out there for that sort of thing, shall we?
In case you haven’t encountered the concept before, A/B testing (aka split testing aka bucket testing aka optimization testing) is essentially randomized focus group surveying; do two (or more, in which case it’s called MVT, multivariate testing) versions of your ad campaign or your landing page or your checkout screen or whatever, metric up whatever your success criteria are, and serve up the different versions to different users and see which version metrics out better. Rinse and repeat.
In the web world, this is a widely used practice and there’s no shortage of tools to help you out with it, from to Adobe® Test&Target™ to Amazon A/B Testing Service to Optimizely to SiteSpect to Visual Website Optimizer and oodles of others. (And as an interesting aside, check out How to Increase Your Mobile App Sales With A/B Testing and Increase iPhone App Downloads by A/B Testing App Names for tips on addressing A/B testing for all your App Store materiel — name, icon, description, and screenshots — by way of selectively advertised landing pages. Neat idea, that.)
In the iOS app world, not so much; until researching this we’d only ever heard of the idea being used extensively in the context of Zynga’s development practices. But a quick Google immediately turned up a flurry of results for Clutch.io:
Clutch’s native mobile A/B testing tool helps you to choose the best way to present information to your users. It works by showing users different variations of your app, and then measures the effectiveness of each variation. Until now, app developers have been forced to build their own tools to do this kind of testing. With Clutch’s A/B testing tool, you get an easy yet powerful platform to start running these tests in minutes…
Well, there we go then! Or … not so much.
Clutch.io is winding down – for more information about the future of Clutch.io and how you can smoothly transition your apps, please see our announcement.
They did open source it so if you want to host your own custom A/B framework server , why there it is at clutchio/clutch along with all the other pieces of the puzzle. But many of us are probably looking for a hosted solution I expect; so let’s keep looking. And, rather to our surprise, there is a veritable plethora of them out there:
Apptimize: $50/monh up to 1M installs
Arise.io: free open beta
Artisan Mobile: $1000/month up to 25K users
Heatma.ps: $18/month up to 10K samples
Leanplum: apply for beta
Optimimo: $49/month up to 10K users
Pathmapp: $19/month to 2K users
Swrve: by request, apparently
And that appears to be an exhaustive list of hosted services … this week. If none of those look your thing and you want to roll your own with your different flavour of analytical choice or whatever, besides the above mentioned clutchio full stack collection there’s a variety of other projects on github to get you started too:
An iOS Library to support A/B Testing remotely defined values in Apps and reporting back to a server. Includes native controls for A/B Testing UIButton text and background image. Includes generic ABTestCase for testing any NSString value…
This is an Objective-C wrapper for the Conductrics service, which provides an API for bandit-style optimization, dynamic targeting, and A/B testing.
Switchboard – easy and super light weight A/B testing for your mobile iPhone or android app. This mobile A/B testing framework allows you with minimal servers to run large amounts of mobile users.
SkyLab is a backend-agnostic framework for multivariate and A/B testing … integrates easily into any existing statistics web service. Depending on your particular needs, this may include posting to an endpoint in test blocks, or perhaps setting an HTTP header for a shared API client…
Remote configuration and A/B Testing framework for iOS. Documentation available online.
And … nope, run out of links finally. That was particularly autodidactical of a post, we’d had no idea this A/B thing was such a hotbed of development frenzy. As always, if we managed to overlook something Dear Readers, let us know in the comments!
Bestly: A/B Testing for Native Mobile Apps
9 Mobile A/B Testing Tools – Optimize Apps Without Submitting an Update
TestNest – A-B test app meta-data before the App Store release
I Spent All Summer Running A/B Tests, and What I Learned Made Me Question the Whole Idea
Continue Reading →