This is the story of ABBY.io, an evaluation and documentation service for A/B testing. It was a side project I invested much time in effort in, but in the end, it never took off, and I pulled the plug.
Let’s dive into ABBY’s history and the steps that lead to its eventual failure.
The original idea was born out of necessity. I wanted to approximate how many users were needed for an A/B test at our company. But all I could find from former tests was a lousily maintained Google Sheet that didn’t help me at all.
Some days later I sat at the airport waiting for my flight to PyData London. With some time until the flight went off I began sketching a prototype of a documentation service. After I laid the groundwork, it became clear that the whole thing is not overly complex. It was reasonable to have a working prototype when returning from the conference.
So I spent my two evenings in London writing the first rough version of ABBY – what a creative name, right?
When I returned to the office I showed it to my colleagues and they encouraged me to go on with it and polish it. After using it for the first A/B tests we realized ABBY would be way more useful, if it could also handle the evaluation part. Until then all people evaluating A/B tests in the company used their method of choice. No standard script existed, only custom R scripts and Python files.
In the course of weeks, ABBY became an indispensable tool for our A/B tests. And the more I worked on the project the more convinced I got that other people must have the same problems.
ABBY for everybody
I talked with my CEO about extending ABBY to a software-as-a-service. I truly appreciate that he didn’t stop me, but gave me the OK to work on it. Immediately after getting home from work that day I purchased a domain and a Bootstrap theme. (Learning 1: Never buy a domain before you need it). The initial plan was to just add a user concept around the existing code.
In retrospect, I’m not surprised that I ended up rewriting the whole thing from scratch. There are so many things that a SaaS product requires that an internal company tool doesn’t need. The good thing was that I could avoid a lot of the design mistakes I made in the first attempt. I knew the bottlenecks and problems from the first implementation. And I was eager to ship quality software that didn’t feel like a hack. (Learning 2: Lean is king: you have to be ashamed of your initial version.)
Launch and gaining traction
My initial plan was to release ABBY.io – again, very creative, I know – as soon as possible. But I wasn’t satisfied with the result and decided that I needed to add one more feature. Again and again. It felt like an endless cycle, where I sat at home and implemented new features, tweaked the design or thought about possible next steps.
By the end of January 2015, after working on the project for more than half a year, I was sick of waiting any longer and put the website live. I tweeted about it and posted a link to Hacker News. Both gave me some initial traffic. There wasn’t much traffic from HN and it didn’t take long for the link to disappear from the start page. But it was enough to attract some people and convinced them to sign up.
It turns out there was some real interest in ABBY.io!
I was lucky enough that I got featured by Product Hunt, too. That got me massive traffic and a good number of signups for a few days. Additionally, I set up an AdWords campaign and spent some bucks to gauge further interest in the product. About 10% of users that clicked the ad signed up. (But admittedly that was favored by the fact that ABBY.io was for free while in beta).
The slow death
So what happened? Why did I shut it down instead of celebrating that I launched a project many people would be happy about?
After being happy to see all the signups come in I took a look at their usage behavior and that’s where I hit a wall. Only three customers were actively using the product, the rest dropped off after a page view or two. 97% of my user base was dead. Worse, they never were alive.
There are three main reasons why I think this endeavour failed:
1. User education
It turned out the problems ABBY.io addressed are real and many people face it. But they don’t know they face it. If you start a project you can’t afford to spend time on user education. Many people I spoke with ran test after test without any documentation; very few realized the benefit of a service like ABBY.io. When someone finds your product they should immediately be able to understand the problem for which you offer a solution.
2. Use case
ABBY.io was a niche product. You couldn’t run A/B tests with it. You were “just” able to evaluate tests and document them so you don’t lose valuable knowledge and insights. This reduced the potential audience enormously. Lots of companies use Optimizely or Google Analytics to run A/B tests. These tools already offer the evaluation part (although not the documentation part). Why should these users add another tool, if they don’t see its benefit?
3. Usability and Onboarding
So obviously ABBY.io couldn’t pick up users expectations. And it really didn’t surprise me once I tried to see the website from their perspective. After signing up you were presented with a white screen and a message to set up your first test. Great, most people actually clicked that. But what happened then was a UI/UX nightmare: a form with about 15 fields (only 2 or so mandatory), without any explanation. And only after that, you could start evaluating your test. Sure, I would also close the browser tab as soon as these abandoning users did.
So I decided to overhaul the flow to create a new A/B test. You could start with an empty test, import existing results or start with an evaluation of a finished test right away.
The idea was probably great and might have saved ABBY.io, but it was way too large for a side-project. I was lucky if I could spend 5-10 hours a week on the project. Implementing this flow took me weeks. And some weekend I just couldn’t find the motivation to continue. And that was the real reason ABBY.io shut down: I just couldn’t find motivation to finish this amount of work for a change that may or may not turn the product around. I simply put too much on my plate for a project with very limited resources.
But I learned an important lesson on the way: Start small and continue small. Never put more on your plate than you can eat. Or in case of a side-project: break down your work on easily achievable chunks.
The most important thing is to stay motivated. And nothing is more frustrating than working on a project for a year before realizing that nobody cares about it.
There are many guides out there how to approach a side project. But the most crucial thing is to gather feedback as early as possible. You won’t lose users with an ugly interface as long as you provide value to your customers.