A look back: our company’s first 3 months and first 1000 user registrations

In the last days of January, the number of Plumbr users exceeded an important milestone – we now have more than 1,000 users! As we started offering Plumbr as a public beta in the beginning of November last year, we can say that „collecting“ the first 1,000 customers took us 3 months.

Let’s dive into the numbers (we are big fans of factual proof!), and elaborate on:

• What did we do to get here?
• What have we learned from the customers and where are we planning to head with product development?
• What are we going to do to reach 10,000? 100,000? Bazillion?

The first users of Plumbr

The beginning was easy. The founders were working in a large IT service provider Webmedia and throughout their day to day Java development jobs experienced tens of different memory-related problems every year. For reasons still unknown the founders became respected specialists in the company and got bogged down by memory troubleshooting issues of different projects. As this got boring to handle manually, they went ahead and created a solution to automate it.

Hence, the first users where the tens (hundreds?) of Java projects developed within Webmedia. The memory patterns provided a very valuable pool to study, and to develop the first version of the leak detection and reporting algorithms. When the internal beta version was launched, we advertised the tool to the Java developers of the company and recruited the first 40 users.

Now to the sad part – those users did not have many problems with memory leaks at that moment. We received data of… well, 2 memory leaks. We didn’t feel comfortable with that knowledge base and started looking for ways to get more data.

Getting out to the wild

Where does one find out whether the problem they are solving is real? Apparently you can Google for it. Or – to be more precise – you can check the Google Keyword Tool. It told us that 135 000 searches for „java.lang.OutOfMemory“ are fired every month. Wow. We had hit the jackpot!

But since we are from Estonia, and by common belief Estonians are slow by nature, we didn’t dare to raise a huge pile of money and pour it to Google’s pockets in exchange of targeted Adwords ads. Instead we decided to grow at a rate where we could interact with the users and improve the product based on their feedback.

Still, it takes months to get the initial PageRank and start receiving organic traffic. How can we enforce this without paying for the users? By luck, one possible solution found us by itself. A friend posted a link to our web page in the Hackernews channel. We only found out about it because the Google Analytics Real Time window (requires login, but is an awesome feature!) suddenly started showing intense traffic. So we picked up on this and posted a link to reddit/r/programming ourselves. Voila – we got 2,000 unique visitors between November 8th and 10th, 2011. Those 2,000 first visitors converted into 194 registrations and 146 downloads.

From there on, until the end of January (when we hit the 1,000 registration mark) we’ve had 26,000 unique web site visitors. This gives us a 3,8% visitor -> registration conversion rate. Not the best conversion rate out there, but at least we had found some channels that we could use for building awareness of the product, and reached the growth rate that we had dreamt of in the beginning.

So, where did these 26,000 unique visitors came from?

The graph below is based on data gathered by Google Analytics. It is a bit skewed: for example, a Hackernews post is automatically tweeted by several bots, and top posts receive extra tweet coverage – and that is why some visitors who were directed to the site via Hackernews, are actually shown as Twitter (t.co) referrals. Nevertheless, I think that the distribution is a pretty good illustration:

It has not been only us advertising in those sites. Unproportional advertising of your own site would conflict the policy of several of them and we would probably be quickly banned as spammers. Fortunately, there have been other visitors who liked our content and shared it, so I think more than half of the blog posts have actually been featured in at least one of those link sharing sites.

If you know of any other communities to whom our content would be interesting, please let us know in the comments. Or even better – share the post yourself! (We will pick it up in the referral statistics, don’t worry)

How (much) do people actually use Plumbr?

Of course, getting a thousand registrations is great, and certainly marks a milestone. But a clever reader has wondered since the beginning – how much have those 1000 users actually used the product, and what might be the bottlenecks they face that inhibit larger usage? Let us look at the numbers and elaborate on this next.

But first – for people who have not used Plumbr before – this is a quick overview of how your experience with Plumbr typically flows:

1. You enter an e-mail address on our home page
2. We automatically assemble a personal download package for you and immediately send out an e-mail with the download link
3. You click on the link, download the zip file, and see installation instructions
4. You unzip and attach Plumbr to your Java server or application
5. You start your Java app and let Plumbr monitor it. With some luck, your application is leak-free and you can breathe relieved. Sometimes Plumbr discovers a leak or few, then it composes a report saying what exactly leaks in your code and gives all the info you need to fix the leak.
6. If you like the product and trust it works well, you leave it attached, and let it take care of worrying about memory leaks

The above effectively describes our usage funnel. And our goal, naturally, is to direct as many users as possible into the later stages of the funnel.

We get feedback from the users in three ways. All of them provide us different data, and for reaching true insights, we need to combine all three.

1. User-initiated feedback in the forums, Skype support channel, and support@ emails. This source gives us qualitative data on how people use Plumbr. Or rather – why they don’t use it. Most of the feedback reports problems with installation or usage.As of today, we have received 100 e-mails, there have been 13 questions in the forums, and around the same number of people have contacted us via Skype. We haven’t measured, but I estimate that we solve around 70-80% of the complaints, and the rest are either mysteries that dissolved themselves or represent unsupported environments that we have yet to extend Plumbr to.
2. Feedback initiated by us. I have followed Rob Fitzpatrick’s advice on personally contacting the first users to get to know our customers. Out of the first 1,000 registrations I have personally e-mailed to more than 900 (some of the rest I have met in person, some contacted us first, and some I just overlooked in the list and missed to contact). I have got around 70 replies. (Anybody out there with more experience – is this good number?)Around one third of them say „indeed I downloaded Plumbr, have not tried yet, but hope to do it in the near future“. Another third praise Plumbr, and say that Plumbr didn’t find any leaks. And the rest like Plumbr and also found a memory leak (albeit some only tested it with the packaged leaking demo app).
3. The third source is purely quantitative and connected to Plumbr usage. As Plumbr is software that you download and run on your own machine, it is a bit tricky, but it is still possible to get some feedback. At launch, Plumbr dials home to check for version upgrades. While the main idea is to alert people when a newer version is out (being in beta we might need to release fast bug fixes), we also get good data of which versions of Plumbr people run and how many people of the ones who downloaded have actually started to use it.We also get a note when people open a leak report (there is a message on the report explaining that since they use a free evaluation copy they need to notify us of receiving the report before they can see the contents).

Let’s visualize the data we have collected:

It seems that out of 1,000 registered users 263 actually started the tool. Shocking, isn’t it?

There is one technical catch though. If we look at the report opening data, we see that 18% of the „open report“ events come from people who have never sent us the „run Plumbr“ event. The reason is that some of the users use Plumbr on servers that forbid outgoing traffic, and the release upgrade check never reaches us.

Assuming that people behind firewalls get (and open) leak reports with the same distribution as the ones who are not, let’s take that 18% into account when analysing our funnel:

Still, shocking. What could be the possible reasons of the low 31% conversion rate?

From the qualitative data we estimate that a whole 30% of people intend to try Plumbr (they download it), but never do because something distracts them and they just add it to their task list.

A fair amount of users probably need Plumbr in an environment that is not supported yet. Many people have asked for Java 5 support (this is one of the top priorities in the list), but we have also received reports of Plumbr not working on Websphere and some Glassfish vesions, with Groovy and some environments.

There are definitely also users who just don’t know how to modify JVM start-up parameters and for whom the installation instruction  is too complex. Some of them contact us, but every participation (in)equality study (see e.g. the 90–9–1 principle) shows you that 90% don’t. Let’s use a best guess and accommodate these assumptions on our funnel graph as well:

These are the biggest barriers we could think of, and that were supported by the qualitative feedback. If you think that we have missed something important, please let us know in the comments below!

What can we do to raise the conversion rates?

We have several ideas on what to do to raise the „new visitors->registrations“ conversion on our web page. We need to improve the landing pages (the home page and the blog post page, where more than 90% of the people arrive to plumbr.io). We need to make a better and more clear offering, explain better what Plumbr is, and publish references and testimonials of current users. We are also planning to A/B test different value proposition and Call-to-actions (I will write a separate blog post on this). We follow closely Peep Laja’s blog  on website conversion and have more ideas than resources to try them out.

However, the web conversion rate is only the tip of the iceberg. What we really need to improve is the conversion to product usage. Without it, two-thirds of every effort made to increasing web site traffic or traffic->download conversion would be lost. And without a bigger user base we might take wrong product development decisions.

From the reasoning above we concluded that although we liked to think that our installation process is very smooth and easy, it is still too complex – complex enough that people who download, don’t get „hooked“ and give up using Plumbr before they get started.

So, we plan to investigate and implement:

• An installer that would somehow detect the Java application servers that exist on the computer, and assist in modifying the start-up arguments.
• An executable that would be able to connect to running Java apps instantly, and re-connect when those are restarted. Possibly also connect to an application running on a remote machine – like jconsole does.
• Include more guidelines and integrate our support channels in the installation process
• Since at the moment Plumbr is mostly tried out by developers, we are thinking of describing the installation for more popular IDEs and application servers separately.

What are the next actions?

We released Plumbr 1.0RC4 on Tuesday, and have dived into the development of the next release by now, implementing the aforementioned directions.

For our blog readers – my blog post pipeline includes topics on customer/user segmentation, channel based conversion analysis, our product licensing plans, and – of course – follow-ups to the above-mentioned topics. Follow our twitter feed to be notified when the posts are published!

And, of course, if you got interested in trying out Plumbr yourself, just tell us your e-mail address and you’ll be flown into our usage funnel instantly!