Plumbr 3.5: usability lessons learned

You have all seen numerous release notes. And are familiar with the format: new features, modifications, fixed bugs, et. al. Boring as hell. So instead of bogging you down with technical details we decided to open up our doors by publishing certain aspects of the iteration retrospective.

Having just finished with 3.5 release, this post focuses to changes we applied to user interaction design. It has definitely been an enlightening month in this regard:

Communicate the severity. First and foremost, we changed the way Plumbr communicates incident severity. On numerous cases we have had to face the situation where a user complains “Plumbr finds insignificant leaks”. And indeed, when drilling down to the case, we have had to admit that we have reported something along those lines for the user

A leak we discovered is occupying 12MB out of your 8GB heap

As the application is dying with OutOfMemoryError, the user quickly makes an assumption that our discoveries are wrong. After all, 12MB is just 0.15% of the total available heap so we must be on the wrong track? Well, as a matter of fact in this particular case we did our job too well, finding the problem so fast that by the moment of detection the leak had only managed to grow to 12MB.

Facing this issue lead to a minor change. The very same messaging now focuses upon leak velocity instead of its size:

A leak we discovered is leaking 6MB/minute

The very same problem is now in a different package. Knowing that you have 8GB of heap available, you can backtrack that the death is imminent in less than a day. Obvious when we now look at it. But for whatever reason we managed to live a year and a half under a rock:

Educate the first-time user. Pre – 3.5 releases we had several installation guides available. Separately, living their own lives isolated from the application. So we actually had no data to understand which way of giving installation instructions would work the best. We had a video tutorial, platform-specific guidelines available online on our site and a readme bundled in the distribution. And no clue which one of them actually helped the user during installation.

Well, no more. We integrated the installation guides to the application installation flow and now have got insight into what works and what not. And oh boy were we surprised. The best way to teach tech-savvy users turned out to be a README.txt file bundled into the distribution. Our front-end developer nearly cried when he understood that all the hard work into making the platform-specific user guides easy to understood had been wasted.

The biggest surprise  was related to the fact that we nearly skipped the README from the test as it seemed “obvious” it would not perform. We only integrated it into the test due to the adaptive nature of the Google Analytics where the worse-performing tests are being demoted and shown less frequently. Well, we were wrong. The README version is currently winning by a mile.

Educate some more. Plumbr is a run-time analyst. In the sense that – it analyzes the application behavior during runtime and detects potential bottlenecks on the fly. Obvious. For us. But not for the first-time user. Again, it has taken us more than necessary to finally nail the issue where many of our users complain that “Plumbr is not finding anything” is strongly correlated to their usage. Or lack of usage, to be frank.

As of 3.5 release we now communicate clearly that attaching the agent to the application is not enough and you need to make sure the application being monitored is actually being used. And we do it within the product, not in the form of documentation. Again, obvious when you look at it. But it again took more than a year for us to understand it.

Summary. We do learn a lot from what you tell us. And we can only promise that we embed the wisdom in the future releases. So go ahead and grab a copy of the memory leak detector while it is still fresh.