We got hacked

Two weeks ago we got hacked. Which was something we have anticipated for the long time – after all, you cannot expect to run a reasonable business and not be a target of some malicious attacks. So as we finally reached our “goal” we had to face the consequences implied.

Full disclosure: no customer data was exposed during the attack and no Plumbr executables were harmed during the attack.

I discovered the problem during my AWS backup process. I created a new AMI by creating a clone from an existing machine and forgot to tick the checkbox “Do not reboot”. So I ended up rebooting the original machine. Not too bad I thought while waiting for the machine to revive. But 15 minutes later had to acknowledge that I do not have SSH access to the rebooted machine anymore. Machine was up and running as our services were available, but the SSH connections all got a timeout message.

I was nowhere near a panic mode yet, as I googled and found out that it can relate to magically failed boot procedures, so I did a few more restarts. No luck. Then I migrated the instance to a different availability zone. Unfortunately without any success to show for. But I was still under the impression that mr Jeff Bezos is to blame and started to figure out how can I access the box without a SSH access.

My first idea was about launching the new virtual machine and attach old EBS volume to it so I could check the file system in it. But being a lazy guy I postponed this one for a while. And then my inner genius woke up and reminded me of few things:

1. We were running a Jenkins instance in this machine.
2. Jenkins has a  built-in Groovy console
3. This console can be used to launch OS level commands
4. The user running Jenkins was a full passwordless sudo user.

So A & B & C & D = gigantic security loophole, but in this case it was beneficial for me as I now had the access to the machine.

And ten minutes later I was frightened out. First stop – somebody had deleted the SSH server configuration. Which I initially thought was the evil me doing things I should not be doing. But then the back hair started to raise as I expect most of you would if you find yourself staring at a C source named backdoor.h.

Now it was time to call the cavalry. Or namely – my fellow co-founder who just happens to have a lot more experience in the field of hacking and getting hacked. 30 minutes after Vladimir stepped in he had figured out that we were running on an OS not patched for six months containing a privilege escalation vulnerability. From there it was clear that the backdoor installed in our backyard was a well-known SSH exploit harvesting SSH passwords and periodically sending the harvested crop to an external IP. Luckily we were not using username/password to access this machine, but had set it up using a Public Key Infrastructure, so no passwords were neither gathered nor sent.

But ok, i needed to restore the SSH configuration to proceed. Via the very same Groovy console I managed to copy the configuration from a different AWS machine to which I luckily had SSH access using a public key. And sudo service sshd restart later I had the access restored.

So – to summarize – we were hit by a privilege escalation vulnerability used to upload C code compiled to a SSH hack. Which is one of the tools in script kid libraries. In this case it seems it was indeed a kid who did not understand how to install a proper rootkit and cover up his tracks.

We considered that there is a chance that all the existing instances could be compromised. So we spent the next couple of days migrating the whole infrastructure to new machines with a bit more focus on security this time.

But the saddest part for my inner genius happened during the aftermath. I was proudly presenting my clever thoughts on how to access the hacked machine via the Jenkins–Groovy hack, when I got a truly surprised stare. And a question – “You can launch any shell script through Jenkins, why bother with this Groovy madness?” And indeed, this time I had outsmarted myself.

In the case you followed us this far you might also be interested in our future posts, so go ahead and subscribe to our Twitter feed. Which I hope will not contain more security breaches anymore – we promise to stick to our bread and butter and write about Java performance optimization topics instead.