To blog Previous post | Next post
We got hacked
Two weeks ago we got hacked. Which was something we have anticipated for the long time – after all, you cannot expect to run a reasonable business and not be a target of some malicious attacks. So as we finally reached our “goal” we had to face the consequences implied.
Full disclosure: no customer data was exposed during the attack and no Plumbr executables were harmed during the attack.
I discovered the problem during my AWS backup process. I created a new AMI by creating a clone from an existing machine and forgot to tick the checkbox “Do not reboot”. So I ended up rebooting the original machine. Not too bad I thought while waiting for the machine to revive. But 15 minutes later had to acknowledge that I do not have SSH access to the rebooted machine anymore. Machine was up and running as our services were available, but the SSH connections all got a timeout message.
I was nowhere near a panic mode yet, as I googled and found out that it can relate to magically failed boot procedures, so I did a few more restarts. No luck. Then I migrated the instance to a different availability zone. Unfortunately without any success to show for. But I was still under the impression that mr Jeff Bezos is to blame and started to figure out how can I access the box without a SSH access.
My first idea was about launching the new virtual machine and attach old EBS volume to it so I could check the file system in it. But being a lazy guy I postponed this one for a while. And then my inner genius woke up and reminded me of few things:
- We were running a Jenkins instance in this machine.
- Jenkins has a built-in Groovy console
- This console can be used to launch OS level commands
- The user running Jenkins was a full passwordless sudo user.
So A & B & C & D = gigantic security loophole, but in this case it was beneficial for me as I now had the access to the machine.
And ten minutes later I was frightened out. First stop – somebody had deleted the SSH server configuration. Which I initially thought was the evil me doing things I should not be doing. But then the back hair started to raise as I expect most of you would if you find yourself staring at a C source named backdoor.h.
Now it was time to call the cavalry. Or namely – my fellow co-founder who just happens to have a lot more experience in the field of hacking and getting hacked. 30 minutes after Vladimir stepped in he had figured out that we were running on an OS not patched for six months containing a privilege escalation vulnerability. From there it was clear that the backdoor installed in our backyard was a well-known SSH exploit harvesting SSH passwords and periodically sending the harvested crop to an external IP. Luckily we were not using username/password to access this machine, but had set it up using a Public Key Infrastructure, so no passwords were neither gathered nor sent.
But ok, i needed to restore the SSH configuration to proceed. Via the very same Groovy console I managed to copy the configuration from a different AWS machine to which I luckily had SSH access using a public key. And sudo service sshd restart later I had the access restored.
So – to summarize – we were hit by a privilege escalation vulnerability used to upload C code compiled to a SSH hack. Which is one of the tools in script kid libraries. In this case it seems it was indeed a kid who did not understand how to install a proper rootkit and cover up his tracks.
We considered that there is a chance that all the existing instances could be compromised. So we spent the next couple of days migrating the whole infrastructure to new machines with a bit more focus on security this time.
But the saddest part for my inner genius happened during the aftermath. I was proudly presenting my clever thoughts on how to access the hacked machine via the Jenkins–Groovy hack, when I got a truly surprised stare. And a question – “You can launch any shell script through Jenkins, why bother with this Groovy madness?” And indeed, this time I had outsmarted myself.
In the case you followed us this far you might also be interested in our future posts, so go ahead and subscribe to our Twitter feed. Which I hope will not contain more security breaches anymore – we promise to stick to our bread and butter and write about Java performance optimization topics instead.
Just a small note – those links to Wikipedia’s articles and similar for SEO and better page rankings are really annoying. I think the audience here are not morons.
I think any webapp server should run in a chrooted environment.
IMO, we have a very long road to go, before that be a case in Java ecosystem.
I somehow doubt that you are using a PKI (i.e. certificates signed by a CA) to log into SSH. Did you mean public key based authentication?
That is correct.
Most freak-outs are people’s wild imagination. I wrote a C compiler. God is god, same as in the Bible.
Your ridiculous copy/paste script is still broken.
I’m fairly confused. Without the famous reboot you would have not noticed the hack, further more, the system had been out of date for 6 months.
So I wonder how you can you be so sure that the hacker have done nothing more — during an unknown time span — and that no user data was exposed ?
Could you elaborate more on this, because by looking at this article all we can see is amateurism.
Analyzing log files gives us the confidence that besides the attempt to harvest SSH password nothing else draw the attention of the hacker. And we saw no evidence of those logs being tampered.
Just because you saw no evidence of log tampering doesn’t mean they didn’t do it. Any self-respecting hacker would attempt to cover their tracks in the syslogs. On the other hand, you’re right that they probably would have tried to cover their entry method too.
High security is a performance bottleneck 😉
Did you run any web application penetration test over your website in order to resuct break-ins?
From http://plusbryan.com/my-first-5-minutes-on-a-server-or-essential-security-for-linux-servers check out the part about automatic updates.
Thank you, that was an interesting article.
Thanks for posting openly. Did you have Jenkins configured with any authentication?
Sure. Wide-open Jenkins would be too much 🙂
So, any ideas how the attacker got through the Jenkins auth? You identified other vulnerabilities (Jenkins running with sudo permissions, out-of-date kernel), but not how the attacker got in to the Jenkins Groovy console. Thanks
If the box was that far behind on patches, there is every chance the person got in through another vulnerability in a different package.
U so smart….