Server Provider Attack Warnings: How to Find Brute Force Malware

Time to Get Started

Now, to get into the nuts and bolts of finding how your server is being abused by hackers, type this in:

auditctl -a exit,always -S connect -k watching

What this is doing is adding a rule for auditd. The rule is looking for any outbound connections and then logs them. The nice thing about auditd is that it includes so much information. The -k watching at the end of the above command line is setting a key for us to search by. You can change that to any single word you want.

Then look in /var/log/audit/audit.log

You might see something like this:

type=SYSCALL msg=audit(1615055175.648:9511984): arch=c000003e syscall=42 success=no exit=-101 a0=5f a1=7f0de8002010 a2=1c a3=4 items=0 ppid=1 pid=21732 auid=4294967295 uid=1001 gid=2002 euid=1001 suid=1001 fsuid=1001 egid=2002 sgid=2002 fsgid=2002 tty=(none) ses=4294967295 comm="gutenblock-64" exe=2F7661722F7777772F74657374736974652F7075626C69635F68746D6C2F77702D636F6E74656E742F677574656E626C6F636B2D3634202864656C6574656429 key="watching"

To search through the logs you can type:

ausearch -k watching

That will list all the log entries with the key=”watching” in it.

In the above log entry you’ll see the pid of the process. This helps us “kill” it, but also provides further information. The comm field is the command-line name of the command used to invoke the process. In this example it’s “gutenblock-64”.

The exe field is the goldmine. It’s an ASCII hex encoded string with the exact path of the executable, when it was executed.

Bingo!

The above converts to:

/var/www/testsite.com/public_html/wp-content/gutenblock-64 (deleted)

It shows (deleted) because the filesystem knows it was already deleted.

In this particular case (an actual case, but the string was changed), the hackers were uploading a file, running it, so it was running in memory, then deleting the file.

Tricky huh?

We now have the path, which would tell us what website (testsite.com) is being abused by the hackers. But we still need to investigate further.

Like I stated earlier, the auditd log entry shows us the pid of the offending process. In this case it was 21732.

To verify the pid to comm, type this (change the 21732 to the pid showing in your auditd log):

ps -p 21732 -o comm=

Next, let’s see what user actually ran this command. Type this at the command prompt:

ls -l /proc/21732/fd

You’ll see this type of information:

lr-x------ 1 Mike7 Mike7 64 Mar 7 11:43 0 -> /dev/null
l-wx------ 1 Mike7 Mike7 64 Mar 7 11:43 1 -> /dev/null
lrwx------ 1 Mike7 Mike7 64 Mar 7 11:43 10 -> 'socket:[132580889]'
lrwx------ 1 Mike7 Mike7 64 Mar 7 11:43 100 -> 'socket:[132582695]'
...

And a whole bunch of similar lines.

So we see that the system user Mike7 (fictitious) is the culprit. We can verify this because the auditd logs also provide the fsuid. In this case it was fsuid=1001.

Look in the /etc/passwd to see which user is 1001.

Mike7:x:1001:2002::/home/Mike7:/bin/bash

If you have set-up separate system users for each website, then you know which website is being abused to attack other websites.

fsuid=1001, then look in /etc/passwd to see which user is 1001:

Marg3:x:1001:2002::/home/Marg3:/bin/bash

For further information, let’s see who is being attacked.

Type:

lsof -u Mike7

You’ll see something like this:

xxx.xxx.xxx.xxx.vultr.com:60298->lhcp3047.webapps.net:https (SYN_SENT)
gutenbloc 27626 Mike7 52u IPv4 132535731 0t0 TCP xxx.xxx.xxx.xxx.vultr.com:35752->ip-166-62-108-229.ip.secureserver.net:http (SYN_SENT)
gutenbloc 27626 Mike7 53u IPv4 132542778 0t0 TCP xxx.xxx.xxx.xxx.vultr.com:55124->hosted.by.liquidnetlimited.com:https (ESTABLISHED)
gutenbloc 27626 Mike7 54u IPv4 132538383 0t0 TCP xxx.xxx.xxx.xxx.vultr.com:55196->ekiaioesiq.c09.mtsvc.net:http (SYN_SENT)
gutenbloc 27626 Mike7 55u IPv4 132539972 0t0 TCP xxx.xxx.xxx.xxx.vultr.com:63150->ip-107-180-26-69.ip.secureserver.net:http (SYN_SENT)
gutenbloc 27626 Mike7 56u IPv4 132537091 0t0 TCP

The xxx.xxx.xxx.xxx will be the IP address of your server. The information after the “->” is the site being attacked.