Digital Evidence and Computer Crime, Second Edition

11.6 File System Traces

Any activity can make an impression on a UNIX file systems, like footprints in snow. Applications can leave remnants on disk either directly in temporary files or indirectly through swap space. For instance, printing creates spool files (usually in /var/spool/lpd) and other applications create temporary files in /tmp and other areas. A TAR file can bring date-time stamps and userids from other systems. Some UNIX systems have a "/proc" file system with information relating to processes running in memory that can be useful for gaining a more complete picture of what was occurring on a system as discussed in Chapter 19.

The simple act of accessing and manipulating files alters their date-time stamps and this information can be correlated with log file entries to gain a better understanding of which user account was involved. For instance, mactime (in TCT and The Sleuth Kit) can use a time range from a wtmp log to generate a chronological list of MAC times for that period as shown here:

# last eco pts/3 66-65-113-65.nyc Sun Oct 20 23:45 - 01:08 (00:23) # mactime -b body -l "Sun Oct 20 23:45 - 01:08 (05:23)" Oct 20 02 23:45:42 452 .a. -rw ------ root root /etc/pam.d/sshd Oct 20 02 23:45:47 124 .a. -rw-r--r-- eco eco /home/eco/.bashrc 191 .a. -rw-r--r-- eco eco /home/eco/. bash_profile Oct 20 02 23:47:30 75428 .a. -r-xr-xr-x root bin /usr/bin/ftp Oct 20 02 23:55:24 22433792 mac -rw-r--r-- eco eco /home/eco/secret.pgp

These MAC times suggests that the FTP client was used to download a file named "secret.pgp," demonstrating that an understanding of how date-time stamps of files are updated under different circumstances can help digital investigators reconstruct the associated events. Process accounting and command history logs may contain information to corroborate this theory.

A summary of common actions and the associated date-time stamp changes on UNIX is provided in Table 11.2. Unlike Windows, this behavior is clearly documented in UNIX manual pages (see man fstat).

When a file is added to or moved out of a directory, the inode change time of the directory listing (".") is updated as well as the last modified and accessed times. One implication of this behavior is that, when a file is deleted on a UNIX system, the ctime of its parent directory is updated. This time can be correlated with the ctime of deleted inodes (and deletion time on ext2/ext3) to get a sense of which file may have been deleted from the directory as shown later in this section.^[13]

Because deleted inodes are not accessible to the file system, deleting a file has the effect of preserving its inode until it is reused. Therefore, when an intruder gains unauthorized access to a UNIX system, installs tools, and deletes files, the inodes of deleted files may be recovered long after the intrusion even if the data are not recoverable. For instance, the following shows ils and mactime from The Sleuth Kit being used to create a chronological list of modification, access, and creation (MAC) times from deleted files on a Solaris system:

examiner1% ils -m -f solaris ufs-image.dd | mactime 4/1/2001 Apr 21 01 16:54:09 0 0 ma. ---------- root root <ufs-bitstream.dd-dead-15265> 0 ma. ---------- root root <ufs-bitstream.dd-dead-15264> Apr 21 01 16:54:17 0 ..c ---------- root root <ufs-bitstrearn:dd-dead-15265> 0 ..c ---------- root root <ufs-bitstream.dd-dead-15264.s Apr 21 01 16:56:41 0 ma. ---------- root root <ufs-bitstream.dd-dead-60580> Apr 21 01 16:56:49 0 ma. ---------- root root <ufs-bitstream.dd-dead-60579> Apr 21 01 16:57:00 0 ..c ---------- root root <ufs-bitstream.dd-dead-60579> 0 ..c ---------- root root <ufs-bitstream.dd-dead-60580> Apr 21 01 16:57:49 0 ma. ---------- root root <ufs-bitstream.dd-dead-3946> Apr 21 01 16:58:10 0 ..c ---------- root root <ufs-bitstream.dd-dead-3946> Apr 21 01 17:00:16 0 ma. ---------- root root <ufs-bitstream.dd-dead-15260> Apr 21 01 17:00:21 0 ma. ---------- root root <ufs-bitstream.dd-dead-15261> Apr 21 01 17:00:30 0 ..c ---------- root root <ufs-bitstream.dd-dead-15261> 0 ..c ---------- root root <ufs-bitstream.dd-dead-15260> Apr 21 01 17:01:43 0 ma. ---------- root root <ufs-bitstream.dd-dead-56788> Apr 21 01 17:01:57 0 ..c ---------- root root <ufe-bitstreamidd-dead-56788> Apr 21 01 17:02:02 0 .a. ---------- root root <ufs-bitstream.dd-dead-56784> Apr 21 01 17:02:09 0 m.. ---------- root root <ufs-bitstream.dd-dead-56784> 0 .a. ---------- root root <ufs-bitstream.dd-dead-56787> Apr 21 01 17:02:10 0 m.. ---------- root root <ufs-bitstream.dd-dead-56787> Apr 21 01 17:02:15 0 ..c ---------- root root <ufs-bitstream.dd-dead-56787> 0 ..c ---------- root root <ufs-bitstream.dd-dead-56784> Apr 21 01 17:07:57 0.a. ---------- root 60001 <ufs-bitstream.dd-dead-53039> Apr 21 01 17:07:59 0 .a. ---------- root root <ufs-bitstream.dd-dead-22817> Apr 21 01 18:48:52 0 mac ---------- root 60001 <ufs-bitstream.dd-dead-15105> 0 mac ---------- root 60001 <ufs-bitstream.dd-dead-11509> 0 mac ---------- root 60001 <ufs-bitstream.dd-dead-7698> 0 .a. ---------- root 60001 <ufs-bitstream.dd-dead-34164> Apr 21 01 18:48:53 0 m.. ---------- root 60001 <ufs-bitstream.dd-dead-53039> 0 m.. ---------- root 60001 <ufs-bitstream.dd-dead-34164> Apr 21 01 18:49:14 0 ..c ---------- root 60001 <ufs-bitstream.dd-dead-S3039> 0 ..c ---------- root 60001 <ufs-bitstrearn.dd-dead-34164> Apr 21 01 18:57:28 0 m.c ---------- root root <ufs-bitstream.dd-dead-22817> Nov 27 02 13:24:58 0 .a. ---------- root bin <ufs-bitstream.dd-dead-64395> Nov 27 02 13:25:00 0 m.. ---------- root bin <ufs-bitstream.dd-dead-64395> Nov 27 02 13:25:06 0 ..c ---------- root bin <ufs-bitstream.dd-dead-64395> Nov 27 02 13:26:11 0 ma. ---------- root root <ufs-bitstream.dd-dead-45494> 0 ma. ---------- root root <ufs-bitstream.dd-dead-45493> Nov 27 02 13:27:01 0 .a. ---------- root root <ufs-bitstream.dd-dead-22816> Nov 27 02 13:27:14 0 ..c ---------- root root <ufs-bitstream.dd-dead-45494> 0 ..c ---------- root root <ufs-bitstream.dd-dead-45493> 0 m.c ---------- root root <ufs-bitstream.dd-dead-22816> Nov 27 02 14:38:59 0 .a. ---------- root bin <ufs-bitstream.dd-dead-64394> Nov 27 02 14:39:13 0 m.. ---------- root bin <ufs-bitstream.dd-dead-64394> Nov 27 02 14:39:43 0 ..c ---------- root bin <Ufs-bitstream.dd-dead-64394> Nov 27 02 15:00:33 0 ma. ---------- root bin <ufs-bitstream.dd-dead-213> Nov 27 02 15:00:43 0 ..c ---------- root bin <ufs-bitstream.dd-dead-213>

The resulting output shows two periods of high activity (April 21, 2001, and November 27, 2002) when a number of files were deleted corresponding with an intruder's activities. The fls utility provides additional information for this time period, showing which directories were modified, accessed, and changed. Combining these data gives digital investigators a sense of where the intruder was operating.

% fls -m / -f solaris ufs-image.dd | mactime 4/1/2001 Sat Apr 21 2001 15:45:28 8192 mac -/drwx-------- 0 0 3 /lost+found Sat Apr 21 2001 15:47:10 512 mac -/drwxr-xr-x 0 0 3776 /usr Sat Apr 21 2001 15:51:57 512 .a. -/drwxrwxr-x 0 3 34006 /opt 9 m.c -/Irwxrwxrwx 0 0 14 /bin ->./usr/bin 512 mac -/drwxrwxr-x 0 3 30225 /mnt 512 mac -/drwxr-xr-x 0 3 37777 /proc 512 .a. -/drwxrwxrwt 3 3 45326 /tmp 512 .a. -/drwxr-xr-x 0 3 64208 /kernel 9 m.c -/Irwxrwxrwx 0 0 20 /lib -> ./usr/lib Sat Apr 21 2001 15:53:25 512 mac -/drwxr-xr-x 0 3 18906 /platform Sat Apr 21 2001 16:32:18 512 mac -/drwxrwxr-x 0 3 19012 /home Sat Apr 21 2001 16:35:59 512 m.c -/drwxrwxr-x 0 3 34006 /opt Sat Apr 21 2001 16:45:56 512 m.c -/drwxrwxr-x 0 3 18898 /devices Sat Apr 21 2001 16:52:58 512 m.c -/drwxr-xr-x 0 3 64208 /kernel Sat Apr 21 2001 16:53:00 512 .a. -/drwxrwxr-x 0 3 41556 /sbin Sat Apr 21 2001 16:53:01 512 m.c -/drwxrwxr-x 0 3 41556 /sbin Sat Apr 21 2001 16:57:54 512 .a. -/drwxr-xr-x 0 3 7552 /var Sat Apr 21 2001 17:04:30 512 .a. -/drwxrwxr-x 0 3 18898 /devices Sat Apr 21 2001 17:07:26 512 mac -/dr-xr-xr-x 0 0 53030 /xfn 512 mac -/dr-xr-xr-x 0 0 30398 /net Sat Apr 21 2001 17:07:35 1032 .a. -/-rw---------- 0 0 87 /.cpr_config Sat Apr 21 2001 17:07:40 512 mac -/drwxr-xr-x 0 0 53037 /vol Sat Apr 21 2001 17:07:47 512 m.c -/drwxr-xr-x 0 3 7552 /var Sat Apr 21 2001 17:07:52 512 .a. -/drwxr-xr-x 0 60001 53038 /cdrom Sat Apr 21 2001 18:48:53 512 m.c -/drwxr-xr-x 0 60001 53038 /cdrom Sat Apr 21 2001 20:22:41 512 m.c -/drwxrwxr-x 0 3 128 /export Sat Apr 21 2001 20:22:42 512 .a. -/drwxrwxr-x 0 3 128 /export Sun Apr 22 2001 22:11:02 804520 m.c -/-rw---------- 0 0 211 /core Sun Apr 22 2001 22:12:32 804520 .a. -/-rw---------- 0 0 211 /core Wed Nov 27 2002 13:26:11 512 m.c -/drwxrwxrwt 3 3 45326 /tmp Wed Nov 27 2002 13:26:21 3072 .a. -/drwxr-xr-x 0 3 49090 /etc Wed Nov 27 2002 13:26:32 1032 m.c -/-rw---------- 0 0 87 /.cpr_config Wed Nov 27 2002 13:26:34 3072 m.c -/drwxr-xr-x 0 3 49090 /etc 3584 m.c -/drwxrwxr-x 0 3 18896 /dev Wed Nov 27 2002 13:26:37 3584 .a. -/drwxrwxr-x 0 3 18896 /dev Wed Nov 27 2002 14:57:03 9 .a. -/lrwxrwxrwx 0 0 14 /bin -> ./usr/bin 9 .a. -/lrwxrwxrwx 0 0 20 /lib -> ./usr/lib

Digital investigators can focus on these periods of high activity, looking for related log files and other data that may help them determine what occurred. When dealing with large amounts of these sorts of data, plotting date—time stamps in a histogram can be useful, showing spikes corresponding to periods of high of activity. For instance, creating a histogram of MAC times using the following command, results in Figure 11.11:

% ils -m -f linux-ext2 honeynet.hda8.dd | mactime -d -i /e1/honeynet/hda8.idx.txt

Figure 11.11: A histogram of deleted inodes from a compromised machine showing a spike on November 8 as a result of an intruder's activities.

Figure 11.11 shows a high number of deleted inodes on November 8, corresponding to the intruder's activities.

^[13]UNIX dates are generally in GMT and may need to be adjusted using the time zone specified in the TZ environment variable.