Intrusion Detection Systems

Intrusion Detection Systems (IDS) play a critical role in the protection of the IT infrastructure Intrusion detection involves monitoring network traffic, detecting attempts to gain unauthorized access to a system or resource, and notifying the appropriate individuals so that counteractions can be taken. This section starts by discussing how IDS systems work; then IDS tools and products are discussed; and finally IDS evasion techniques are discussed.

IDS Types and Components

Intrusion detection was really born in the 1980s when James Anderson put forth the concept in a paper titled "Computer Security Threat Monitoring and Surveillance." IDS systems can be divided into two broad categories: network-based intrusion-detection systems (NIDS) and host-based intrusion-detection systems (HIDS). Both can be configured to scan for attacks, track a hacker's movements, or alert an administrator to ongoing attacks. Most intrusion detection systems consist of more than one application or hardware device. IDS systems are composed of the following parts:

• Network sensors Detects and sends data to the system.
• Central monitoring system Processes and analyzes data sent from sensors.
• Report analysis Offers information about how to counteract a specific event.
• Database and storage components Performs trend analysis and stores the IP address and information about the attacker.
• Response box Inputs information from the previously listed components and forms an appropriate response.

The key to what type of activity the IDS will detect depends on where the network sensors are placed. This requires some consideration because after all, a sensor in the demilitarized zone (DMZ) will work well at detecting misuse there but will prove useless for attackers who are inside the network. Even when you have determined where to place sensors, they still require specific tuning. Without specific tuning, the sensor will generate alerts for all traffic that matches a given criteria, regardless of whether the traffic is indeed something that should generate an alert. An IDS must be trained to look for suspicious activity. Figure 10.1 details the relationship between IDS systems and the types of responses they can produce.

Figure 10.1. IDS true/false matrix.

Pattern Matching and Anomaly Detection

Pattern matching, protocol decoding, and anomaly detection are some of the basic characteristics and analysis methods used by IDS systems. Each type takes slightly different approaches to detecting intrusions. A graph showing the relationship of these types and the vendors that use each method is shown in Figure 10.2.

Figure 10.2. IDS types.

Anomaly detection systems require the administrator to make use of profiles of authorized activities or place the IDS into a learning mode so that it can learn what constitutes normal activity. A considerable amount of time needs to be dedicated to make sure that the IDS produces few false negatives. If an attacker can slowly change his activity over time, the IDS might actually be fooled into thinking that the new behavior is actually acceptable. Anomaly detection is good at spotting behavior that greatly differs from normal activity. As an example, if a group of users who only log in during the day suddenly start trying to log in at 3 a.m., the IDS can trigger an alert that something is wrong.

Exam Alert

A false negative is the worst type of event, as it means that an attack occurred but that the IDS failed to detect it.

Somewhere in the middle of the spectrum of intrusion detection is protocol decoding. Protocol decoding IDS systems have the capability to reassemble packets and look at higher layer activity. If the IDS knows the normal activity of the protocol, it can pick out abnormal activity. Protocol decoding intrusion detection requires the IDS to maintain state information. As an example, let's look at the domain name server (DNS) service. DNS is a two-step process. Therefore, a protocol matching IDS can detect that when a number of DNS responses occur without a DNS request, a cache poisoning attack might be happening. To effectively detect these intrusions, an IDS must reimplement a wide variety of application-layer protocols to detect suspicious or invalid behavior.

On the opposite end of the scale, there is pattern matching. Snort is a good example of a pattern matching IDS. Pattern matching IDS systems rely on a database of known attacks. These known attacks are loaded into the system as signatures. As soon as the signatures are loaded into the IDS, the IDS can begin to guard the network. The signatures are usually given a number or name so that the administrator can easily identify an attack when it sets off an alert. Alerts can be triggered for fragmented IP packets, streams of SYN packets (DoS), or malformed Internet Control Message Protocol (ICMP) packets. The alert might be configured to change to the firewall configuration, set off an alarm, or even page the administrator. The biggest disadvantage to the pattern matching system is that the IDS can only trigger on signatures that have been loaded. A new or obfusticated attack might go undetected. Obfusticated attacks are those that are disguised.

Snort

Snort is a freeware IDS developed by Martin Roesch and Brian Caswell. It's considered a lightweight, network-based IDS that can be set up on a Linux or Windows host. Although the core program has a command-line interface, two popular GUIs can be used. They include SnortSnarf and IDS Center. Snort operates as a network sniffer and logs activity that matches predefined signatures. Signatures can be designed for a wide range of traffic, including Internet Protocol (IP), Transmission Control Protocol (TCP), User Datagram Protocol (UDP), and Internet Control Message Protocol (ICMP).

Snort rules are made up of two basic parts:

• Rule header This is where the rules actions are identified.
• Rule options This is where the rules alert messages are identified.

Here is a sample rule:

Alert tcp any any -> any 80 (content: "hacker"; msg: "Hacker Site Accessed";)

The text up to the first parentheses is the Rule Header. The first part is known as the rule action. Alert is the action used in the preceding sample rule; rule actions can include

• Alert
• Log
• Pass
• Activate
• Dynamic

The next item is the protocol. In the example, TCP was used. After the protocol is the source address and mask. Although the example uses any any, it could have been a specific network such as 10.10.0.0/16. This is followed by the target IP address and mask. The final entry of the rule header designates the port. This example specifies 80.

The section enclosed inside the parentheses are the rule options. Rule options are not required but are usually the reason for creating the rule. The rule options are as follows (content: "hacker"; msg: "Hacker Site Accessed";). The first portion specifies the action, which is to examine port 80 traffic for the word "hacker". If a match occurs, a message should be generated that reads, "Hacker Site Accessed," and the IDS would create a record that a hacker site might have been accessed. The rule option is where Snort has a lot of flexibility. Table 10.1 lists some common keywords Snort can use.

Although the CEH exam will not expect you to be a Snort expert, it is a good idea to have a basic understanding of how it works and to understand basic rules. A few of these are shown in Table 10.2.

Although these are good examples of basic Snort rules, they can be much more complex. Following is an example of one developed to alert upon detection of the Microsoft Blaster worm:

[View full width]

alert tcp $EXTERNAL_NET any -> $HOME_NET 135 (msg:"NETBIOS DCERPCISystemActivator bind attempt"; low:to_server,established; content:"|05|";distance:0; within:1; content:"|0b|"; distance:1; within:1;byte_test:1,&,1,0,relative; content:"|A0 01 00 00 00 00 00 00 C0 00 00 00 00 00 00 46|"; distance:29; within:16; reference:cve,CAN-2003-0352;classtype :attempted-admin; sid:2192; rev:1;)

Building Snort rules is only half the work. After a Snort alert occurs, it is important to be able to analyze the signature output. To really be able to determine what attackers are doing and how they are doing it, it is important to be able to perform signature analysis. The goal of the signature is to be able to identify malicious activity and be able to track down the offender. This activity can be categorized as

• Scans and enumeration
• Denial of service (DoS) attacks
• Exploits

If you have never used an IDS, you might be surprised at the number of alerts it produces in just a few hours after you connect to the Internet. Shown in Figure 10.3 is the signature of an Nmap ACK scan.

Figure 10.3. Nmap ACK scan log.

As you can see, the attacker is located at 192.168.13.236 and is scanning 192.168.13.235. On the third line of each scan, you should see the ***A***, which indicates the ACK scan. The other telltale sign is repeating sequence and acknowledgement numbers. That is not the normal behavior for TCP.

If this section has raised your interest in getting to know more about Snort, there are a host of books that can help you through the process. One good book by Syngress is Snort 2.1 Intrusion Detection, by Jay Beale. Now let's look at some of the ways that hackers attempt to bypass IDS and prevent it from detecting their activities.

IDS Evasion

Attackers can use a range of techniques to attempt to prevent detection. These include techniques such as

• Flooding
• Evasion
• Session splicing

One of the most basic is to attempt to overload the IDS by flooding. Flooding is nothing more than attempting to overload the IDS by flooding it with traffic. The attacker might even insert a number of low priority IDS triggers to attempt to keep it busy while a few more damaging attacks slip by. Generating such a huge amount of traffic forces the administrator to sort through all the data and somehow try to make sense of it all. The real target and intent of the attacker might be totally lost within the blizzard of messages, beeps, and alerts generated.

Evasion is another effective technique. It occurs when an IDS discards the packet that is accepted by the host it is addressed to. As an example, TCP segments marked with a SYN flag might also include data. Because this is an infrequent occurrence, an IDS might ignore the contents of these packets, thereby allowing the packets to go undetected.

Session splicing works by delivering the payload over multiple packets, which defeats simple pattern matching without session reconstruction. This payload can be delivered in many different manners and even spread out over a long period of time. It is really a form of fragmentation. By breaking up the payload over many different packets, many IDS systems will fail to detect its true purpose. IP fragments typically arrive in the order sent, but they don't have to. By sending the packets out of order and playing with fragment IDs, reassembly can become much more complicated. If the IDS cannot keep all fragments in memory for reassembling, an attacker could slip by.

IDS Evasion Tools

Several tools are available that can be used to evade IDS systems. Most of these tools exploit one or more of the techniques discussed in the previous section. Some of the better known tools are discussed in the following:

• Stick Uses the straightforward technique of firing numerous attacks to purposely trigger IDS events. Although the IDS system attempts to keep up with the new flood of events, it could eventually become flooded and a DoS of the IDS might result.
• ADMutate Borrows ideas from virus writers to create a polymorphic buffer-overflow engine. An attacker feeds ADMutate a buffer-overflow exploit to generate hundreds or thousands of functionally equivalent exploits, but each has a slightly different signature.
• Mendax Builds an arbitrary exploit from an input text file and develops a number of evasion techniques from the input. The restructured exploit is then sent to the target.
• NIDSbench Includes fragrouter, tcpreplay, and idstest. Fragrouter fragments traffic, which might prevent the IDS from detecting its true content.
• Nessus Can also be used to test IDS systems and has the capability to perform session splicing attacks.

IDS systems are not perfect and cannot be expected to catch all attacks. Even when sensors are in the right location to detect attacks, a variety of tools and techniques are available to avoid detection. For IDS systems to be effective, the individuals responsible for them must continually monitor and investigate network activity to stay on top of changes in hacking tools and techniques.