Cisco Network Security Troubleshooting Handbook
| CBAC problems arise mainly from lack of understanding and misconfiguration. This section discusses the problems you may run into while running CBAC in your network. It shows a step-by-step troubleshooting process for isolating and fixing any problem. The following list outlines the most typical problems in running CBAC:
Selection of Software for IOS Firewall Issues
To turn the router's IOS firewall on, you need a special IOS image that has the firewall feature set, which is a paid feature. Example 5-16 shows the output of the router with the IOS Firewall features. Example 5-16 shows a Cisco 2600 router with 64 MB of RAM (60416 K + 5120 K), 16 MB of flash memory (16384 K), and a code image called flash: c2600-ik9o3s3-mz.122-15.T5.bin. Example 5-16. show version Output from an IOS Firewall Router
If for any reason you have changed the name of the IOS code, and show version is not giving you the right information, a quick way to find out if you have the IOS Firewall features or not is to execute ip inspect ? in the global configuration mode. If you have the IOS Firewall feature set, the output shows the options available; otherwise, you receive a message such as "command not recognized." It is important to note that if the image name contains the letter "o," you have only the CBAC feature on the IOS image. If the image name contains "o3," you have CBAC, IDS and auth-proxy. Before upgrading or changing an IOS image on the router to add the firewall feature, be sure the router meets the RAM and flash requirements. The best way to find the version needed for the IOS Firewall is to use the Software Advisor Tool in the following location (registered users only): http://www.cisco.com/cgi-bin/Support/CompNav/Index.pl The Software Advisor tool can be used to determine the appropriate Cisco IOS software image to download, based on feature set, release, and platform. When you use the Software Advisor, search the feature list for "Context-Based Access Control (CBAC)". If you want to upgrade the software from your existing version keeping the same feature sets, you can use the "Compare the feature" option to find the desired version of IOS. Unable to Connect (Inbound and Outbound) across CBAC
If you are unable to connect to the Internet or from the Internet to the internal network across a CBAC router, you need to first find out if there is any session created (upon seeing the first request packet the inspection engine creates a session). If the session is created, check the state of the session by executing show ip inspect session on the router. If there is no session, the packet is not even reaching the inspection engine. Among many, the following are some of the common causes of this problem:
Packet Failure to Reach the Router's Incoming Interface
If there are routing problems, or problems with Layer 2 between the host and the router, the packet may not reach to the router's incoming interface. Send a ping from the host to the router's incoming interface and see if there is any response. If the host is in the same subnet as the router's incoming interface, and you get the ping response, be sure to set up a default gateway on the host pointing to the router's incoming interface. If there is no ping reply, execute show arp on the router and see if the host's arp entry is there. If it is not there, be sure your host is not assigned a duplicate IP address in the network and troubleshoot the problem further on the switch or hub. You may be running into a bad interface, a bad cable, or misconfiguration on the switch. If the sending host is in a different subnet (multiple hops away), send a ping to the router's incoming interface and see if there is any response. If there is no response, perform a tracert from the host and see where the packets are being dropped. Then troubleshoot the routing issue in that specific device. If you get a reply of the ping packets, be sure to open all the necessary ports for the other traffic there are problems with. Without the ping test, if you want to see quickly if the packets are reaching the router's incoming interface or not, define an ACL with the log keyword to the end of the ACL. This will send the information about the traffic that is allowed or being dropped by the ACL to syslog. If there is no syslog message generated, the packets are not hitting the router at all. Example 5-17 shows the configuration required to see if the Telnet packets from host 10.1.1.10 are hitting the router's interface Ethernet 0 going to 20.1.1.1. Example 5-17. Configuration Required Troubleshooting Connectivity Problem
Misconfigured ACL
The first packet that the client sends to create a connection must be permitted by both the incoming access list on the interface on which it arrives and the outgoing access list on the interface through which it leaves. CBAC opens channels for the packets only after the conversation initiation packet. If either access list denies the conversation initiation, CBAC does not create a conversation, and therefore won't create the temporary access list entries that permit the return traffic. The packets will be dropped. You can find out if the packet initiation is denied through an ACL with debug ip packet detail ACL command. The ACL in this command defines the source and destination addresses and ports of the initiation traffic. The goal is to see if this initiation is being denied because of misconfigured ACL, NAT, or routing problems. Example 5-18 shows how to use debug ip packet detail ACL, to capture the debug output when host 10.1.1.1 is trying to send an initial packet for Telnet to the host 20.1.1.1. Example 5-18. Show How To Use the debug ip packet detail Command
Misconfigured NAT and Routing
After you have verified that the packets are reaching to the router's interface and are not being dropped by the ACL, verify if you have configured a destination NAT. If so, verify if the NAT translation is being built up or not by executing show ip nat translation. If the proper translation is built up, be sure there is a route for the destination you are trying to reach by executing show ip route destination_ip/network. If you do not find a route, define one for the destination network. Note that the initial packet must leave the router before the inspection engine inspects and creates a session with SIS_OPENING. As discussed earlier, debug ip packet detail ACL can be used in conjunction with the show commands mentioned in this section to determine if the failure of the initial packet is caused either by the NAT or by routing. However, it is beyond the scope of this chapter to go through NAT or routing troubleshooting. However, the debug ip nat command can assist in getting details on why NAT is not working. IP Inspection Applied In the Wrong Direction
The direction of inspection rule is a point of confusion for many who first configure CBAC because they think of a firewall as filtering traffic coming from an outside network to an inside network. So, the assumption is that the inspection should be applied to traffic coming from the untrusted side. The problem with this is that CBAC works in terms of conversations, not packets. So, if the conversations are initiated from inside (most of the time, this is the case), inspection must be applied to the outbound direction. Here is a useful rule of thumb: Inspection is applied in the direction of conversation initiation and the extended ACL is applied on the opposite direction. UDP Inspection Is Not Configured
A DNS query works on protocol UDP and port number 53. Assume the DNS server for your corporation is located at the Internet service provider (ISP) network, which is on the unprotected side of the CBAC router. If one of your internal hosts on the protected side of the CBAC router tries to reach a website on the protected or unprotected side using a domain name, it will make a DNS query for name resolution to the IP address, before it can make the actual web connection. If you do not inspect UDP on the CBAC router, this name resolution will fail. This will cause the failure of the actual web request. After going through the preceding troubleshooting steps, if you are still having issues, execute show ip inspect session again. If the session is there and the status that is shown is anything but SIS_OPEN, troubleshoot this further by checking the following:
Return Traffic Might Not Be Coming Back to the Router
With show ip inspect session command output, if you find that the session is being created but staying half-open, other devices along the path may be blocking the return traffic. There also may be issues with the route on the other destination device. If NAT is not working properly on the router, the destination device may not be able to respond back to the wrongly translated IP address of the source. As shown in Example 5-18, debug ip packet detail ACL can be used with the addition of the following line in the access-list 101 to find out if the return traffic is coming back to the router or not. access-list 101 permit tcp host 20.1.1.1 eq 23 host 10.1.1.1 ICMP Traffic Is Not Inspected
Before version 12.2(11) YU & 12.2(15) T, there was no Internet Control Message Protocol (ICMP) inspection engine in CBAC for ICMP traffic. However, some ICMP packets are either necessary or really helpful on almost every real network. For example, for path MTU discovery, you need to allow ICMP packet-too-big, and for debugging connectivity issue across, you must allow echo and echo reply. However, keep in mind that some ICMP packets are dangerous, and allowing all ICMP packets always creates some level of security risk. So, if you are running a version older than 12.2(11) YU or 12.2 (15) T, allow the necessary ICMP packets for the return traffic on the incoming interface (outside) of the unprotected network. Example 5-19 shows commonly used ICMP services on the ACL applied as inbound on the outside interface. Example 5-19. Common ICMP Services Allowed on the Outside Interface of a CBAC Router
If you are running version 12.2(11) YU or 12.2 (15) T and above, you can configure inspection rules for ICMP inspection. There Is a Problem with Inspecting Single Channel Protocol
For single channel protocols such as HTTP or SMTP, generic inspection (inspecting TCP) is sufficient. However, to achieve additional security against protocol violation, you have the option of inspecting with SMTP or HTTP. The big confusion comes when inspecting with HTTP. With HTTP inspection (if Java-list is not configured), by default Java filtering is turned on. So, connection requests for all the websites serving Java applets are denied by the CBAC. Hence, if you run into a problem with browsing certain sites, either remove inspection for HTTP or configure Java-list in conjunction with ACL to allow the trusted sites for Java applets. Before 12.3(7) T, CBAC didn't have any option for inspecting ESMTP traffic. So, if you inspect SMTP, and your mail server runs ESMTP (for example, Microsoft Exchange Server), you will have problems getting e-mail. As work-arounds, either turn off SMTP inspection on the router or turn off ESMTP in the mail server as shown in the following link: http://www.microsoft.com/exchange/en/55/help/default.asp?url=/exchange/en/55/help/documents/server/xog05031.htm If you are running IOS version 12.3(7)T or above, and want to inspect ESMTP, refer to the following link for more details: http://www.cisco.com/en/US/products/sw/iosswrel/ps5207/products_feature_guide09186a00801ed6ee.html Required Multi-Channel Protocol is Not Inspected
If there are connectivity issues with multi-channel protocols such as FTP or H323 for example, make sure you are inspecting those specific protocols. In multi-channel protocols, the data channels are built up based on the negotiation that takes place using a control channel. So, the IOS firewall engine must inspect the control channel to get the necessary information from the negotiation that takes place to create openings on the ACL for the data channel. So, if you do not inspect the application layer protocol, the inspection engine for that protocol is not used. Instead, a generic inspection engine, such as TCP or UDP, is used, which does not go into the payload to get all the required information for the subsequent data channels once the control channel is built up. Hence the data channels fail. For FTP a typical symptom is this: when you execute ls, it hangs infinitely. If inspecting multi-channel protocol does not resolve the issues, then most likely there is a software bug, which can be confirmed by Cisco Support if presented with the debug ip inspect multi_channel_protcol (for example, FTP). IP URL Filtering Blocking The Connection
If the IOS firewall is configured with an external URL filtering server, the URL server's policy has determined whether a connection is allowed or denied. So, if there is a configuration error in defining the policy, the connection across the IOS firewall router will fail. Therefore, you will need to recheck the URL server policy to be sure your URL server is not accidentally denying connections due to misconfiguration. Redundancy or Asymmetric Routing Problems
IOS Firewall does not support router redundancy or load sharing among multiple routers; instead, it supports interface redundancy and load sharing. This means that if the request packet goes through router A, it must return back through router A. Otherwise, if CBAC is configured, router B drops the packets if router B receives them. If the packet leaves from router A serial interface 0 and returns on serial interface 1, if you have the same access-list 101 applied on both of the interfaces, the return packets will make it through and the connection will be established without any problem. This is because when CBAC creates a new channel, it installs the temporary access list entries on the interfaces used for the initial packet, but those same access lists may be installed on backup interfaces that provide additional paths to the same destinations. It is even possible to use CBAC with load sharing, if all the parallel interfaces are configured identically. If you configure the same access list and inspection parameters on two interfaces that are alternate paths to the same destination, connectivity should work. Remember to use the same access lists (with the same access list numbers) on both redundant interfaces. This works fine with dial backup for serial interface. Performance Issues
If traffic or responses from the Command Line Interface (CLI) parser are slow, use the commands shown in Examples 5-20 and 5-21 to check if you are running short of memory and CPU. Example 5-20. Sample Output of CPU Utilization Under Normal Conditions
Example 5-21. Sample Output of Memory Utilization Under Normal Conditions
If CPU utilization goes beyond 50%, investigate the issue further to see where the problem is. Find out if there were any changes to the CBAC configuration. If there were no configuration changes, your network may be under attack and running short of memory and CPU, as the router processes all the unnecessary packets. The best way to find that out is to analyze the syslog and see if there is an alert message from the router indicating that the router is going to aggressive mode and calming down. If you realize the system is under attack, it is best to block the traffic from the upstream router with the ACL and inform your ISP about the situation so that they can block the traffic. Because most of the time the traffic comes with spoofed random IP, the upstream router may be configured with rate limiting so that innocent users' traffic is not completely blocked. It is important to note that CBAC protects your internal network from the attack but at the cost of performance, and if an attack continues for a long time, the rate limiting on the upstream router may be applied as a work-around until the ISP identifies the source of the attack. If you are not under attack and still having performance problems, check the following items to troubleshoot the issue:
Timeouts for TCP, UDP, and DNS
There are a number of timeouts in CBAC configuration. As the TCP is connection-oriented, it has a mechanism for tearing down the connections when the job is finis hed. So, leaving the TCP connection timeout values at the defaults (as shown in Table 5-3) is recommended unless otherwise required for certain applications. However, UDP connections heavily rely on the CBAC timeout values, which are connectionless, and there is no mechanism to let the other party or the CBAC router know when the job is finished. Choosing an appropriate timeout value for UDP connections is very difficult. For UDP, generic UDP inspection simply creates a channel when it sees the first request packet from the client, and keeps that channel open until no traffic has been seen between the client and the server for a preset time. If you set the timeout for too short a time, you may kill certain types of connections prematurely. For example, a Network File System (NFS) may remain mounted across CBAC for a long time with no activity. As NFS is UDP based, the connection may time out prematurely due to a UDP session timeout on CBAC. Therefore, when NFS is called for on any activity, CBAC requires the rebuilding of the session. Performing NFS across CBAC or any firewall is strongly discouraged. And if you set the idle timeout for UDP too high, you may be accumulating a number of UDP sessions for nothing. The problem is even more serious when you have a DNS server on ISP and your client needs to make the entire DNS query to the ISP DNS server across the firewall. Luckily, the IOS Firewall has the option of setting up the timeout for DNS, which is much shorter than the generic UDP timeout. Table 5-3 shows different default timeout values for TCP and UDP. With the commands listed in the table, it is possible to reset these timeout values.
Short Threshold Values for Half-open and New Connections
It is extremely important to establish baselines for your network to find out what is normal before performing this function: manipulating different threshold values for the total number of half-open connections, or total new connection request rate. If you ambitiously set these threshold values too low, the router may go into aggressive mode and start dropping legitimate traffic. Because of this, you will not only experience packet drops, but the performance of the connections will be very poor. To prevent this, you must determine the values of these thresholds for your network. HTTP Inspection Dilemma
If you have inspection turned on for HTTP, Java filtering is turned on by default as discussed before. HTTP inspection causes the packets to be inspected in strict order. So any out-of-order packets received are dropped, causing the packets to be retransmitted. This may cause a delay in downloading web pages. You can verify this by running the following debugs on the router: debug ip inspect detailed debug ip inspect tcp debug ip inpsect http
When performance degrades due to out-of-order packets, you will receive a message as follows with the three debug commands listed previously: Feb 17 17:37:59.428 CST: CBAC* sis 83207F08 L4 inspect result: DROP packet 833BE CC4 (200.1.1.1:80) (20.1.1.1:4383) bytes 272 http You can get around the problem in two ways: either turn off HTTP inspection or upgrade the code to version 12.3(5.13) or 12.3(5.13) T or above. Note Generally per-packet load balancing increases chances of out-of-order packets. Hence, for better performance, install per-flow load balancing instead of per-packet load balancing.
Switching Path
CBAC can work on Process, Fast, or Cisco Express Forwarding (CEF) switching paths. However, CBAC works best with CEF. If there are performance issues with CBAC, first try turning off CBAC and verifying if the performance issue is caused by CBAC. If it is, turn on CBAC with process switching. This may slow down the packets, but it should work. Then turn on CEF or Fast switching and compare the performance. If CEF or Fast switching fails to provide better performance, the most likely cause is IOS software anomalies. For help with IOS software anomalies, consult with Cisco support. Large ACL
ACL comprises multiple access controls elements (ACEs), and they are processed sequentially in top-down fashion. So it is very important to configure the entries that are most likely to be matched toward the top of the access list. The performance impact of an access list increases linearly with the increase in the number of ACEs. So, try summarizing the address ranges of the ACEs as much as possible. The smaller the list is, the lesser the CPU utilization and time to sequentially search for an entry. If you have a huge number of ACEs even after summarization, configure turbo ACL. Note that the turbo ACL feature is available on only 7200, 7500, and 12000 series routers. To turn on turbo ACL, simply execute the access-list compiled command after creating the ACL. The turbo ACL feature compiles the ACL ACEs into a set of special lookup tables while maintaining the first-match requirement. These tables allow for making permit or deny decisions quickly and consistently, which boosts performance dramatically. Keep in mind, however, the following caution: if you have less than five ACEs, a standard or extended ACL may perform better than turbo ACL. This is because, the turbo ACL compilation may take more CPU cycles than the CPU cycles gained with finding a faster match. Additionally, try to use netflow with ACL to boost the performance, because when net flow is enabled, only the first packet in a given flow goes through ACL checking. The rest of the packets bypass ACL checks. Netflow can be turned on in the Cisco router with the ip route-cache flow command under the interface configuration mode. Reverse DNS and IDENT Protocols
If there is a problem in which users can connect to servers but have to wait a long before data comes back, something might be preventing the server from doing reverse DNS or IDENTD. To work around the DNS issue, be sure to allow UDP/53 to the DNS server from the destination server, provided the DNS server is behind the CBAC router. To get around the problem for IDENT protocol, be sure to allow TCP/113 through the CBAC router so that the server can communicate with the client on TCP/113. Running Older Code
If you are running Cisco IOS version less than 12.2(8) T, you should seriously consider upgrading the code base, because a major performance initiative materialized from this specific version. Here are some of the changes that were made to improve the performance of CBAC:
Intermittent Packet Drops
Intermittent packet drops can occur for various reasons. Go through the performance troubleshooting steps to be sure you are not hitting any performance issue. In addition, you may need to check the following to troubleshoot the intermittent packet drops issue:
IP URL Filtering Is Not Working
Troubleshooting IP URL filtering-related issues is fairly simple. With the help of ip urlfilter audit-trail command, to troubleshoot any issue you can turn on an audit trail for IP URL filtering and get all the required messages as shown in Table 5-4.
|