CCNP BCMSN Exam Certification Guide (3rd Edition)

10-2. Watching Data Pass Through a Firewall

Sometimes you might want to know what sort of traffic has passed through a firewall to reach a certain host. At other times, you might need to troubleshoot why traffic is not being forwarded through the firewall. In this case, you would want to verify that packets arrived on one firewall interface but did not go out another interface.

You can use two methods to watch or verify that packets have passed through a firewall:

• Capture session Packets passing through an interface and matching given conditions are captured in a buffer and can be displayed later.

• Debug packet Packets matching conditions defined in a debug command are reported as they pass through the firewall.

NOTE

Beginning with PIX 7.x, the debug packet method is no longer supported.

These methods require different configuration steps, and each affects the firewall resources in different ways. Table 10-11 compares the capture and debug packet methods.

Using Capture

You can define one or more capture sessions on a firewall, each operating independently. Captured packets are stored in a memory buffer and can be viewed much like a protocol analyzer or sniffer trace.

Defining a Capture Session

Two basic steps are involved in defining a capture session:

You can configure the access list by entering one or more access control entry (ACE) statements with the following configuration command:

Firewall(config)# access-list acl_id [line line-num] [extended] permit protocol {source_addr source_mask [operator sport] [destination_addr destination_mask [operator dport]]

Here, only the permit keyword is necessary, because permitted traffic is really only flagged for capture. An implicit deny statement is at the end of the access list, which causes all other traffic to pass without being captured.

The matched protocol can be ip (any IP protocol), tcp (6), udp (17), ah (51), eigrp (88), esp (50), gre (47), igmp (2), igrp (9), ipinip (4), nos (94), ospf (89), pcp (108), snp (109), icmp (1), or a number from 1 to 255.

Source and destination addresses can be explicit IP addresses or subnets, and the masks are regular subnet masks. If you need to specify addresses, be sure the addresses are relevant with respect to NAT. A capture session can monitor inbound traffic on an interface before NAT is performed, and it can monitor outbound traffic after NAT is performed. If you need to match against a source or destination port number, you can add an optional operator: lt (less than), gt (greater than), eq (equal to), neq (not equal to), or range (lies between two port limits). The operator compares the port number to the value given by port (a single decimal number; for a range, give two numbers for lower and upper limits).

TIP

You might find it handy to include something like cap in the access list name, as in acl_cap_testprobe. This way, you can guess that the ACL has a special purpose just by looking at its name.

To define and start a capture session on a firewall interface, you can use the following privileged EXEC command:

Firewall# capture capture_name [access-list acl_name] [ethernet-type type] [interface if-name] [buffer bytes] [circular-buffer] [packet-length bytes]

The capture session is named capture_name (an arbitrary text string). The access list named acl_name is used to identify packets to be captured. If an access list is not used or defined, all IP packets are matched. However, you should use an access list to specifically match trafficespecially on a busy firewall.

You can define the protocol to capture in the access list. You can also specify an Ethernet type code in the capture command instead if you need to capture a protocol that the ACL doesn't support. By default, all Ethernet types are flagged for capture. You can specify one as type using one of these values: arp (ARP requests and replies), ip (TCP/IP), pppoe (PPP over Ethernet), pppoed (PPP over Ethernet Discovery), ip6 (IP version 6), ipx (Novell IPX), or rarp (Reverse ARP).

You should specify which interface the capture session will monitor with the interface keyword and the interface if-name (defined with the nameif command).

You can also make adjustments to the capture buffer. By default, the capture session buffer is 512 KB in the main firewall memory. With the buffer keyword, the buffer can be resized to bytes. By default, the capture session stops when the buffer is full. However, you can use the circular-buffer keyword to allow the capture to work continuously; when the buffer fills, the capture stores the next packet at the beginning of the buffer.

By default, up to 68 bytes of each captured packet are stored in the buffer. You can change this limit with the packet-length keyword to bytes (up to the MTU or maximum packet size). The default value gives enough information to include the IP and upper-layer protocol headers. Be aware that if you increase the packet length, you can view the contents of captured packets, including any cleartext user IDs, passwords, or other confidential information.

Beginning with PIX 7.x, you can add the type keyword to specify a type of data to capture. The following command syntax is used:

Firewall# capture capture_name type {raw-data | isakmp | asp-drop drop-reason} [buffer bytes] [circular-buffer] [packet-length bytes]

The raw-data type is used in capture sessions by default, even when the type keyword is unavailable. In other words, raw IP packets can be captured. You can also capture certain VPN traffic by specifying type isakmp.

Another novel feature involves capturing packets that are dropped rather than forwarded through the firewall. This allows you to analyze the contents of dropped packets so that you can see exactly why the packet was dropped. No other capture filtering by access list or interface is necessary, because packets can be dropped for a wide variety of reasons.

You can use the type asp-drop keywords along with one of the drop-reason keywords listed in Table 10-12.

As a simple example, the following capture session is created to capture all packets that have been dropped by an interface access list:

Firewall# capture ACLdroptest type asp-drop acl-drop

An inbound SMTP session is attempted from an outside host, which is blocked by an access list applied to the outside interface. First, the following Syslog messages were collected for the failed session:

Feb 08 2005 00:25:41 single_vf : %PIX-4-106023: Deny tcp src outside:172.21.4.48/3407 dst inside:172.16.1.5/25 by access-group "acl_outside" Feb 08 2005 00:25:44 single_vf : %PIX-4-106023: Deny tcp src outside:172.21.4.48/3407 dst inside: 172.16.1.5/25 by access-group "acl_outside" Feb 08 2005 00:25:50 single_vf : %PIX-4-106023: Deny tcp src outside:172.21.4.48/3407 dst inside: 172.16.1.5/25 by access-group "acl_outside"

The denied packets can be correlated to the following packets in the capture session:

Firewall# show capture test 3 packets captured 1: 00:25:41.114312 172.21.4.48.3407 > 172.16.1.5.25: S 3520305660:3520305660(0) win 65520 <mss 1260,nop,nop,sackOK> 2: 00:25:44.026197 172.21.4.48.3407 > 172.16.1.5.25: S 3520305660:3520305660(0) win 65520 <mss 1260,nop,nop,sackOK> 3: 00:25:50.122659 172.21.4.48.3407 > 172.16.1.5.25: S 3520305660:3520305660(0) win 65520 <mss 1260,nop,nop,sackOK> 3 packets shown Firewall#

TIP

You can repeat these steps to define several capture sessions. You can assign multiple capture sessions to the same interface. You also can reuse an ACL in multiple capture sessions if needed. Each capture session is independent and captures its own data in a separate capture buffer.

You can use multiple capture sessions to troubleshoot difficult problems in which the firewall is not forwarding traffic for some reason. Configure one capture session on one interface and a similar capture session on another interface. Use the same access list in both capture sessions. Then you can see traffic arriving on one interface but not appearing on the other.

Aside from the normal traffic inspection engines, access lists, and service policies that might be dropping the packets, consider other information contained in the capture buffer. Be sure to look for packet-related parameters such as the don't fragment (DF) bit or the TCP maximum segment size (MSS) that could be causing packets to be silently dropped.

Getting Results from a Capture Session

After you have defined a capture session, you need to monitor it for activity and retrieve the captured data. If you have defined several capture sessions, you might have trouble remembering which one is performing a certain function. You can list the current capture sessions with the following command:

Firewall# show capture

For example, the firewall used in the following show capture output has three capture sessions defined:

Firewall# show capture capture Aserver-out access-list interface outside capture Aserver-in access-list interface inside capture A-trunk interface outside

Notice that one session is bound to the inside interface, and two other separate sessions are active on the outside interface.

You can display the contents of a capture session buffer at any time, even if the capture is still active. To view the buffer contents from a console, Telnet, or SSH session, you can use the following command:

A summary of each packet saved in the capture buffer named capture_name is displayed, even though the capture session is still active. You can also configure another access list named acl_name ahead of time and use that ACL as a display filter. Only packets that are permitted by the display filter access list are displayed.

With PIX 7.x and FWSM 2.x platforms, you can use the decode keyword to display captured packets in an abbreviated form. This is the default display format. You can also display a subset of captured packets. You can use the packet-number keyword to specify packet number packet as the first to display. You can also use the count keyword to set the number of packets to display as count.

For example, you could use the following command to display packets 100 through 157 in the capture buffer:

Firewall# show capture test packet-number 100 count 57

The top portion of Figure 10-5 shows an example of a TCP packet displayed from the capture session named "test." Only the basic IP and TCP information is shown. This format is useful if you are looking at a list of packets as a record of traffic flow.

To see more detail about the captured packets, you can add the detail keyword to the show capture command. The same sample packet is shown in the middle of Figure 10-5. Here, the source and destination MAC addresses are shown along with the IP addresses. Many IP and TCP fields contained in the packet are also shown.

Until this point, only the packet header information has been displayed. You can also see the contents of the packet payload (up to packet-length bytes, as given in the capture command) by using the dump keyword. The bottom portion of Figure 10-5 shows the sample packet in the dumpformat.

The captured IP packet contents are shown as a hexadecimal decode, along with the ASCII equivalent of each byte.

Using a Capture Session to Display Trunk Contents

You can also use a capture session to capture and decode packets traveling over a trunking interface. In the following example, interface ethernet0 is configured as an 802.1Q trunk, passing VLANs 41 (interface dmz3) and 998 (interface outside). A capture session named "trunk-test" is enabled on the outside interface, with no access list as a filter:

When the capture buffer is displayed, notice how each packet is shown along with its 802.1Q VLAN number tag:

Firewall# show capture trunk-test 4434 packets captured 00:52:55.034116 802.1Q vlan#998 P0 arp reply 172.16.89.191 is-at 0:d:28:a7:83:80 (0:d:28:a7:83:80) 00:52:55.034589 802.1Q vlan#998 P0 arp reply 172.16.89.191 is-at 0:c:30:10:26:0 (0:c:30:10:26:0) 00:52:55.860902 802.1Q vlan#998 P0 arp who-has 172.16.89.253 tell 128.163.89.11 00:52:55.860978 802.1Q vlan#998 P0 arp who-has 172.16.89.254 tell 128.163.89.11 00:53:01.841844 802.1Q vlan#998 P0 172.21.4.6.3862 > 172.16.89.161.23: S 2411823264:2411823264(0) win 65520 <mss 1260,nop,nop,sackOK> [output deleted]

Some firewall documentation explains that you should configure the capture session to collect only VLAN or 802.1Q EtherTypes by adding the vlan or 802.1q keywords, respectively. As of PIX 6.3(3), this isn't necessary. The firewall interprets the 802.1Q encapsulation correctly with a normal capture session definition.

Copying Capture Buffer Contents

Sometimes you might find that viewing the contents of a capture buffer from a command-line interface (CLI) becomes too cumbersome or confusing. This can happen when the capture buffer becomes very largetoo large to navigate with CLI commands or display filters.

At other times, the capture buffer might contain useful information that deserves further review. For example, you might have a PC-based tool that can import captured data for viewing and analysis. You also might want to archive the capture buffer for future use. You can extract a capture buffer from a firewall in several ways, as discussed in the following sections.

Copying to an External TFTP Server

You can copy a capture session to a TFTP server with the following command:

Firewall# copy capture:capture-name tftp://server/path [pcap]

The entire buffer from the capture session named capture-name is copied to the TFTP server at IP address server into the file and directory defined by path, which is relative to the TFTP server's root directory.

In the following, the capture session named bigtest is copied to the TFTP server at 192.168.254.10 as file bigtest in the TFTP root directory:

Firewall# copy capture:bigtest tftp://192.168.254.10/bigtest

The resulting capture file contains the same text that is seen with the show capture command. You can also save the capture buffer in the PCAP format, which can be imported into many network analysis tools. To do this, add the pcap keyword.

TIP

The PCAP capture file format is used by the tcpdump analysis utility. It can be imported directly into the Ethereal network analysis application. It can also be converted and imported into other commercial network analysis tools. You can go to http://www.tcpdump.org for more information about tcpdump and PCAP. See http://www.ethereal.com for more about the Ethereal application.

Copying the Capture Buffer to a Web Browser

From a web browser, you can display a capture buffer as if you had used the show capture command. You also can download the capture buffer in PCAP format and save it as a fileall without leaving your web browser and without needing a TFTP server running on your PC. Follow these steps to accomplish this:

Figure 10-6 shows the capture session named Aserver-in from the firewall at 192.168.254.1 displayed in a browser window.

You can also use the web browser to download the capture buffer as a file in PCAP format. To do this, add the /pcap keyword to the end of the URL. This time, the browser automatically fetches the capture file rather than displaying the capture text. Figure 10-7 shows an example of saving the capture session named icmp from the firewall at 192.168.254.1 to the local machine. Notice that the browser saves the capture data to a file called pcap by default. You can change the filename and location through the web browser's file download dialog box.

As soon as you have the capture file downloaded in PCAP format, you can use a network analysis tool to examine and interpret the contents. For example, Ethereal is a free network protocol analyzer (http://www.ethereal.com) that can import PCAP files directly. Figure 10-8 shows how Ethereal has been used to open the sample Aserver-in capture file.

Likewise, you can use other commercial protocol analyzers as long as they can convert or import the PCAP file format. Figures 10-9 and 10-10 show how the EtherPeek NX protocol analyzer (http://www.wildpackets.com) can be used to convert and import the capture file. Notice that the ProConvert tool is used first to convert the PCAP (tcpdump) file into EtherPeek format.

Controlling a Capture Session

After a capture session is defined and activated, you might need to stop it as soon as some interesting data is captured. You might also want to clear the buffer so that new data can be captured in an empty buffer. When you are finished with a capture session, you need to delete it.

You can use the following commands to control an existing capture session:

A Capture Example

A firewall separates a web server on the inside from a user on the outside. The web server has worked correctly in the past, but users are calling to complain that they can't get the server to respond with valid browser content. Naturally, the user believes the firewall is to blame!

You find that you can ping both the user and the web server from the firewall. As well, the outside user can ping the web server through the firewall. After inspecting the firewall configuration, you also find that the address translation and access list definitions look correct. You also spend some time searching the Syslog archives for any messages that might indicate that the firewall is somehow blocking the HTTP connections. Unfortunately, you don't see anything interesting.

The capture feature can provide some low-level data about this problem. Because you can configure multiple capture sessions, it is wise to capture this traffic flow on both the outside and inside interfaces. If you can see what packets have arrived on both sides of the firewall, you're more likely to see the traffic from the firewall's perspective.

First, you configure two access listsone for each firewall interface, configured to permit traffic between the user's PC and the web server. These access lists will trigger the capture for packets that match the permit statements.

Figure 10-11 shows a simple network diagram of this scenario. The web server is at 172.19.32.9 (the local address) on the inside, and it has a static translation to 10.4.4.10 (the global address) on the outside. The user PC is at 10.4.4.33 on the outside.

You define the following access lists:

Firewall(config)# access-list cap_outside permit ip any host 10.4.4.10 Firewall(config)# access-list cap_outside permit ip host 10.4.4.10 any Firewall(config)# access-list cap_inside permit ip any host 172.19.32.9 Firewall(config)# access-list cap_inside permit ip host 172.19.32.9 any

Next, you define and enable the two capture sessions:

Firewall# capture web_inside access-list cap_inside interface inside Firewall# capture web_outside access-list cap_outside interface outside

Now you instruct the user to try opening a web browser to the web server's URL. As soon as that happens, you can display the contents of both capture buffers as follows:

Firewall# show capture web_outside 2 packets captured 19:24:27.241885 10.4.4.33.1193 > 10.4.4.10.80: S 3375443541:3375443541(0) win 4096 <mss 1460> 19:24:27.242403 10.4.4.10.80 > 10.4.4.33.1193: R 917139784:917139784(0) ack 3375443542 win 0

Here, only two packets are seen at the firewall's outside interface:

• A packet from the user's PC to the web server (TCP port 80) with the TCP SYN flag set (S).

• A reply from the web server address to the user's PC.

Why are there only two packets when there should be at least a three-way TCP handshake? A clue is present, because the return packet has the TCP RST flag set (R). Something has caused the connection to be reset before it has been established.

A look at the capture on the inside interface might provide some evidence about who is resetting the HTTP connections:

Firewall# show capture web_inside 2 packets captured 19:23:56.171469 10.4.4.33.1192 > 172.19.32.9.80: S 2178639828:2178639828(0) win 4096 <mss 1380> 19:23:56.171759 172.19.32.9.80 > 10.4.4.33.1192: R 0:0(0) ack 2178639829 win 0

Again, only two packets are seen at the inside interface. The reply packet that has reset the connection did indeed come from the web server's IP address on the inside. In fact, the RST flag (R) was already set when the packet arrived at the firewall's inside interface. Therefore, you can conclude that something is misconfigured on the web server that causes it to deny or reset every HTTP connection. The problem is not within the firewall.

Using Debug Packet

In FWSM 2.x and PIX 6.x, you can enable a single debug packet session so that the firewall reports how it has handled packets from specific traffic flows. This feature is not available beginning with PIX 7.x.

Only the debug messages are displayed to the trace channel, or the first active Telnet session open to the firewall. No packets are captured or stored during the debug packet session. However, the packet header and upper-layer protocol headers are shown in the debug messages. As well, up to 50 bytes of the packet payload contents are displayed in hex and ASCII.

TIP

A debug session can impact the performance of a busy firewall. The firewall must match packets as it inspects them and generate the appropriate debug information. Obviously, adding this process to the Adaptive Security Algorithm can also add significant delays to the firewall's throughput. Because debug output is also sent to a management session interactively, as matching packets are forwarded, a terminal session can become so congested with output that it becomes unusable.

As soon as the debug output begins, be ready to stop it by entering the no debug packet interface-name command.

If your terminal session is unresponsive and you can't enter the command, start a new terminal session (preferably through Telnet or SSH) and enter the command from there.

Be aware that the no debug all and undebug all commands have no effect on the debug packet session. Debug packet is a special type of debug session and must be disabled explicitly.

You can define and activate a packet debug session with the following command:

Firewall# debug packet if_name [src source_ip [netmask mask]] [dst dest_ip [netmask mask]] [[proto icmp] | [proto {tcp | udp} [sport src_port] [dport dest_port]] [rx | tx | both]

The debug process displays only packets matching a set of parameters. The packets also must pass through the firewall interface named if_name (outside, for example) in the receive direction (rx, or inbound), the transmit direction (tx, or outbound), or either direction (both, inbound or outbound).

You can (and should) be as specific as possible in identifying the debugged traffic. You can use the src and dst keywords to specify the source address and destination address, respectively. If you also provide a netmask, it follows the normal subnet mask format (a 1 bit matches, and a 0 bit is a wildcard). You can match a protocol as ICMP (proto icmp), UDP (proto udp), or TCP (proto tcp), and the source and destination port numbers (sport and dport) if needed.

You can select the traffic direction, relative to the interface if_name, as receive only (rx), transmit only (tx), or in both directions (both).

As a simple example, the following debug packet command is used to verify how a firewall handles an ICMP echo request from a host on the outside (172.21.4.6) to a host on the inside (global address 10.63.89.161, local address 192.168.199.100):

Firewall# debug packet outside dst 10.63.89.161 both --------- PACKET --------- -- IP -- 172.21.4.6==>10.63.89.161 ver = 0x4 hlen = 0x5 tos = 0x0 tlen = 0x3c id = 0xf3a8 flags = 0x0 frag off=0x0 ttl = 0x7e proto=0x1 chksum = 0xbeb8 -- ICMP -- type = 0x8 code = 0x0 checksum=0x485c identifier = 0x400 seq = 0x100 -- DATA -- 00000010: 61 62 63 64 | abcd 00000020: 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 | efghijklmnopqrst 00000030: 75 76 77 61 62 63 64 65 66 67 68 69 a1 | uvwabcdefghi. --------- END OF PACKET --------- --------- PACKET --------- -- IP -- 172.21.4.6==>192.168.199.100 ver = 0x4 hlen = 0x5 tos = 0x0 tlen = 0x3c id = 0xf3a8 flags = 0x0 frag off=0x0 ttl = 0x7e proto=0x1 chksum = 0x10f0 -- ICMP -- type = 0x8 code = 0x0 checksum=0x485c identifier = 0x400 seq = 0x100 -- DATA -- 00000010: 61 62 63 64 | abcd 00000020: 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 | efghijklmnopqrst 00000030: 75 76 77 61 62 63 64 65 66 67 68 69 a1 | uvwabcdefghi. --------- END OF PACKET --------- --------- PACKET --------- -- IP -- 192.168.199.100==>172.21.4.6 ver = 0x4 hlen = 0x5 tos = 0x0 tlen = 0x3c id = 0xf3a8 flags = 0x0 frag off=0x0 ttl = 0xff proto=0x1 chksum = 0x8fef -- ICMP -- type = 0x0 code = 0x0 checksum=0x505c identifier = 0x400 seq = 0x100 -- DATA -- 00000010: 61 62 63 64 | abcd 00000020: 65 66 67 68 69 6a 6b 6c 6d 6e 6f 70 71 72 73 74 | efghijklmnopqrst 00000030: 75 76 77 61 62 63 64 65 66 67 68 69 a1 | uvwabcdefghi. --------- END OF PACKET --------- Firewall# no debug packet outside

Notice that the first inbound packet is destined for the global address of the internal host. The firewall displays the second packet when it performs the address translation, with a new destination of the local address. The third packet is the echo reply from the internal host.