CCNP BCMSN Exam Certification Guide (3rd Edition)

2017-07-07 02:10:07

10-1. Checking Firewall Vital Signs

After a firewall is put into production, it is important to know how to check its health. You can proactively monitor various resources and statistics in an effort to determine when the firewall has a problem or becomes underpowered. As well, when users complain of problems or slow response times, you should be able to look into some of the firewall's inner workings to quickly spot issues.

How a firewall will behave under the load of a production network with real users and real appli-cations is difficult to predict. When a problem is reported, you can rely on many of the available firewall statistics to see what is wrong. However, those numbers don't mean much if you don't have anything to compare them against.

You should make every effort to determine a baseline or average for several firewall parameters. Do this while the firewall is operating normally and all users are satisfied with its performance. Monitor the following statistics periodically so that you get an idea about their values and how they change during a business day or week:

CPU utilization

Memory usage

Xlate table size

Conn table size

Firewall throughput

Failover activity and synchronization rates (if applicable)

The following sections show you how to monitor these functions.

Using the Syslog Information

Firewalls normally stay busy silently inspecting traffic, denying some packets and forwarding others. The only way to "feel the pulse" of your firewall, to see what it's been doing, is to look back through the Syslog messages it has generated.

You can view the most recent Syslog information, whether or not you have a Syslog server set up. You can enable buffered logging with the logging buffered level configuration command and view the buffer with the show logging command. However, only the most recent messages are kept in the buffer. On a busy firewall, the buffer can be completely overwritten in only a few seconds, so you might not be able to find the information you need.

Instead, you should seriously consider setting up machines to collect Syslog messages from every firewall in your network. Syslog information gives a very useful historic record in the following ways:

Troubleshooting information Events in the life of the firewall itself (reloads, failovers, memory shortages, and so on) are recorded, along with the reasons some packets were denied or dropped.

Performance and activity Address translations and connections are recorded as they are created and torn down. The duration (both time and data volume) of each connection is also recorded so that an analysis of data throughput can be computed over time.

Security audit trail The results of user authentication can be recorded for a historical record or for usage information. Address translation and connection information can be recorded and used as evidence when someone from the outside attempts malicious activity.
The same can be recorded for internal users attempting similar activity toward the outside world. In the case of port address translation (PAT), the Syslog record is the only evidence that can be used to backtrack from the outside to find which internal user was using the PAT global address at a certain date and time.

TIP

When you set up a Syslog server, make sure you have a system (both hardware and the Syslog software) that can handle the appropriate Syslog message rate sent by the firewall(s). This rate varies according to the firewall's connection load. More importantly, it varies according to the Syslog severity level configured on the firewall.

Refer to Chapter 9, "Firewall Logging," for detailed information about Syslog planning and analysis.

After you configure a firewall to send Syslog information to a server or memory buffer, you should verify its operation. You can use the show logging command for this, paying attention to the first dozen lines of output. Look at each type of logging to see if the appropriate ones are enabled and if they are set to the correct severity level. In particular, you should see a count of the total number of messages sent to each enabled destination. In the following example, both buffer and trap (Syslog server) logging are enabled and active:

Notice that the total numbers of buffer and trap logging messages are different. This is because each logging method is configured independently. For example, trap logging might have been enabled at an earlier date than buffer logging. Otherwise, each logging method can be configured with a different severity level threshold, causing each one to generate a different type and volume of message.

Checking System Resources

A firewall inspects traffic and performs its functions by using a combination of system resources. From a hardware standpoint, these resources are very straightforward and include the CPU and system memory. The following sections analyze these resources.

Firewall CPU Load

You get a general idea about the processing load on a Cisco firewall by using the show cpu usage command. For example, the following firewall appliance has a 5-second average of 27 percent. The command output also shows that the CPU is under a consistent load of about 24 percent:

Firewall# show cpu usage CPU utilization for 5 seconds = 27%; 1 minute: 25%; 5 minutes: 24% Firewall#

As a rule of thumb, the CPU utilization should stay below an average of 80 percent. The utilization might spike or temporarily peak at a greater value, as seen in the short-term 5-second utilization. This is normal behavior, because the CPU could be processing a periodic task or a short burst of traffic.

In extreme situations, you might see Syslog message ID 211003 (PIX-3-211003, for example) being generated. This message is sent when the firewall CPU has consistently been at 100%.

If the CPU stays above 80 percent during a time you consider to be a normal traffic load, without a significant attack occurring, you should consider upgrading the firewall or lightening its traffic load. Some of the possibilities for doing so are as follows:

Replace the firewall with a higher-performance model.

Implement firewall load balancing so that the traffic load is balanced across two or more firewall platforms. Refer to Chapter 8, "Firewall Load Balancing," for detailed coverage.

Make configuration changes to reduce the firewall's processing load; remove any unnecessary activities it is performing.

TIP

If your firewall is configured to run multiple security contexts, remember that these "virtual firewalls" are all being emulated by one firewall platform. If the CPU usage is running high, you might have too many different contexts configured or too many contexts that are heavily used. You can get an idea of the breakdown of CPU resources across the contexts with the show cpu usage context all command, as in the following example:

Firewall# show cpu usage CPU utilization for 5 seconds = 86%; 1 minute: 84%; 5 minutes: 83% Firewall# show cpu usage context all 5 sec 1 min 5 min Context Name 2.2% 2.5% 2.4% system 1.0% 1.1% 1.0% admin 54.6% 52.2% 53.1% CustomerA 27.3% 27.2% 25.6% CustomerB 0.9% 1.0% 0.9% Test Firewall#

The utilization values from the first command are totals for the sum of the context values from the second command. In this example, the CustomerA context is using the most CPU resources and is a good candidate for being moved onto its own firewall platform.

To find out more about which activities are taxing the CPU, you can try to track down the most-used processes. The firewall's CPU continuously performs a number of different tasks, such as processing inbound packets; inspecting ICMP, UDP, and TCP traffic; interacting with the console and user sessions; maintaining failover communication; operating routing protocols; and so on.

In addition, a large number of tasks involve maintaining timers. When xlate and conn entries are created, timers must be started; when the various timers expire, those entries must be deleted. You also can time and control authenticated user sessions. Each of these timer functions has its own process, requiring periodic attention from the CPU. In fact, a process runs continuously just to keep the CPU utilization values computed and updated!

To see the entire list of active firewall processes, you can use the show processes command. Unless you are a Cisco Technical Assistance Center (TAC) engineer, the only interesting information is found in the following columns:

Process Displays a descriptive name for the process, such as update_cpu_usage, that computes the values for the show cpu usage command.

Runtime Lists the actual amount of time the CPU has spent on each process, in milliseconds. The runtime values begin at 0 when the firewall is first booted and accumulate until it is powered down or reloaded.

NOTE

Each line of the show processes command begins with three flag characters that give more information about the process. The first character denotes the process priority; it can be one of the following: D (dead), L (low), M (medium), or C (critical).

The second character denotes the process state: r (ready to run), s (sleeping), x (dead), w (idle), or * (running).

For example, the following processes are running at medium and critical priority, respectively:

Mrd 003cb154 013ab118 00d9d580 9130720 013a9190 7660/8192 557poll Csi 0071b139 01d48580 00d9d490 0 01d46618 7340/8192 update _cpu_usage

Now suppose you see that the CPU utilization is running a consistent 25 percent, and you want to know why. Although it is somewhat involved, you can discover which processes are "hogging" the CPU by following these steps:

1.	Run the show processes command. Copy the output and paste it to another location, such as Windows Notepad.
2.	Wait about a minute, and run the show processes command again. Copy and paste this output into another location.
3.	For each process listed, subtract the most recent runtime from the first runtime shown. The result is the number of milliseconds the CPU spent on that process during the last minute.

You can use a spreadsheet application such as Microsoft Excel to quickly compute the runtime difference on all the processes. Table 10-1 shows an example from a PIX 535 firewall. The actual runtime difference is shown in the rightmost column.

Table 10-1. Computing Actual Firewall Process Runtimes
Process Name	Runtime t=0	Runtime t=60 seconds	Msec CPU in 1 minute
arp_timer	0	0	0
FragDBGC	10	10	0
VAC poll	0	0	0
dbgtrace	0	0	0
Logger	305450	305450	0
tcp_fast	0	0	0
tcp_slow	0	0	0
xlate clean	0	0	0
uxlate clean	0	0	0
tcp_intercept_timer_process	0	0	0
route_process	0	0	0
PIX Garbage Collector	11200	11200	0
84 isakmp_time_keeper	0	0	0
perfmon	0	0	0
IPsec timer handler	0	0	0
qos_metric_daemon	0	0	0
IP Background	0	0	0
pix/trace	0	0	0
pix/tconsole	0	0	0
84 ci/console	56510	56510	0
lu_ctl	580	580	0
update_cpu_usage	0	0	0
84 uauth_in	0	0	0
uauth_thread	0	0	0
udp_timer	0	0	0
i82543_timer	2877196270	2877208860	12590
557mcfix	0	0	0
557poll	1816337704	1816363294	25590
557timer	0	0	0
pix/intf0	545907020	545915350	8330
riprx/0	0	0	0
riptx/0	0	0	0
pix/intf1	401710340	401715820	5480
riprx/1	0	0	0
riptx/1	0	0	0
pix/intf2	344850	344850	0
riprx/2	0	0	0
riptx/2	0	0	0
pix/intf3	0	0	0
riprx/3	0	0	0
riptx/3	0	0	0
lu_xmit_timer	0	0	0
lu_rx	7651280	7651280	0
fover_thread	340	340	0
fover_rx	49280	49280	0
fover_tx	5680	5680	0
fover_rep	0	0	0
fover_lu_rep	0	0	0
84 fover_parse	96370	96370	0
68 turboacl_process	3060	3060	0
radius_rcvauth	0	0	0
radius_rcvacct	0	0	0
radius_snd	0	0	0
isakmp_receiver	80	80	0
listen/ssh_1	0	0	0
Crypto CA	30	30	0
listen/http1	0	0	0
ssh/timer	0	0	0

From this information, you can see that the only processes that have used CPU time are i82543_timer, 557_poll, pix/intf0, and pix/intf1. Although some of the process names are intuitive, these aren't particularly obvious:

i82543_timer A timer maintained for the firewall's Gigabit Ethernet interfaces. These interfaces use an Intel 82543 controller chip.

557_poll A process that polls each Ethernet-based interface. When packets arrive at an interface, they are held until the CPU polls and retrieves them for further processing. 557_poll usually is the most-used process, because data is always arriving at the firewall's door.

pix/intf0 A process that updates the interface counters and statistics for firewall interface intf0. In this example, that is the "outside" Gigabit Ethernet interface.

pix/intf1 A similar process for the intf1 interface. In this example, that is the "inside" Gigabit Ethernet interface.

During this 60-second example, the CPU has only been involved in retrieving packets from the outside and inside interfaces. The actual ASA algorithm is being performed too, although it isn't broken out in the list of processes. Notice that almost all the processes are related to I/O, where data must be moved into or out of the firewall in some fashion. The internal processes simply aren't shown because a firewall can become overloaded only by interacting with external things.

Firewall Memory

Cisco firewalls base their entire operation on the use of internal RAM memory. This memory can be broken up and allocated for many processes and other uses. Some examples of memory usage are as follows:

OS image A firewall's operating system image is stored in Flash (nonvolatile) memory. When the firewall is first powered up or reloaded, a small bootstrap executable copies the image from Flash into RAM. From that point on, the CPU runs the image from RAM.

Configuration A firewall's configuration is stored in Flash (nonvolatile) memory with the write mem command. When the firewall boots, the configuration file is copied from Flash to RAM, where each command is executed.

Turbo ACLs In PIX 6.3, when an access list is "compiled" into a turbo ACL, the results are stored in an area of RAM separate from the running configuration. Compiling TurboACLs takes a minimum of 2.1 MB of memory; the more complex the original ACL is, the more memory its compiled version can require. After each ACL is compiled, it too requires at least 2.1 MB of memory. PIX 7.x automatically compiles all ACLs and does not have this memory requirement.

Downloadable ACLs If the firewall is configured to accept per-user ACLs that are down-loaded from an authentication server, the contents of those dynamic ACLs are stored in memory.

Interactive processes Management sessions through the console, Telnet, and SSH can all require memory. For example, when a command's output is filtered with the | include keywords, the original output is buffered in memory before being filtered to the session. Other processes include user authentication (uauth), content filtering, and VPN security associations (SAs).

Routing information The firewall maintains its own "routing table" in memory so that it can make packet-forwarding decisions. Routes are learned through static route configuration and through dynamic routing protocols such as RIP and OSPF. The firewall also maintains an ARP cache in memory for MAC-to-IP address resolution.

Capture sessions Each active capture session is immediately allocated a block of 512 KB of memory to use as a packet-capture buffer.

Xlate table entries As address translations are created, they are added to the xlate table in memory.

Conn table entries As connections are established through the firewall, they are added to the conn table in memory.

Packets awaiting stateful inspection As packets are pulled from interface queues, they are stored in memory. The firewall CPU works through this packet buffer as it inspects the traffic with its fixup procedures.

From a monitoring standpoint, firewall memory is organized in two ways:

Main memory The entire contents of the firewall's RAM memory.

Blocks Portions of main memory that are allocated as pools of fixed-size entries. Each size block is used for a specific purpose.

Likewise, you have two ways to query a firewall's memory usage. To see the utilization of the firewall's main memory, you can use the show memory command. The following example shows a PIX 525 with 256 MB of RAM, which is seen to have about 85 MB in use, whereas about 183 MB of memory is still free. The percentages are values shown by PIX 7.x as a quick gauge.

Firewall# show memory Free memory: 183131072 bytes (68%) Used memory: 85304384 bytes (32%) ------------- ---------------- Total memory: 268435456 bytes (100%) Firewall#

You can use the show memory detail command to display statistics about how the firewall memory has been fragmented during allocations. However, this information is useful mainly to Cisco TAC engineers.

You can also see how the firewall is managing its block memory with the show blocks command. Consider the following block statistics for a PIX 535:

Firewall# show blocks SIZE MAX LOW CNT 4 100 93 99 80 100 98 100 256 1100 1025 1036 1550 2688 1912 1920 2560 40 40 40 4096 30 30 30 8192 60 60 60 16384 100 100 100 65536 10 10 10 Firewall#

Notice that statistics are shown for each different block size, ranging from 4-byte blocks up to 65,536-byte blocks. Table 10-2 lists the possible block sizes and their purposes.

Table 10-2. Firewall Memory Blocks and Their Uses
Size in Bytes	Description
4	A general-purpose "scratch pad" for a variety of housekeeping functions: DNS, ISAKMP, URL filtering, uauth, H.323, TFTP, and some TCP uses
80	Failover "hello" messages between failover units TCP ACK packets that are sent during TCP intercept (limiting embryonic connections)
256	Syslog messages Stateful failover updates TCP
1550	Queue for Ethernet packets awaiting inspection (10/100 and Gigabit Ethernet interfaces other than i82543) Buffer for URL-filtering packets
2560	IKE messages
4096	QoS metrics
8192	QoS metrics
16384	Queue for Gigabit Ethernet packets awaiting inspection (64-bit 66-MHz i82543 interfaces only)
65536	QoS metrics

A firewall begins by allocating a default number of each block size when it boots up. The default number of blocks in each size varies with the firewall model, the number and type of interfaces, and the amount of available memory. For example, a PIX 535 might begin by allocating 1,444 1550-byte blocks.

During operation, a firewall might use most or all of the blocks of a certain size. This might occur if Ethernet packets arrive faster than they can be inspected and processed. When the number of blocks approaches 0, the firewall attempts to allocate more blocks of that size from the available memory.

The output from the show blocks command reports the current state of each block size. The count labels, MAX, LOW, and CNT, are not very intuitive; you should think of these in relation to blocks that are available, not blocks that are used. The count labels have the following meanings:

MAX The maximum number of blocks that are available for use. The maximum can increase if additional blocks are allocated at some point.

LOW The lowest number of blocks that have been available since the firewall was booted.

CNT The number of blocks currently available for use.

You can use the clear blocks command to return the maximum number of available blocks to the system defaults and to set the low block counts to the currently available values. This command can be useful if you notice that an extraordinary number of blocks have been allocated and you want to bring the firewall block allocation closer to its default state without a reboot.

Normally, you shouldn't see any of the CNT values staying at 0. If some values do tend to remain at 0, the firewall cannot allocate any more memory to the block size shown.

TIP

Notice that some firewalls set aside memory blocks for Ethernet (1550-byte) and Gigabit Ethernet (16384-byte). Why is there a difference? The 1550-byte blocks are used for traditional Ethernet, for both 10/100 and lower-performance Gigabit Ethernet interfaces. The maximum transmission unit (MTU) for Ethernet is 1500 bytes.

The 16384-byte blocks are reserved for firewalls that have higher-performance Gigabit Ethernet interfaces (based on the Intel i82543 controller) installed. These interfaces also use an MTU of 1500 bytes. However, the firewall can achieve better performance by moving the 1500-byte packets into and out of the interface in large numbers. In other words, the best performance is obtained when about ten packets are moved at a time.

To see what types of interface controllers your firewall has, you can use the show interface command. You can focus on the controller hardware by filtering the output with the show interface | include (line protocol | Hardware) command, as shown in this example:

Firewall# show interface | include (line protocol | Hardware) interface gb-ethernet0 "outside" is up, line protocol is up Hardware is i82543 rev02 gigabit ethernet, address is 0003.47df.8580 interface gb-ethernet1 "inside" is up, line protocol is up Hardware is i82543 rev02 gigabit ethernet, address is 0003.47df.85fc interface ethernet0 "stateful" is up, line protocol is up Hardware is i82559 ethernet, address is 0002.b3ad.8466 interface ethernet1 "dmz" is up, line protocol is up Hardware is i82559 ethernet, address is 0002.b3ad.7f4b Firewall#

With PIX 7.x, the results are very similar:

Firewall# show interface | include (line protocol | Hardware) Interface GigabitEthernet0 "outside", is up, line protocol is up Hardware is i82542 rev03, BW 1000 Mbps Interface Ethernet0 "", is administratively down, line protocol is up Hardware is i82559, BW 100 Mbps[output omitted]

Checking Stateful Inspection Resources

As a firewall inspects and passes traffic, it maintains two tables of entries: address translations (xlates) and connections (conns). You can get an idea of the inspection load by looking at the size of these tables.

Xlate Table Size

To see the translation table size, use the following command:

Firewall# show xlate count

The output from this command shows the current number of xlates in use and the maximum number that have been built since the firewall was booted. The Cisco PIX Firewall in the following example currently has built 15,273 translations. At some time in the past, a maximum of 22,368 xlates were in use:

Firweall# show xlate count 15273 in use, 22368 most used Firewall#

The xlate count is the sum of both static xlates (from the static command) and dynamic xlates (from nat and global commands).

When you see the xlate count, it might not be obvious whether there are too many translations. When a host passes through the firewall, it can use only one translation if it falls in a static mapping. If a host triggers a dynamic translation, the firewall creates one xlate for every unique PAT connection.

In other words, it isn't possible to estimate or calculate ahead of time the number of xlates that will be used. Instead, be sure to look at the xlate count periodically so that you can get an average baseline value. Then, if you see the xlate count jump to a much higher value later, you can assume that something is wrong. In that case, your firewall could be faced with building xlates during an attack of malicious activity.

Conn Table Size

A firewall creates and tears down connection entries for UDP and TCP connections between pairs of hosts. You can get connection statistics with the following command:

Firewall# show conn count

The output of this command shows the current number of connections in use and the maximum number of connections that have been built since the firewall was booted. The firewall in the following example has 4,495 connections currently in its conn table, and it has had up to 577,536 simultaneous connections in the past!

Firewall# show conn count 4495 in use, 577536 most used Firewall#

The conn count is the sum of all types of connections. The count fluctuates as connections are built and torn down. Again, you should periodically use this command to get a feel for the average number of connections your network uses. If the conn count is excessive, as in the maximum number in the preceding example, an attack is likely in progress.

If you have a failover pair of firewalls, shouldn't the xlate and conn counts be identical in each unit? The xlate and conn entries are replicated only from the active unit to the standby unit if you have configured stateful failover. Otherwise, the active unit shows positive counts for both xlates and conns, but the standby unit shows counts of 0.

Also, it is not unusual for the active and standby table counts to be quite different, even when stateful failover is being used. The entire xlate and conn tables are not replicated between the units. Rather, only new entries (either created or torn down) are replicated.

If the stateful failover link goes down, the standby unit misses new table entries. When the link comes back up, only new entries from that point on are received. If the standby unit loses power or reboots, its entire xlate and conn tables are lost. When it comes back up, it receives the continual flow of only new entriesnot the entire contents of the tables. Existing conn table entries from the active unit are replicated to the standby unit only if they are actively passing data. Existing connections that are idle are not replicated.

Also, the two units might show a disproportionate number of conn table entries if HTTP replication is not enabled between the failover pair. In this case, the active unit maintains HTTP connection entries but does not replicate those to the standby unit.

A failover pair has been configured for stateful failover in the following example. Somewhere along the way, the standby unit lost power and was rebooted. Notice that the table counts are quite different; the standby unit lost its tables and has received only new table changes, and HTTP replication has not been configured on the failover pair. The active unit is shown first, followed by the standby unit.

[Active unit] Firewall# show conn count 4263 in use, 577536 most used Firewall# show xlate count 15166 in use, 22368 most used Firewall# ________________________________________________________________ [Standby unit] Firewall# show conn count 686 in use, 690 most used Firewall# show xlate count 659 in use, 665 most used Firewall#

Checking Firewall Throughput

Many of the firewall statistics that you might display are based on incrementing counters or "snapshot" values. These give you an idea of the volume of activity over a long period of time, but not of the rate. For example, to gauge your firewall's throughput, you might want to see the number of bytes per second being forwarded on an interface or the number of TCP connections per second that are being inspected.

A Cisco firewall keeps several running statistics that you can display. You also can use an external application to perform some analysis on firewall counters and messages. The following sections describe several ways to show the firewall throughput.

PDM

The PIX Device Manager (PDM) default view shows several useful throughput calculations. Figure 10-1 shows a sample PDM display, where you can determine the following throughput measures:

Current interface throughput, in kbps, is shown in the upper-right portion of the display. This is the aggregate or total of input and output rates. (You can select an interface to see the current input and output throughput values.)

UDP and TCP connections per second, in the middle-right portion of the display.

A history of the first (lowest-numbered) interface throughput, in kbps, shown in the lower right of the display. Separate graphs are shown for input and output throughput. Figure 10-1 shows a graph of the "outside" interface activity.

Figure 10-1. A Sample PDM Display of Firewall Throughput

Syslog

Some Syslog analysis applications can parse the history of Syslog messages generated by a firewall. If the firewall is configured to report about connections being set up and torn down (logging severity level 6, informational, by default), the Syslog analyzer can calculate the number of connections per second, the interface data rate per second, and so on.

For example, a Syslog analysis tool is used to present information about the bandwidth being passed through firewall interfaces. Figure 10-2 shows a graph of utilization per unit time, as "bandwidth usage per hour." In this case, rather than showing the throughput for an individual interface, the total aggregate bandwidth for all interfaces is shown over time.

Figure 10-2. An Example of Firewall Throughput Reporting by Syslog Analysis

Traffic Counters

Cisco firewalls can report traffic throughput on each interface through the command-line interface (CLI). This can be handy if you are connected to a firewall over a console, Telnet, or SSH session, and you want to check the throughput.

The firewall keeps running counters of input and output data on each interface while it is operational. These counters begin at bootup or at the last counter reset, and they accumulate until you issue the command to display them. The firewall also computes the average throughput in bytes per second, but this is based on the total time elapsed since the counters were last reset.

To see the traffic counters and throughput information, follow these steps:

(Optional) Reset the traffic counters:

Firewall# clear traffic

The data counters, rates, and elapsed time values are reset to 0 for each interface. This step is important so that you establish a time frame for taking throughput measurements.

NOTE

If you don't begin by resetting the counters, you might end up displaying traffic statistics that have been running for a very long time, as in this example:

Firewall# show traffic outside: received (in 509866.850 secs): 785027658 packets 2630237219 bytes 1000 pkts/sec 5007 bytes/sec transmitted (in 509866.850 secs): 640777558 packets 494377136 bytes 1004 pkts/sec 0 bytes/sec inside: received (in 509866.850 secs): 642131249 packets 1622203492 bytes 1006 pkts/sec 3004 bytes/sec transmitted (in 509866.850 secs): 770623656 packets 1358715766 bytes 1005 pkts/sec 2007 bytes/sec

The counters have been accumulating for 509,866 seconds, or about 5.9 days. Because the data throughput values fluctuate all the time, this is not an accurate estimate of the current throughput.

Wait, and then display the traffic statistics:

Firewall# show traffic

You should wait a few seconds or even a minute before displaying the traffic statistics. The byte and packet counters shown will have accumulated over that time. The byte and packet rates (per second) will be calculated as an average over that time. For example, the following firewall's statistics were shown after 70 seconds had elapsed. Its outside interface (Gigabit Ethernet) received an average of 2,826,360 bytes per second over those 70 seconds.

Firewall# show traffic stateful: received (in 70.750 secs): 15 packets 1560 bytes 0 pkts/sec 22 bytes/sec transmitted (in 70.750 secs): 1096 packets 1307978 bytes 15 pkts/sec 18487 bytes/sec outside: received (in 70.750 secs): 258717 packets 199964989 bytes 3656 pkts/sec 2826360 bytes/sec transmitted (in 70.750 secs): 210554 packets 48903164 bytes 2976 pkts/sec 691210 bytes/sec inside: received (in 70.750 secs): 210849 packets 53713701 bytes 2980 pkts/sec 759204 bytes/sec transmitted (in 70.750 secs): 256901 packets 166768097 bytes 3631 pkts/sec 2357146 bytes/sec Firewall#

Beginning with PIX 7.x, the show traffic command breaks down the traffic statistics based on logical interfaces and on the aggregate performance of physical interfaces. For example, a firewall might have two physical interfaces, GigabitEthernet0 and GigabitEthernet1. You can assign each of these a logical name such as outside or inside.

In addition, you can configure the physical interface to trunk multiple VLANs; each VLAN subinterface (GigabitEthernet0.1, for example) can become a unique logical interface (dmz, intf3, and so on) while remaining a part of the aggregate physical interface.

In the following example, logical interfaces outside, inside, and Test are mapped from physical interfaces GigabitEthernet0, GigabitEthernet1.1, and GigabitEthernet1.2, respectively.

Firewall# show traffic outside: received (in 2701.540 secs): 15754 packets 12405921 bytes 5 pkts/sec 4592 bytes/sec transmitted (in 2701.540 secs): 1447 packets 126813 bytes 0 pkts/sec 46 bytes/sec inside: received (in 2701.550 secs): 1533 packets 132357 bytes 0 pkts/sec 48 bytes/sec transmitted (in 2701.550 secs): 15936 packets 12364101 bytes 5 pkts/sec 4576 bytes/sec Test: received (in 2701.550 secs): 185 packets 19980 bytes 0 pkts/sec 7 bytes/sec transmitted (in 2701.550 secs): 276 packets 29808 bytes 0 pkts/sec 11 bytes/sec ---------------------------------------- Aggregated Traffic on Physical Interface ---------------------------------------- GigabitEthernet0: received (in 2705.540 secs): 15782 packets 12706533 bytes 5 pkts/sec 4696 bytes/sec transmitted (in 2705.540 secs): 1447 packets 157397 bytes 0 pkts/sec 58 bytes/sec GigabitEthernet1: received (in 2705.540 secs): 1541 packets 171097 bytes 0 pkts/sec 63 bytes/sec transmitted (in 2705.540 secs): 15959 packets 12695684 bytes 5 pkts/sec 4692 bytes/sec Firewall#

Aggregate interface performance also comes into play for firewalls that are configured for multiple-context security mode. Each context has its own set of logical interfaces (inside and outside, for example) that are mapped from a physical interface or subinterface in the system execution space. From any user context, show traffic shows only the logical interfaces used by that context. However, the system execution space shows a breakdown that includes the aggregate physical interfaces.

Perfmon Counters

A Cisco firewall also keeps statistics about its stateful inspection performance. These values are called performance monitors or perfmon. From this information, you can get a good idea about the types of traffic passing through the firewall. You can also use the displayed rates to see a load distri-bution of the inspection or fixup processes.

To configure and view the performance statistics, follow these steps:

(Optional) Set the perfmon calculation interval:

Firewall(config)# perfmon interval seconds

Perfmon statistics are calculated and optionally reported at intervals of seconds (the default is 120 seconds). This means the results are computed as averages over this interval of time.

(Optional) Automatically display perfmon results on the firewall console:

Firewall(config)# perfmon {verbose | quiet}

By default, a firewall displays perfmon information only when a command is entered manually, as quiet mode.

The firewall can automatically display the perfmon statistics at each interval of time if you use the verbose keyword. As soon as this is enabled, the results are displayed only on the firewall consolenot on any active Telnet or SSH sessions. This might be handy if you have a terminal connected to the console and you want to see the firewall performance continually updated on the screen. However, if you choose too short an interval, the console connection can become very congested with new perfmon information. As a result, you might not be able to interact with the firewall over the console connection.

Display the perfmon statistics:

Firewall# show perfmon

Firewall performance is reported in several groups, with the following labels:

- New xlate entry creation "Xlates"

- New connection entry creation "Connections" (the total of all connection types), "TCP Conns," and "UDP Conns"

- Web content filtering "URL Access" (new URLs being requested; reported even if Websense or N2H2 servers aren't being used) and "URL Server Req" (new Websense and N2H2 server requests)

- Fixups "TCP Fixup" (new TCP packets inspected), "TCPIntercept" (TCP intercept or embryonic connection inspections if an embryonic connection limit is set), "HTTP Fixup" (new HTTP packets inspected), and "FTP Fixup" (new FTP packets inspected)

- User authentication "AAA Authen" (uauth authentication requests processed), "AAA Author" (uauth authorization requests), and "AAA Account" (uauth accounting requests)

In the following example, the firewall has just created 107 new TCP connections, inspected 6196 TCP packets, and inspected 2631 HTTP packets, all in the previous second:

Firewall# show perfmon PERFMON STATS: Current Average Xlates 53/s 0/s Connections 146/s 0/s TCP Conns 107/s 0/s UDP Conns 38/s 0/s URL Access 79/s 0/s URL Server Req 0/s 0/s TCP Fixup 6196/s 0/s TCPIntercept 0/s 0/s HTTP Fixup 2631/s 0/s FTP Fixup 3/s 0/s AAA Authen 0/s 0/s AAA Author 0/s 0/s AAA Account 0/s 0/s Firewall# The Current values are calculated as the number of operations over the last second before the command was issued. The Average values are calculated as an average over the perfmon reporting interval, or the last 120 seconds by default. (Because of a cosmetic bug in the PIX operating system, all the average values might be shown as 0/s.)

TIP

If you are unsure of the perfmon settings, you can use the perfmon settings command to see them. Notice that this command doesn't use a show keyword, like most other EXEC firewall commands. The perfmon interval and reporting mode are shown.

Checking Inspection Engine and Service Policy Activity

Beginning with PIX 7.x, application inspection is performed by independent inspection engines that are referenced in service policies. You can get information about the activity of the various inspection engines by displaying the active service policies.

One service policy can be applied to a firewall interface to define the actions to take on matching traffic in the inbound and outbound directions. A default service policy also is configured by default and is applied to all firewall interfaces. Any traffic not matched by an interface service policy is matched by the default global service policy.

You can use the following command to display all active service policy statistics:

Firewall# show service-policy

The modular policy configuration of each service policy is shown. This includes the target interface, the service policy name, the class map used to match traffic, and each policy action.

For each inspection engine, the number of packets inspected, dropped, and dropped with reset are shown. Packets are counted only in the service policy where they are matched and inspected. The default global service policy matches all packets that aren't matched elsewhere.

In the following example, notice the packet counts for HTTP traffic and how each service policy has matched a different set of packets to inspect. The output of this command gives you a quick snapshot of all the inspections and actions that have been configured on the firewall and are actually being used.

Firewall# show service-policy Global policy: Service-policy: asa_global_fw_policy Class-map: inspection_default Inspect: dns maximum-length 512, packet 363, drop 0, reset-drop 0 Inspect: ftp, packet 0, drop 0, reset-drop 0 Inspect: h323 h225, packet 0, drop 0, reset-drop 0 Inspect: h323 ras, packet 0, drop 0, reset-drop 0 Inspect: rsh, packet 0, drop 0, reset-drop 0 Inspect: rtsp, packet 26601, drop 0, reset-drop 0 Inspect: esmtp, packet 1668, drop 0, reset-drop 0 Inspect: sqlnet, packet 0, drop 0, reset-drop 0 Inspect: skinny, packet 0, drop 0, reset-drop 0 Inspect: sunrpc, packet 0, drop 0, reset-drop 0 Inspect: xdmcp, packet 0, drop 0, reset-drop 0 Inspect: sip, packet 0, drop 0, reset-drop 0 Inspect: netbios, packet 0, drop 0, reset-drop 0 Inspect: tftp, packet 0, drop 0, reset-drop 0 Inspect: http, packet 36614, drop 0, reset-drop 0 Inspect: icmp, packet 3911, drop 0, reset-drop 0 Inspect: icmp error, packet 171, drop 0, reset-drop 0 Class-map: asa_class_tftp Inspect: tftp, packet 0, drop 0, reset-drop 0 Interface outside: Service-policy: test-policy Class-map: test Inspect: http, packet 369, drop 0, reset-drop 0 Priority: Interface outside: aggregate drop 0, aggregate transmit 0 Interface inside: Service-policy: PolicyA Class-map: http_class Inspect: http test_http, packet 99400, drop 41, reset-drop 0 Class-map: ftp_class Inspect: ftp strict Filter_ftp, packet 696, drop 0, reset-drop 0 Class-map: test Priority: Interface inside: aggregate drop 0, aggregate transmit 0 Firewall#

Checking Failover Operation

If you have a failover pair of firewalls, you should periodically check to see that the failover mechanisms are actually working properly. Use the techniques described in the following sections to gauge the failover performance.

Verifying Failover Roles

First, you should verify that the active failover unit is indeed the one you are expecting. When a failover pair is initially configured for failover, only one of them becomes the active unit. The other assumes the standby (passive) mode. If a failover occurs, the two units swap roles.

Recall that the two failover units also have a "primary" and "secondary" designation. This has nothing to do with the actual failover operation, other than to distinguish one unit from the other. Usually, the secondary unit is purchased with a failover license at a lower price.

Why are the failover units hard to distinguish from each other? The active unit always uses the active IP addresses on its interfaces, and the standby unit uses the standby IP addresses. As soon as a failover happens, the two units swap IP addresses. (Keep in mind that the same thing happens with MAC addresses.) This means if you open a Telnet or SSH session to the active IP address on an interface, you won't know which physical unit answers. To make matters more difficult, both failover units also have the same host name and command-line prompt!

After you open a session, you can use the show failover command to learn the identity of the physical unit (primary or secondary), as well as its current failover role (active or standby).

For example, suppose a failover pair is configured with an inside active address of 192.168.254.1 and an inside standby address of 192.168.254.2. When failover is first enabled, the primary unit takes on the active failover role.

An SSH session is opened to the active address 192.168.254.1. You would use the show failover command to see which unit is currently active:

Firewall# show failover Failover On Cable status: Normal Reconnect timeout 0:00:00 Poll frequency 15 seconds Last Failover at: 08:34:03 EST Sun Dec 28 2003 This host: Primary - Active Active time: 7304955 (sec) Interface stateful (192.168.199.1): Normal Interface dmz2 (127.0.0.1): Link Down (Shutdown) Interface outside (172.16.110.65): Normal Interface inside (192.168.254.1): Normal Other host: Secondary - Standby Active time: 2770785 (sec) Interface stateful (192.168.199.2): Normal Interface dmz2 (0.0.0.0): Link Down (Shutdown) Interface outside (172.16.110.66): Normal Interface inside (192.168.254.2): Normal [output deleted]

In the highlighted output, it's easy to see that failover is enabled (on), that the host to which you're connected is the primary unit, and that it has the active role. Notice that the pair of units is polling each other every 15 seconds (the default). This means that it takes up to two or three poll intervals (30 to 45 seconds) for one unit to detect that the other unit has failed.

You can also use this command to quickly see the status of every firewall interfaceon both units at the same time. For each interface that is being used, you should see the Normal status listed. If you see Waiting, one of the units has missed one hello message from the other and suspects there might be a problem. Testing means that three hellos have been missed, so the interface is currently going through a series of tests. If the tests fail, the interface is marked as Failed.

Verifying Failover Communication

If failover is enabled, you might want to verify that the two units are communicating properly over the failover links. Cisco firewalls can exchange failover information in three ways:

Failover cable An asynchronous serial data cable (DB-9 connectors) connects the two units. Failover information is exchanged at 115,200 kbps.

LAN-based failover A firewall interface (a physical Fast or Gigabit Ethernet interface only) is used to exchange data between the units. This allows the failover units to be geographically separated.

Stateful failover Information about new connections, xlates, and uauth entries is sent from the active unit to the standby unit. The standby unit receives these updates so that it can perform a stateful failover when assuming the active role. (Stateful failover can be used in addition to the failover cable or LAN-based failover.)

If a failover cable is being used, look for the cable status in the show failover command output:

Firewall# show failover Failover On Cable status: Normal Reconnect timeout 0:00:00 Poll frequency 15 seconds Last Failover at: 08:34:03 EST Sun Dec 28 2004 This host: Primary - Active [output deleted]

In this example, the cable is in place and the status is Normal. This means the two units are ex-changing failover information successfully. With the failover cable, each unit can also determine if the other unit is powered on. If the companion unit has lost power, the cable status shows other side is powered off.

TIP

Keep in mind that the failover cable itself determines which unit is primary and which is secondary. One end of the cable is labeled "primary" and should be plugged into the unit that has the primary firewall license. Obviously, the other end of the cable plugs into the secondary unit, usually the one with the failover license.

If you are using LAN-based failover, the status of the failover cable is irrelevant. The status of the interface dedicated to LAN-based failover communication, however, is important. The primary and secondary failover units are identified through configuration commands.

With PIX 6.3, rather than sifting through output showing the status of the LAN-based interface, you can quickly see the status with the show failover lan command. In the following example, the dmz interface is being used for failover communication. The failover peer at each end of the LAN connection is seen to be Normal.

Firewall# show failover lan Lan Based Failover is Active interface dmz (192.168.1.1): Normal, peer (192.168.1.2): Normal

PIX 7.x and later doesn't have this command. Instead, look at the first few lines of the show failover command output:

NOTE

Unlike with the failover cable, it isn't possible to detect the power status of the other unit with LAN-based failover. The cable carries a power signal from each unit so that it is easy to sense a loss of power. No power signals can be carried over a LAN-based failover connection, simply because only IP packets can be exchanged between the two units. If one unit has lost power, nothing in the IP failover packets indicates that. The other unit notices only the absence of failover packets.

In all releases except PIX 7.x, you can also use the show failover lan detail command to add a generous amount of debugging information. Most of the output messages are coded values that aren't intuitive. However, you can see failover message counters and retransmission queue statistics that show how congested the LAN-based failover link has been.

If you are concerned about two firewalls failing over with little impact to a production network, you have likely configured stateful failover. This type of failover works in conjunction with a failover cable or LAN-based failover. The basic housekeeping functions are communicated over the cable or the LAN interface, and the stateful interface is reserved for sending dynamic updates about connection or translation entries.

If it works properly, stateful failover keeps the standby unit fully informed about the state of every active TCP and UDP connection in the active unit. In addition, the xlate table entries and ARP table entries are replicated. Should the standby unit need to take over, it already has the stateful infor-mation and can preserve existing connections during the failover transition.

A failover pair keeps detailed statistics about the stateful information exchange. You'll find this at the end of the show failover command output. You should only be concerned about verifying effective stateful updates so that the two firewall units stay synchronized at all times.

For example, the active unit shows the following output from the show failover command in PIX 7.x:

Firewall# show failover Failover On Cable status: N/A - LAN-based failover enabled Failover unit Primary Failover LAN Interface: Failover Ethernet2 (up) Unit Poll frequency 3 seconds, holdtime 9 seconds Interface Poll frequency 15 seconds Interface Policy 2 Monitored Interfaces 3 of 250 maximum Group 1 last failover at: 10:29:18 EST Jan 30 2005 Group 2 last failover at: 10:29:27 EST Jan 30 2005 Stateful Failover Logical Update Statistics Link : Failover Ethernet2 (up) Stateful Obj xmit xerr rcv rerr General 0 0 0 0 sys cmd 13531 0 13531 0 up time 0 0 0 0 RPC services 0 0 0 0 TCP conn 0 0 0 0 UDP conn 0 0 0 0 ARP tbl 29 0 0 0 Xlate_Timeout 0 0 0 0 Logical Update Queue Information Cur Max Total Recv Q: 0 1 13531 Xmit Q: 0 1 13573 Firewall#

Here, the number of these types of replicated stateful messages are shown:

General The total number of stateful messages

sys cmd System commands that have been replicated to the other unit

up time Uptime reports (the elapsed time since the unit has been booted)

RPC services Remote Procedure Call entry updates (PIX 7.x)

xlate Translation or xlate table entry updates (FWSM and PIX 6.3 only)

TCP conn Conn table TCP connection entry updates

UDP conn Conn table UDP connection entry updates

ARP tbl ARP table entry updates

L2BRIDGE tbl MAC address table entry updates (PIX 7.x transparent firewall mode onlynot shown in the preceding output)

RIP Tbl RIP route advertisement updates (FWSM and PIX 6.3 onlynot shown in the preceding output)

Xlate Timeout Xlate entries removed after timeout (PIX 7.x only)

The xmit and rcv columns show how many of each message type have been transmitted or received by this firewall, respectively. While the unit is in active mode, you should see the transmit counters increasing much more than the receive counters. This is because the active unit is tasked with keeping the standby unit updated.

The xerr and rerr columns show the number of transmit and receive errors encountered while exchanging messages. If you find a large number of transmit errors, the sending firewall unit could not successfully send failover messages because of network congestion or a slow LAN interface. Receive errors indicate failover messages that arrived corrupted.

Determining if a Failover Has Occurred

If a failover pair of firewalls is operating correctly, and stateful failover is being used to synchronize the state information, you may never realize when a failover takes place. How can you determine if the units have failed over?

You can use the show failover command to see a record of the last failover event. The output from this command displays the date and time, along with the total amount of time that each unit has assumed the active role. In the following example, the failover occurred on December 28 at 8:34:03 a.m. Be aware that you must have already set the firewall clock (both active and standby units) or have configured the units to use NTP. Otherwise, the failover time stamp will be incorrect, and you'll have no idea when it actually occurred.

At this point, you should also take note of the failover roles. When this command was entered, the primary unit was in the active role. This means that when the failover occurred, the other unit (the secondary) was active.

The Active time values only serve to give you an idea of how much time each unit has spent in the active role. This is the total elapsed active time since the last rebootnot the amount of time the unit has been active since the last failover. In the sample output, the two units may have failed over more than once. Only the last failover event is noted, and you have no knowledge of any previous ones. Therefore, these units might have traded roles and accumulated active duty time on several occasions.

If Syslog messages have been generated and recorded, you can find a detailed record of each failover and the symptoms surrounding it, complete with time stamps.

Determining the Cause of a Failover

Now consider the importance of knowing why a failover has happened. For a failover to be triggered, one or both firewalls must have detected a problemeither the other unit was unresponsive with failover polls, or an interface had a problem. If a problem exists, you should try to identify it and get it fixed; otherwise, you might be left with only one working firewall out of the pair.

TIP

For proper failover operation, each firewall unit must be able to send failover messages to the other unit on every useable or monitored interface. This means you should make sure each interface on the primary unit can reach each corresponding interface on the secondary unit. A failure of just one interface can trigger a failover condition. (Beginning with PIX 7.x, you can configure a failover policy that triggers a failover when the number of failed monitored interfaces increases above a threshold.)

If you have firewall interfaces that aren't being used, make sure you shut down those interfaces with the interface hardware_id shutdown configuration command (PIX 6.3) or the shutdown interface configuration command (PIX 7.x). You should also make sure that unused interfaces don't have a valid IP address configured. You can use the ip address interface 0.0.0.0 255.255.255.255 configuration command. If any unused interface is left enabled, it could inadvertently trigger a failover just from a lack of connectivity between the failover units.

You can diagnose the cause of a failover event using one of the following two methods:

The status of interfaces reported by the show failover command

Failover messages recorded in the Syslog history

If a firewall interface fails for some reason, it might trigger a failover. However, if the interface becomes usable again, it is shown as Normal in the show failover output. If it fails and stays failed, you can use that command to find the broken interface, even at a later date.

The most detailed way to track down the failure is to sift through the Syslog message history. A Syslog server is a must here, because the failover event might be buried within many thousands or millions of other Syslog messages. The firewall logging buffer simply isn't large enough to store a long history of messages.

Usually, only the active failover unit is configured to generate "trap" (Syslog) logging to a Syslog server. The standby unit can generate its own Syslog messages, but that causes both units to send duplicate messages to the server. That doubles the amount of message storage required and is usually considered redundant.

Therefore, begin by finding the date and time of the failover event with the show failover command. This gives you a window of time to use when searching through the archived Syslog data.

Failover messages are generated with the identity of the sending firewall unit embedded in the message text: (Primary) or (Secondary). This is important, because it uniquely identifies which physical firewall unit is reporting the failover activity. If logging from the standby unit has not been enabled, you can conclude that any messages found must have come from the unit that was active at that time.

The active unit messages found on the Syslog server tell only half of the failover story; to find the standby unit's testimony, you have to look elsewhere. For this, it is handy to enable buffered logging in addition to trap (Syslog) logging on the active unit. Unlike trap logging, when buffered logging is enabled, it is enabled on both the active and standby units.

Therefore, the standby unit (after failover) has in its logging buffer a brief record of the failover. In fact, the failover messages are the very last messages recorded in the buffer. Why? This is because that unit was active up until the failover. It would have recorded all sorts of Syslog messages in its buffer, because they were also sent to the Syslog server. Any failover messages would be recorded there, up until the unit switched to standby mode. Then the unit becomes passive and doesn't really generate any further Syslog messages.

Therefore, use the search terms Primary or Secondary when you search through the Syslog messages. Then go to the standby unit (after failover) and get a record of its logging buffer with the show logging command.

An Example of Finding the Cause of a Failover

A failover has occurred within a pair of Cisco PIX Firewalls. From the show failover command, you find this time stamp:

Firewall# show failover Failover On Cable status: Normal Reconnect timeout 0:00:00 Poll frequency 15 seconds Last Failover at: 08:34:03 EST Mon Feb 28 2005 This host: Primary - Active Active time: 7319775 (sec)

On the Syslog server, you perform a search for the terms Primary and Secondary in the message text around that time frame. You're looking for any failover messages that might have been sent by the primary or secondary firewall units. Here are the results of the search:

Feb 28 2005 8:34AM Firewall LOCAL4 ALERT %PIX-1-104001: (Primary) Switching to ACTIVE - mate want me Active. Feb 28 2005 8:34AM Firewall LOCAL4 ALERT %PIX-1-105003: (Primary) Monitoring on interface 3 waiting Feb 28 2005 8:34AM Firewall LOCAL4 ALERT %PIX-1-105003: (Primary) Monitoring on interface 0 waiting Feb 28 2005 8:34AM Firewall LOCAL4 ALERT %PIX-1-105004: (Primary) Monitoring on interface 3 normal Feb 28 2005 8:34AM Firewall LOCAL4 ALERT %PIX-1-105004: (Primary) Monitoring on interface 0 normal Feb 28 2005 9:09AM Firewall LOCAL4 ALERT %PIX-1-105008: (Primary) Testing Interface 2 Feb 28 2005 9:09AM Firewall LOCAL4 ALERT %PIX-1-105009: (Primary) Testing on interface 2 Passed Feb 28 2005 9:09AM Firewall LOCAL4 ALERT %PIX-1-105008: (Primary) Testing Interface 2 Feb 28 2005 9:09AM Firewall LOCAL4 ALERT %PIX-1-105009: (Primary) Testing on interface 2 Passed Feb 28 2005 9:10AM Firewall LOCAL4 ALERT %PIX-1-105008: (Primary) Testing Interface 2 Feb 28 2005 9:10AM Firewall LOCAL4 ALERT %PIX-1-105009: (Primary) Testing on interface 2 Passed Feb 28 2005 9:10AM Firewall LOCAL4 ALERT %PIX-1-105008: (Primary) Testing Interface 2 Feb 28 2005 9:10AM Firewall LOCAL4 ALERT %PIX-1-105009: (Primary) Testing on interface 2 Passed Feb 28 2005 9:10AM Firewall LOCAL4 ALERT %PIX-1-105004: (Primary) Monitoring on interface 2 normal

From this record, it's evident that something began to happen at 8:34 a.m. on February 28. The primary unit reports that the other unit ("mate") has decided that it should assume the active role. Obviously, before this time, the primary unit had been in the standby role.

When the primary unit becomes active, it begins to monitor on several interfaces, determining if the interfaces are working properly and if it can detect the other unit on them too. Notice that at 9:09 a.m., it begins several testing phases on interface 2. Evidently, interface 2 (missing from the earlier monitor tests) had a problem from 8:34 until 9:09.

You still don't know what triggered the failover, because you have a record only from the unit that had just become active at that time. That also means that the other unit, which had been active before the failover, was generating other Syslog messages before the failover. However, the failover caused it to enter the standby mode, so no further Syslog information was sent to the server.

As a last step, you connect to that unit (currently in standby mode) and have a look at its logging buffer:

Firewall# show logging Syslog logging: enabled Facility: 20 Timestamp logging: disabled Standby logging: disabled Console logging: disabled Monitor logging: disabled Buffer logging: level warnings, 553574 messages logged Trap logging: level warnings, 553574 messages logged Logging to inside 192.168.100.100 History logging: disabled Device ID: disabled 305006: Dst IP is network/broadcast IP, translation creation failed for icmp src outside:64.170.37.34 dst inside:169.163.69.0 (type 8, code 0) 305006: Dst IP is network/broadcast IP, translation creation failed for icmp src outside:64.170.37.34 dst inside:169.163.69.0 (type 8, code 0) 500004: Invalid transport field for protocol=17, from 172.21.68.65/47808 to 1.0.0.0/0 [output deleted] 411002: Line protocol on Interface outside, changed state to down 105007: (Secondary) Link status 'Down' on interface 2 104002: (Secondary) Switching to STNDBY - interface check, mate is healthier 105003: (Secondary) Monitoring on interface 3 waiting 105003: (Secondary) Monitoring on interface 0 waiting 105004: (Secondary) Monitoring on interface 3 normal 105004: (Secondary) Monitoring on interface 0 normal 105006: (Secondary) Link status 'Up' on interface 2 105003: (Secondary) Monitoring on interface 2 waiting 105004: (Secondary) Monitoring on interface 2 normal Firewall#

Aha! The buffered record shows that the "outside" interface went down, triggering the failover. This unit decided that because its own interface went down hard, it should immediately relinquish the active role.

Unfortunately, logging time stamps were not configured, so the firewall didn't add any date and time information to the messages in its own buffer. (This is true only for PIX 7.x or later; earlier releases don't add time stamps to buffered logging messages.) This doesn't really matter, because these are the final recorded messages and must have occurred right before the last time the unit entered the standby role.

Intervening in a Failover Election

Cisco firewalls do not toggle their roles as failures come and go. For example, if an active unit fails, it automatically enters the standby role. Even if its failure is cured, it does not resume its former active role.

The idea is that after a failure has taken place, a network administrator needs to diagnose the problem and fix it. After a failed firewall unit is repaired, you have to manually inform the pair that they need to reset the failed condition in their failover status. When a problem is resolved on a failed standby unit, you can use the failover reset command on the active unit. Both units recognize that the standby unit has become "unfailed" and resume normal failover communication.

Sometimes you might need to manually toggle the roles. This might be necessary if you need to perform some maintenance on one unit, but it is unfortunately already in the active role. You can approach this in two ways:

Use the no failover active command on the active unit. It immediately relinquishes the active role to its failover peer.

Use the failover active command on the standby unit to force it to immediately assume the active role.

Checking Firewall Interfaces

You can use the show traffic command to see throughput information about firewall interfaces, but you can monitor other interface statistics as well. You can use the show interface command to see a wealth of information about the interface operation, many types of error conditions, and packet buffering.

As with the Cisco IOS software, the show interface command can produce such a condensed dump of interface parameters that it becomes difficult to interpret. To make this easier, think of the com-mand output as being broken into various sections.

Figure 10-3 shows an example of the show interface command in PIX 7.x. Other software releases show similar information but are organized slightly differently. Only the "inside" interface is shown, for clarity. If you just glance through the lines of output, looking for any glaring error condition, you are unlikely to find anything.

Figure 10-3. A Breakdown of Information Presented by the show interface Command

As the figure shows, the lines of output are organized into groups of related information. Each of these groups is explained in detail in the following sections to correspond with the figure. The parameters in each group appear in table format for quick reference. The sample values in Figure 10-3 are shown, along with an explanation of the parameter.

Interface Name and Status

Table 10-3 describes the interface name and status information displayed by the show interface command as depicted in Figure 10-3.

Table 10-3. show interface Command Output: Interface Name and Status Information
Parameter	Sample Value	Description
Hardware interface	GigabitEthernet1	The physical interface names listed by the show version command.
Interface name	inside	An arbitrary name configured by the nameif command.
Interface status	up	The interface is enabled (not shut down).
Line protocol	up	The Gigabit Ethernet link is established or up.

Interface Control

Table 10-4 describes the interface control information displayed by the show interface command as depicted in Figure 10-3.

Table 10-4. show interface Command Output: Interface Control Information
Parameter	Sample Value	Description
Hardware	i82542 rev03	The interface controller on a Gigabit Ethernet interface.
BW	1000 Mbps	The interface bandwidth (1 Gbps in this case).
Duplex mode	(Fullduplex)	The current mode can be full or half. If autonegotiation is configured, this appears as Auto-Duplex(Full duplex). This is configured with the interface (PIX 6.3) or duplex (PIX 7.x) command.
Speed	Auto-Speed(1000 Mbps)	The autonegotiated speed is 1 Gbps. This is configured with the interface (PIX 6.3) or speed (PIX 7.x) command.

The interface bandwidth or speed shown should match that of the network device on the other end of the connection.

However, be aware of problems caused by duplex mode configuration. Duplex mode must be configured identically on the firewall and the network device to which it connects. Duplex mode can be autonegotiated only if the other end of the connection is also set to autonegotiate. Otherwise, autonegotiation is unable to "sense" what mode the other side is using, and duplex mode defaults to half-duplex.

TIP

You can use the interface information to troubleshoot duplex mismatch problems. First, look at the duplex setting that is reported. If the interface is set to Auto-duplex and the actual mode is half-duplex, most likely the far end of the connection is not configured for autonegotiation. The firewall then attempts to negotiate duplex mode and falls back to half-duplex as a default.

In addition, nonzero values for collisions, late collisions, and output errors can also indicate a duplex mismatch. If one end is using fullduplex mode and the other end half-duplex mode, there is a good chance that the two devices will attempt to transmit at the same time and cause a frame collision. The resulting data becomes scrambled and appears as an output error.

Best practice is to hard-code or configure specific interface speed and duplex settings to avoid any problems with misconfiguration or autonegotiation.

Interface Addresses

Table 10-5 describes the interface address information displayed by the show interface command as depicted in Figure 10-3.

Table 10-5. show interface Command Output: Interface Address Information
Parameter	Sample Value	Description
MAC address	0003.4725.2e32	The interface MAC address
MTU	MTU 1500	The maximum transmission unitthe largest packet that can be transmitted or received over the interface
IP address	192.168.254.1	The interface IP address
Subnet mask	255.255.255.0	The interface subnet mask

The MAC address shown is the burned-in address (BIA) that is preprogrammed on the interface. If failover is enabled, the MAC address changes according to the firewall's failover role. The active unit always takes on the MAC address of the primary unit's interface BIA. The standby unit always takes on the MAC address defined for the secondary unit. You can also configure both units to have specific MAC addresses with the failover mac address command.

The IP address shown is the address configured for the interface. If failover is enabled, the active unit takes on the IP address configured for the primary unit's interface. The standby unit takes on the IP address configured for the secondary unit's interface.

The MTU value shown defaults to 1500 bytes for Ethernet interfaces and can be configured with the mtu command. In PIX 7.x, the interface MTU is shown as MTU not set if the interface is shut down or if it has not been configured with a logical name with the nameif command.

Inbound Packet Statistics

Table 10-6 describes the packet statistics information displayed by the show interface command as depicted in Figure 10-3.

Table 10-6. show interface Command Output: Packet Statistics
Counter	Sample Value	Description
Packets input	74613701	The number of packets received on the interface
Bytes input	50030269037331	The number of bytes received on the interface.
No buffer	258406	The number of packets that could not be transferred from the interface to the CPU. No buffer space (packet-sized blocks) was available to queue these packets, so they were dropped.
Received broadcasts	1375986	The number of broadcast packets received.
Runts	0	The number of received packets that were dropped because they were smaller than the minimum Ethernet packet size (64 bytes).
Giants	0	The number of received packets that were dropped because they were larger than the interface's MTU (1500 bytes is the default).
Input errors	46	Usually a catchall count of any of the five errors listed next.
CRC	0	The number of whole packets received whose cyclic redundancy check (CRC) value differed from the expected value. This can indicate that corrupted data is being transmitted from the far end.
Frame	46	The number of incomplete packets received. These packets can also show a CRC error. They are truncated because of packet collisions or a duplex mismatch.
Overrun	0	The number of times the inbound interface controller could not even queue a packet because packets were arriving too fast to be handled. If this count increases, it indicates slowness within the network interface card (NIC), not with the firewall CPU.
Ignored	0	The number of times a packet was ignored by the interface controller because its packet buffer was too full.
Abort	0	The number of packets that were dropped because their processing was aborted for some reason.

Each of these counters is accumulated from the time the firewall was booted or from the time the interface counter was last cleared with the clear interface [if_name] [stats] command.

The last three input errorsoverrun, ignored, and abortare traditional values that have been carried over from the Cisco IOS software on routers. You typically don't see these error counts increase because of the firewall queuing strategy. This is described further in the "Packet Queue Status" section a bit later.

Outbound Packet Statistics

Table 10-7 describes the outbound packet statistical information displayed by the show interface command as depicted in Figure 10-3.

Table 10-7. show interface Command Output: Outbound Packet Statistics
Counter	Sample Value	Description
Packets output	2349219007	The total number of packets transmitted on the interface.
Bytes	7440196846237	The total number of bytes transmitted on the interface.
Underruns	0	The number of times the interface has transmitted a packet faster than the CPU could keep up. In this case, the interface didn't have a complete packet or group of packets to send.
Output errors	0	The number of frames that could not be sent because too many collisions occurred.
Collisions	0	The number of times a collision occurred when a packet was trying to be sent, causing the packets to be retransmitted. Collisions are expected on a half-duplex connection.
Late collisions	0	The number of times a collision occurred but the other stations didn't adhere to the normal collision back-off window. Therefore, these packets were lost because the buffer was already freed. You may see late collisions on a half-duplex connection.
Deferred	27	The number of times a packet could not be sent because the interface was already busy transmitting.

Each of these outbound counters is accumulated from the time the firewall was booted or from the time the interface counter was last cleared with the clear interface [if_name] [stats] command.

You might also see the counters listed in Table 10-8 displayed, depending on the firewall platform and the type of connection.

Table 10-8. show interface Command Output: Additional Interface Statistics
Counter	Description
Interface resets	The number of times the interface was reset or reinitialized. This can happen if the interface is shut down and brought back up or if something prevents the firewall from transmitting on the interface for more than 3 seconds.
Babbles	The number of times a packet could not be sent because another station was transmitting out of turn for longer than the length of a packet. This term normally is not used by Cisco firewalls.
Lost carrier	The number of times the Ethernet carrier signal was lost while a packet was being sent. The carrier signal is essential to bringing the link up and to coordinating transmission on a half-duplex connection.
No carrier	The number of times the Ethernet carrier signal was not detected when a packet needed to be sent. This term normally is not used by Cisco firewalls.

Beginning with PIX 7.x in multiple-context security mode, the show interface command presents a shortened amount of information when it is used from a user context. For example, the following output was produced from the admin context:

Firewall/admin# show interface outside Interface Ethernet0 "outside", is up, line protocol is up MAC address 00a0.c900.0101, MTU 1500 IP address 192.168.93.138, subnet mask 255.255.255.128 Received 7299 packets, 787753 bytes Transmitted 7398 packets, 790589 bytes Dropped 0 packets Firewall/admin#

Notice that user context interfaces are known only by their logical names (outside, for example). As well, only the addressing and data counter information is shown. The interface error counters are reserved for the system execution space, where the actual physical interfaces are configured.

When an interface is configured as a VLAN trunk, you might see some additional information from the show interface command. PIX 7.x, for example, might produce the following information:

Received 214095 VLAN untagged packets, 174551017 bytes Transmitted 103055 VLAN untagged packets, 10106195 bytes Dropped 1456 VLAN untagged packets

Here, VLAN untagged packets represents packets that are sent or received over the native (untagged) VLAN on the 802.1Q trunk.

PIX 6.3 can produce the following output for a trunk interface, including a breakdown of traffic activity on specific VLANs:

279 aggregate VLAN packets input, 110463 bytes 87 aggregate VLAN packets output, 6412 bytes 8 vlan41 packets input, 540 bytes 0 vlan41 packets output, 0 bytes 0 invalid VLAN ID errors, 6 native VLAN errors

Beginning with PIX 7.x, if an interface is not completely configured, you might see some additional information. If the nameif command has not yet been used to assign a logical name to an interface, the following line is shown from the show interface command:

Available but not configured via nameif

Finally, in multiple-context security mode, physical interfaces must be mapped from the system execution space to the appropriate user contexts. If an interface has not been mapped with the allocate-interface context configuration command, the following line is shown:

Available for allocation to a context

Packet Queue Status

A Cisco firewall uses several different buffers or queues as it handles packets in a network. To mon-itor a firewall's performance, it is a good idea to become familiar with the buffering process and statistics.

Figure 10-4 illustrates how packets arriving on an interface are queued for inspection and how other inspected packets are queued for transmission on the interface.

Figure 10-4. Firewall Interface Packet Queues

Each firewall interface has its own inbound and outbound queues, arranged as a hardware queue and a software queue in each direction. Beginning with PIX 7.x, the outbound software queue is known as the best-effort queue (BEQ). Each interface also has an outbound priority queue, the low-latency queue (LLQ), that can be used to forward high-priority traffic. The LLQ is always serviced before the BEQ.

Basically, incoming packets arrive from the physical interface and go into a hardware interface queue (if one is present) on the firewall platform. If that queue overflows before packets can be emptied for inspection, new packets are pushed into the input software queue.

In the outbound direction, the process is similar but reversed. As packets are inspected and approved for forwarding, they are moved into an output queue. Before PIX 7.x, packets were copied right into the output hardware queue if there was room. If not, they went into the output software queue. All packet delivery was done on a best-effort basis, with no quality of service possible.

Beginning with PIX 7.x, outbound packets can be moved into an output BEQ or an output LLQ, depending on the results of a service policy. If priority queuing is enabled, packets are always pulled from the LLQ before the BEQ is serviced. If priority queuing is not enabled, all outbound packets go into the BEQ.

Hardware and software queue statistics are reported through the show interface command, as in the following example:

Firewall# show interface gigabitethernet0 Interface GigabitEthernet0 "outside", is up, line protocol is up Hardware is i82542 rev03, BW 1000 Mbps (Full-duplex), Auto-Speed(1000 Mbps) Description: Outside Public Network MAC address 0003.4708.ec54, MTU 1500 IP address 10.1.1.1, subnet mask 255.255.255.0 478686 packets input, 591360505 bytes, 0 no buffer Received 828 broadcasts, 0 runts, 0 giants 0 input errors, 0 CRC, 0 frame, 0 overrun, 0 ignored, 0 abort 332357 packets output, 23589813 bytes, 0 underruns 0 output errors, 0 collisions, 0 interface resets 0 babbles, 0 late collisions, 0 deferred 0 lost carrier, 0 no carrier input queue (curr/max blocks): hardware (0/25) software (0/0) output queue (curr/max blocks): hardware (3/122) software (0/0)

Each firewall interface has both input and output queues; the current state of the queues is displayed as a ratio of current/maximum blocks used. Table 10-9 lists the values reported in the preceding command output and describes each value.

Table 10-9. show interface Command Output: Packet Queue Status Information
Parameter	Sample Value	Description
Input queue hardware	0/25	No packets are currently in the hardware queue. 25 packets are the most that have ever been in the queue.
Input queue software	0/0	No packets are currently in the software queue. The queue has always been empty.
Output queue hardware	3/122	Three packets are currently in the hardware queue. It has held up to 122 packets in the past.
Output queue software	0/0	No packets are currently in the software queue. The queue has always been empty.

It is common to see the hardware queue (either input or output) reported to have activity. You should see a nonzero number for the maximum queue level when the hardware queue has been used. However, you should see nonzero values for the software queue only when the hardware queue has been full sometime in the past. This isn't necessarily a bad thing.

If you see large values reported for a software queue, and the current number consistently stays close to the maximum number, your firewall CPU is having trouble keeping up with the interface load.

TIP

Fast Ethernet firewall interfaces (10/100) always report an inbound hardware queue statistic of 128/128. As well, you always see some inbound software queue activity. This indicates that this type of interface doesn't use a hardware queue. Instead, all inbound packets are copied into the software queue directly.

This is not true for the Fast Ethernet outbound queues, which use both hardware and software queues.

Outbound priority queues are available only beginning with PIX software release 7.0. The show interface command doesn't report on the outbound priority queue. Instead, you can use the following command to view output queue statistics:

Firewall# show priority-queue statistics [if_name]

This command displays the current statistics about both the priority queue (LLQ) and the best-effort queue (BEQ) of a firewall interface. These are shown as the queue type in the command output. If no interface name is given, all interfaces are shown.

For example, the following statistics resulted from a firewall's outside interface:

Firewall# show priority-queue statistics outside Priority-Queue Statistics interface outside Queue Type = BE Packets Dropped = 0 Packets Transmit = 132213 Packets Enqueued = 0 Current Q Length = 0 Max Q Length = 0 Queue Type = LLQ Packets Dropped = 0 Packets Transmit = 1826 Packets Enqueued = 5 Current Q Length = 0 Max Q Length = 32 Firewall#

Table 10-10 lists the fields displayed in the command output. The descriptions pertain to the priority queue.

Table 10-10. show priority-queue statistics Command Output: Packet Queue Status Information
Parameter	Sample Value	Description
Queue Type	LLQ	The low-latency queue (LLQ), also called the priority queue
Packets Dropped	0	Packets that were dropped because the priority queue was full
Packets Transmit	1826	Packets that were candidates for the priority queue and were successfully transmitted (cumulative total)
Packets Enqueued	5	The running total of packets that have been placed in the priority queue
Current Q Length	0	The current priority queue depth, or the number of packets currently in the priority queue
Max Q Length	32	The largest number of packets stored in the priority queue since the last reboot or clear priority-queue statistics command

Sometimes you might see a larger number in the Packets Transmit counter than the Packets Enqueued counter. That might seem odd, because outbound priority packets should be put into the LLQ before being transmitted. The difference is that some firewall platforms can write priority packets into the output hardware queue directly, so they don't actually pass through the LLQ first.

10-1. Checking Firewall Vital Signs

Using the Syslog Information

Checking System Resources

Firewall CPU Load

Table 10-1. Computing Actual Firewall Process Runtimes

Firewall Memory

Table 10-2. Firewall Memory Blocks and Their Uses

Checking Stateful Inspection Resources

Xlate Table Size

Conn Table Size

Checking Firewall Throughput

PDM

Figure 10-1. A Sample PDM Display of Firewall Throughput

Syslog

Figure 10-2. An Example of Firewall Throughput Reporting by Syslog Analysis

Traffic Counters

Perfmon Counters

Checking Inspection Engine and Service Policy Activity

Checking Failover Operation

Verifying Failover Roles

Verifying Failover Communication

Determining if a Failover Has Occurred

Determining the Cause of a Failover

An Example of Finding the Cause of a Failover

Intervening in a Failover Election

Checking Firewall Interfaces

Figure 10-3. A Breakdown of Information Presented by the show interface Command

Interface Name and Status

Table 10-3. show interface Command Output: Interface Name and Status Information

Interface Control

Table 10-4. show interface Command Output: Interface Control Information

Interface Addresses

Table 10-5. show interface Command Output: Interface Address Information

Inbound Packet Statistics

Table 10-6. show interface Command Output: Packet Statistics

Outbound Packet Statistics

Table 10-7. show interface Command Output: Outbound Packet Statistics

Table 10-8. show interface Command Output: Additional Interface Statistics

Packet Queue Status

Figure 10-4. Firewall Interface Packet Queues

Table 10-9. show interface Command Output: Packet Queue Status Information

Table 10-10. show priority-queue statistics Command Output: Packet Queue Status Information

Категории