Cisco Network Security Troubleshooting Handbook

The problem areas of AAA on PIX firewall and FWSM can be categorized as follows based on their implementations:

• Firewall management with AAA troubleshooting

• Cut-Through Proxy with AAA troubleshooting

• Extended authentication (X-Auth) for IPsec VPN with AAA troubleshooting

The sections that follow discuss these problem areas.

Firewall Management with AAA Troubleshooting

You can connect to the PIX firewall in several ways to configure or monitor the PIX firewall. You can get CLI access to the PIX firewall for configuration and monitoring purpose in the following ways:

• Using Console Console port is ready to be used on the PIX firewall as soon as you power on the PIX firewall. You do not need any configuration on the PIX firewall.

• Using Telnet If you want to remotely manage the PIX firewall using Telnet, you can do so on all interfaces except the outside interface. For Telnet access to the outside interface, you must configure IPsec to include IP traffic that is generated by the PIX firewall, and enable Telnet on the outside interface.

• Using SSH SSH can be configured instead of Telnet for secured access to the PIX firewall. We recommend that you manage PIX firewall with SSH instead of Telnet. Additionally, managing PIX Firewall from an outside network with Telnet is not possible, as Telnet is not allowed to the outside interface. Although it is possible to get around the problem by terminating a VPN tunnel on the outside interface, SSH is the preferred method to manage the PIX firewall from the outside.

• Using Telnet over IPsec As mentioned before, Telnet is not allowed to the outside interface of the PIX firewall because of security reasons. So if you really need to Telnet to the outside interface, the only way to accomplish that is by creating an IPsec tunnel to the outside interface and then allowing the Telnet to the outside interface.

• PDM/ASDM/Firewall MC/Security Monitor Connection between the Management Software (PDM/ASDM/Firewall MC) and the Monitoring Software (Security Monitor) takes place using HTTPS (http/SSL). You need to configure a web server and generate a certificate to allow the HTTPS connection to the firewall.

Once the connection parameters are configured on the firewall, you can add security to the connection by configuring AAA, which provides per-user control for the connections. This section presents the configuration and troubleshooting setup as follows:

• Login authentication issues

• Enable authentication issues

• Authorization issues

• Accounting issues

Login Authentication Issues

This section delves into the details of both login authentication and troubleshooting. Before you attempt to enable authentication, you should configure the authentication and be sure that it works with the basic login credentials without configuring AAA. This section explains the following topics in detail:

• Configuring basic authentication with different communication methods on the Firewall

• Configuring local user database authentication

• Configuring external AAA server authentication

• Configuring a fallback method for authentication

• Troubleshooting login authentication

Configuring Basic Authentication with Different Communication Methods on the Firewall

Before you configure local user database or AAA for login authentication on the PIX firewall, be sure that you can connect to the PIX firewall with the Basic Login credentials. For login and enable authentication, you can use passwd to configure login password, and enable password to configure enable authentication. The following list of configuration methods on the PIX Firewall details the configuration for basic authentication for these different connection methods:

• Basic Authentication for Telnet on the PIX firewall To turn on Telnet on the PIX firewall, you must use the telnet command. Example 10-3 shows how to turn on telnet on the PIX firewall for 192.168.1.0 network to the inside interface of the PIX firewall.

  Example 10-3. Shows How to Configure Telnet Access to the PIX Firewall

  PIX# configure terminal ! Set login password with the following command. If you want to set encrypted ! password you must provide 16 bytes string. PIX(config)# passwd cisco12345678912 encrypted ! The following command will set up the enable password on the PIX firewall. PIX(config)# enable password cisco76687463274 encrypted ! The following command allows 192.168.1.0 network to be able to telnet to the ! inside interface on the PIX firewall PIX(config)# telnet 192.168.1.0 255.255.255.0 inside ! Following command is used to verify that the telnet is enabled. PIX(config)# show running-config telnet 192.168.1.0 255.255.255.0 inside PIX(config)#

• Basic Authentication for SSH connection on the PIX firewall SSH requires that an RSA key be created on the PIX firewall. To create an RSA key, you must have Data Encryption Standard (DES) or 3DES features turned on (this can be verified with the show version command). If you do not have the DES/3DES feature enabled, you need to send an e-mail to licensing@cisco.com with your show version output and request for a DES/3DES activation key. Then you can insert the key into the PIX with the activation-key command. Use crypto key generate rsa general-keys to generate the RSA key on the PIX Firewall, and execute the SSH command to allow the addresses that can SSH into the PIX Firewall. Example 10-4 shows a sample configuration for turning on SSH on the PIX firewall.

  Example 10-4. Sample Configuration SSH on the PIX Firewall

  PIX# configure terminal ! Set login password with the following command. If you want to set encrypted ! password you must provide 16 bytes string. PIX(config)# passwd cisco12345678912 encrypted ! The following command will set up the enable password on the PIX firewall. PIX(config)# enable password cisco76687463274 encrypted ! Following two lines are needed to generate the RSA key PIX(config)# hostname PIX PIX(config)# domain-name cisco.com ! The following command will remove the existing key from the PIX. PIX(config)# crypto key zeroize rsa ! Generate the key with the following command PIX(config)# crypto key generate rsa general-keys ! Save the RSA key with the following command PIX(config)# write memory ! Turn on and allow the SSH connection from 192.168.1.0/24 network with the ! following commands PIX(config)# ssh 192.168.1.0 255.255.255.0 inside PIX(config)# ssh timeout 60 PIX(config)# exit ! Save the configuration with the following command PIX# write memory PIX#

  After successful configuration of SSH on the PIX firewall, you may use an SSH client (for example, putty.exe, which can be downloaded freely from the Internet by searching with the string "putty.exe") to connect to the PIX firewall. If the firewall gives you the option of accepting a certificate, then the PIX firewall is configured correctly for SSH. When asking for a username, enter pix as username and for the password enter the password configured on the PIX firewall. For enable access, just enter the enable password configured on the PIX.

• Basic Authentication for VPN connection to allow Telnet to the outside interface of the PIX firewall with Basic Authentication You need to configure Remote Access VPN using IPsec protocol to allow the Telnet access from outside to the outside interface of the PIX firewall. Refer to Chapter 7, "Troubleshooting IPsec VPN on PIX Firewalls" for details on Remote Access VPN configuration using IPsec. Then, following the procedure explained earlier under "Basic Authentication for Telnet on the PIX Firewall," allow the Telnet access to the IP address assigned by the PIX to the VPN client. As the IP address is assigned from the pool, we recommend that you add the pool with the Telnet command to allow access. Here is an example to illustrate the point. Assume that you have an IP Pool defined for the VPN traffic with network address 192.168.1.0, so that the VPN client will get one of the IP addresses from this pool (It could be any IP address from the pool). Therefore, you need to include this network with the telnet command as follows:

  PIX(config)# telnet 192.16.1.0 255.255.255.0 outside

• Basic Authentication for PDM/ASDM/Firewall MC/Security Monitor Access (HTTPS) For PDM/ASDM/Firewall MC/Security Monitor access, you need to configure a web server on the firewall and generate a certificate for the HTTPS connection. Example 10-5 shows how to configure HTTPS on the firewall.

  Example 10-5. Configuring HTTPS on the PIX Firewall

  PIX# configure terminal ! Turn on the web server with the following command. PIX(config)# http server enable ! Allow the network that can connect to the web server using ASDM/Firewall MC or ! Security Monitor PIX(config)# http 192.168.1.0 255.255.255.0 inside PIX(config)# exit ! Save the configuration with the following command PIX# write memory PIX#

Regardless of the type of method you choose to connect to the PIX firewall for management purposes, it is strongly recommended that you control access to the PIX with AAA as covered in the section that follows.

Configuring Local User Database Authentication

If you want to configure per user authentication for login, you have the option to configure this per-user authentication with the local user database on the PIX firewall.

Work through the following steps for configuring local user database authentication:

Configuring External AAA Server Authentication

As mentioned before, the local user database is not very scalable when there are a large number of users to manage. Therefore, you must use an external AAA server for the central point of user management. Configuring an external AAA server-based authentication for login requires configuring both an AAA server and the PIX firewall. Work through to following steps to accomplish this task:

Caution

While adding authentication statements, always have a connection open to the PIX in the event that backing out the commands is necessary. Additionally, do not save the configuration unless you thoroughly test your AAA configuration so that if you are locked out, you need only perform a reboot to return your original configuration.

Configuring Fallback Method for Authentication

Beginning with PIX version 6.3.4, you can configure a local user database as a fallback method in case the AAA server is unavailable. Fallback method is available to use only with the local user database.

To configure fallback, you need to combine the procedure for authentication using both the AAA server and the local user database. For example, to turn on fallback for SSH connection to the firewall, use the following command:

PIX(config)# aaa authentication ssh console AuthTacacs LOCAL

With the preceding configuration, if the TACACS+ server is unavailable, the local user database will be used for SSH authentication. Note that if multiple TACACS+ servers are defined with the AuthTacacs server tag, the firewall will try all TACACS+ servers in the list before it uses the local user database.

If a fallback method is used, you need to configure the AAA server that is used as a primary method for authentication with proper parameters on the firewall. For example, if the fallback method is used, you might not want to increase the maximum failed attempts to a high number, because that could delay the authentication if the AAA server is down. Following is a list of options available for configuring different parameters for the AAA server:

• Max-failed-attempts This is the maximum number of AAA requests attempted to each server within an AAA server group. The parameter must be within [1-5] inclusive. The default value is 3.

  PIX(config)# [no] aaa-server AAA-Server-Tag max-failed-attempts number

• Deadtime Deadtime is the number of timeout minutes for declaring an AAA server group as unresponsive. The parameter must be within [0-1440] inclusive. The default value is 10.

  PIX(config)# [no] aaa-server AAA-Server-Tag deadtime minutes

Note

The AAA fallback is available only for administrative session traffic authentication and command authorization, not for cut-through proxy or x-auth traffic.

Troubleshooting Login Authentication

Before delving into the details of troubleshooting authentication problem for SSH, Telnet and other types of connections, it is worth of going through a debug output for a successful user authentication, which is shown in Example 10-7.

Example 10-7. Successful User Authentication for SSH Session with Output from the debug aaa authentication and debug tacacs Commands

Work through the following steps to troubleshoot authentications problem with login:

Enable Authentication

This section explains how to configure and troubleshoot Enable authentication with various methods.

Configuring Enable Authentication

Enable password authentication can be performed one of the following ways:

Troubleshooting Enable Authentication

Troubleshooting steps for enable password authentication are almost same as those discussed for Login Authentication. Example 10-10 shows a successful enable password authentication with TACACS+ protocol:

Example 10-10. Successful User Authentication with TACACS+ Protocol

Additional information that is important and specific to enable password authentication is as follows:

• Under Advanced TACACS+ Settings of User Profile, or under Enable Option of Group profile, be sure "No enable Privilege" is not checked. You must assign privilege level 15. Otherwise, enable authentication will fail.

• If login is done externally with an AAA server but enable authentication is performed locally, be sure the user name matches between the AAA server and the local user database.

• If a separate password is used for enable authentication, be sure you are entering the password correctly. If the external database is configured on the Cisco Secure ACS server for enable password, change it to use the same password as login to see if the external server is causing the problem. If you find it is an external database issue, investigate the issue on the AAA server (refer to Chapter 13, "Troubleshooting Cisco Secure ACS on Windows" for details).

• If enable password is configured locally with different privileges, be sure that you are authenticating enable password with that specific privilege level. For example, if the enable password is at level 10, configure, then be sure to go into enable mode by typing enable 10 not just enable.

Command Authorization

With the blessings of privilege levels, you can control the commands users can execute on the firewall after the enable authentication with the local user database. If TACACS+ is used, more granular command authorization is possible with the TACACS+ server. This section explains how to control command authorization for different methods of authentication for enable access:

• Command Authorization based on Enable Password Privilege level

• Command Authorization using local user database

• Command Authorization using an external AAA server

Command Authorization Based on Enable Password Privilege Level

As discussed before, you can configure multiple enable passwords with different privilege levels locally. Every command on the PIX firewall has a privilege level assigned. If you authenticate enable access with a password that has the same or higher privilege level than the level of the command, that command will be allowed. For example, if your enable password is defined as cisco3, which has level 3, you can execute every command that is at level 0. You must have the authorization turned on to use a local user database with the following command to control the command execution; otherwise, you will be able to execute all commands regardless of the privilege level assigned to you.

Command Authorization Using Local User Database

Command authorization done locally on PIX Firewall is based on privilege level. There are 16 privilege levels in PIX Firewall. Two of them are set to the defaults 0 and 15. Anything from 1 to 14 has the same sets of commands that the privilege level 0 has. Every command has a predefined privilege level and most of them are at level 15. However, it is possible to change the command privilege level from higher to lower. For instance, show running-config is a level 15 command, and it is possible to bring this down to level 10. So, a user with level 10 or above can execute this command. Users with levels lower than 10 will not be able to execute a level 10 command. Example 10-11 walks you through the privilege level concept and different commands available with local database command authorization. The example also shows how to turn on command authorization with the following command:

PIX(config)# aaa authorization command LOCAL

Example 10-11. Privilege Level Command Execution and How to Turn on Command Authorization Based on the Privilege Level

Command Authorization Using an External AAA Server

Command Authorization with the local user database is not scalable. Hence, with a larger user database, you should consider implementing command authorization using an external AAA server. Command Authorization is possible with TACACS+ protocol but not with the RADIUS protocol.

Command authorization by TACACS+ is based on submitting the entire command expression to a TACACS+ server for authorization before execution. While this section deals only with command authorization for a TACACS+ server, a local user database as discussed in the preceding section will be used as a fallback method. Note that fallback using a local user database is possible only with PIX Version 6.3.4 and above.

Configuring command authorization with the external TACACS+ server is a two-step process: configuring the TACACS+ server, and configuring PIX Firewall.

Work through the steps that follow to configure an ACS in Windows for command authorization:

Once you are finished with the Cisco Secure ACS server configuration, you need to configure PIX Firewall for command authorization. Example 10-12 shows the configuration needed on the PIX firewall to turn on command authorization.

Example 10-12. AAA Configuration Needed for the Authorization with TACACS+ Server and Local User Database as a Fallback Method

Troubleshooting Steps

To troubleshoot command authorization, run the following commands:

PIX# debug aaa authorization PIX# debug tacacs

Before you look at some of the possible causes of failure, go through Example 10-13 for a successful command authorization using tacacs+ protocol:

Example 10-13. Command Authorization in Action for Administrative Session

Command authorization might not work due to one or more of the following reasons:

• If the authorization does not work at all, be sure that authorization is turned on with the following command:

  aaa authorization command TACSERVER LOCAL

• If authorization is performed with the local user database, be sure that the command is at the same or at a lower privilege level than the user's privilege. Otherwise, you need to log in as admin user with privilege level 15 to bring the privilege level down to the user level.

• If authorization is turned on with the TACACS+ protocol, run debug aaa authorization and debug tacacs to find out how the command is sent out to the AAA server. Revise the AAA server group profile configuration to ensure that the command and arguments are match the permit as you see it in the debug on the PIX firewall.

• If expected permit commands are denied, and vice versa, revise the logic of the commands, and correct if needed.

Accounting

Beginning with PIX Version 7.0, Accounting for firewall administrative sessions is possible. Previously it was only available with the syslog. To enable accounting for a special connection type, use the following command:

[View full width]

You can enable accounting for capturing and sending the command executed on the firewall with the following command:

Firewall(config)# [no] aaa accounting command [privilege level] AAA-Server-Tag

Refer to Table 10-8 for an explanation of the arguments of the accounting command.

If you are having problem with accounting, run the following debug commands on the PIX firewall:

PIX# debug aaa accounting PIX# debug radius

or,

PIX# debug tacacs

Information can be shown either on the console or syslog. If Cisco Secure ACS is used as an external AAA server, you can view accounting records by clicking the Reports and Activity button. Then click TACACS+ Accounting or RADIUS Accounting. All accounting reports are in comma-separated value (.csv) format and can be displayed in a web browser.

Cut-Through Proxy Authentication

You can configure the firewall to force the users to authenticate before connections are permitted. After successful user authentication, the user's authentication status is cached to permit subsequent connections from the same IP address from where the initial connection authenticated user initiated. The firewall functions as an authentication proxy for the subsequent connections. Subsequent connections simply "cut through" the firewall very efficiently. That's why this feature is called Cut-Through Proxy.

While firewall can intercept all types of IP traffic, it can authenticate only the FTP, Telnet, SSH, HTTP, and HTTPS protocols traffic because of the authentication scheme available for these protocols.

Note that Figure 10-2 will be used for the remainder of the chapter as the AAA for Management and Cut-Through Proxy.

If the cut-through proxy authentication is enabled on the firewall, when the user connections go through the firewall, they are intercepted by AAA, and the firewall presents a username/password prompt. How a user provides a username and password differs as follows:

• For Telnet/SSH The user sees a username prompt, and then a request for a password. If authentication (and authorization) is successful at the firewall, the user is prompted for username and password by the destination host beyond. With Telnet/SSH, the username/password needs to be entered twiceonce for the firewall, and once for the end host.

• For FTP The user sees a username prompt first and then the password. The user needs to enter local_username@remote_username for username and local_password@remote_password for password. The firewall sends the local_username and local_password to the AAA server and authenticates the user. If authentication (and authorization) is successful at the firewall, the remote_username and remote_password are passed to the destination FTP server beyond. With FTP, you have the option to enter the username/password once.

• For HTTP/HTTPS The user sees a window displayed in the browser requesting username and password. If authentication (and authorization) is successful, the user arrives at the destination website beyond.

Authentication for Cut-Through Proxy

Cut-Through Proxy authentication can be performed by either of the following methods:

• Cut-Through Proxy Authentication with Local User Database

• Cut-Through Proxy Authentication Using TACACS+/RADIUS Protocol

The sections that follow go through the configuration of these two methods, followed by the troubleshooting steps for Cut-Through Proxy Authentication.

Cut-Through Proxy Authentication with Local User Database

To configure cut-through proxy with the local user database, you need to configure the local user database with the following command on the firewall:

[View full width]

To remove a user, use the following command:

PIX(config)# no username [name]

You can turn on the authentication for the connections going across the firewall with one of the following ways:

Example 10-14 shows how to turn on authentication for both inbound and outbound traffic using the local user database.

Example 10-14. Turning on Cut-Through Proxy Authentication With Include/Exclude Option Using Local User Database

A local user database is not scalable for a large network. An external AAA server must be configured to address the scalability issue, which we discuss next.

Cut-Through Proxy Authentication Using RADIUS or TACACS+ Protocol

The configuration on the firewall to turn on cut-through proxy authentication is very similar to that of a local user database, which is discussed in the preceding section. The only exception is the server-groupinstead of using LOCAL, a server group for either RADIUS or TACACS+ protocol must be defined.

Before you attempt any configuration on the firewall for cut-through proxy, we recommend strongly that you configure the AAA server first.

Work through the following steps to configure the Cisco Secure ACS Server so that Firewall can use it for authentication purposes:

Once AAA server configuration is complete, work through the following steps to configure firewall:

Troubleshooting Cut-Through Proxy Authentication

You must ensure that the connection across the firewall works before turning on cut-through proxy authentication. Once cut-through proxy authentication is turned on and if authentication fails, you need to enable logging with debug level turned on (see the "Syslog" subsection under the section entitled "Diagnostic Commands and Tools"). You need to run debug aaa authentication, and debug radius or debug tacacs to find out the details on authentication failure for the cut-through proxy authentication. If the syslog does not provide the cause of failure, you need to analyze the log on the AAA server. For details on how to analyze the log on the CS ACS server, refer to Chapter 13, "Troubleshooting Cisco Secure ACS on Windows."

Before delving into the details of the different issues with cut-through proxy authentication, this section first examines the look of a successful cut-through proxy authentication in the syslog. Example 10-18 shows a successful cut-through proxy authentication for inbound traffic authentication using TACACS+ protocol initiating the packet from 50.1.1.1 to inside 192.168.1.2 (20.1.1.102) and vice versa (see Figure 10-2).

Example 10-18. Successful Authentication for TACACS+ (Inbound)

For additional details of a successful cut-through proxy authentication, refer to Example 10-19.

Example 10-19. Successful Cut-Through Proxy Authentication with Debug AAA Authentication And Debug Tacacs

Following is a list of some possible causes of cut-through proxy authentication problems:

• Incorrect username or bad password User authentication might fail due to an incorrect username or bad password. In either case, the user will see the following message after getting three opportunities to enter the username and password.

  Error: Max number of tries exceeded.

  Example 10-20 shows syslog for authentication failure with TACACS+ protocol for inbound traffic.

  Example 10-20. Syslog Message for Bad Authentication (Username or Password) TACACS+ (Inbound)

  PIX(config)# logging monitor debugging PIX-A# Jul 25 2005 07:48:16 : %PIX-6-302013: Built inbound TCP connection 5981 for outside:50.1.1.1/11006 (50.1.1.1/11006) to inside:192.168.1.2/23 (192.168.1.2/23) Jul 25 2005 07:48:16 : %PIX-6-109001: Auth start for user '???' from 50.1.1.1/11006 to 192.168.1.2/23 Jul 25 2005 07:48:20 : %PIX-6-609001: Built local-host outside:171.69.89.217 Jul 25 2005 07:48:20 : %PIX-6-302013: Built outbound TCP connection 5983 for outside:171.69.89.217/49 (171.69.89.217/49) to NP Identity Ifc:172.16.172.164/ 1433 (172.16.172.164/1433) Jul 25 2005 07:48:24 : %PIX-6-302014: Teardown TCP connection 5983 for outside:171.69.89.217/49 to NP Identity Ifc:172.16.172.164/1433 duration 0:00:03 bytes 108 TCP FINs Jul 25 2005 07:48:30 : %PIX-6-609001: Built local-host outside:171.69.89.217 Jul 25 2005 07:48:30 : %PIX-6-302013: Built outbound TCP connection 5984 for outside:171.69.89.217/49 (171.69.89.217/49) to NP Identity Ifc:172.16.172.164/ 1434 (172.16.172.164/1434) Jul 25 2005 07:48:32 : %PIX-6-302014: Teardown TCP connection 5984 for outside:171.69.89.217/49 to NP Identity Ifc:172.16.172.164/1434 duration 0:00:01 bytes 107 TCP FINs Jul 25 2005 07:48:39 : %PIX-6-609001: Built local-host outside:171.69.89.217 Jul 25 2005 07:48:39 : %PIX-6-302013: Built outbound TCP connection 5985 for outside:171.69.89.217/49 (171.69.89.217/49) to NP Identity Ifc:172.16.172.164/ 1435 (172.16.172.164/1435) ! Following failed message indicates a failure with either username/password. Jul 25 2005 07:48:41 : %PIX-6-109006: Authentication failed for user 'kodom' from 50.1.1.1/11006 to 192.168.1.2/23 on interface outside Jul 25 2005 07:48:41 : %PIX-6-302014: Teardown TCP connection 5985 for outside:171.69.89.217/49 to NP Identity Ifc:172.16.172.164/1435 duration 0:00:01 bytes 109 TCP FINs Jul 25 2005 07:48:42 : %PIX-6-302014: Teardown TCP connection 5981 for outside:50.1.1.1/ 11006 to inside:192.168.1.2/23 duration 0:00:25 bytes 374 TCP FINs PIX#

• Communication Problem between PIX and AAA Server Several conditions might be considered communication problems between the PIX and AAA Server:

  - PIX may not have proper routing. Execute ping from the PIX to verify.

  - PIX has connectivity, but TACACS+ or RADIUS protocols might be blocked by a firewall between PIX and the AAA server.

  - The shared secret might be a mismatch between the PIX and AAA Server.

  - PIX might not be defined as the AAA client on the AAA Server.

  Under any of these circumstances, PIX might be unable to send the authentication information to the AAA server, and this will result in authentication failure. The user sees a username, then a password, then RADIUS server failed, and then finally Error: Max number of tries exceeded. See Example 10-21 to see the syslog message when TACACS+ is used for inbound authentication, and the authentication fails.

  Example 10-21. Syslog Message When PIX Has Communication Problem with AAA Server

  PIX(config)# logging monitor debug PIX-A# Jul 25 2005 07:55:12 : %PIX-6-302013: Built inbound TCP connection 5987 for outside:50.1.1.1/11007 (50.1.1.1/11007) to inside:192.168.1.2/23 (192.168.1.2/23) Jul 25 2005 07:55:12 : %PIX-6-109001: Auth start for user '???' from 50.1.1.1/11007 to 192.168.1.2/23 Jul 25 2005 07:55:16 : %PIX-6-609001: Built local-host outside:171.69.89.217 Jul 25 2005 07:55:16 : %PIX-6-302013: Built outbound TCP connection 5989 for outside:171.69.89.217/49 (171.69.89.217/49) to NP Identity Ifc:172.16.172.164/ 1436 (172.16.172.164/1436) Jul 25 2005 07:55:16 : %PIX-6-302014: Teardown TCP connection 5989 for outside:171.69.89.217/49 to NP Identity Ifc:172.16.172.164/1436 duration 0:00:00 bytes 0 TCP Reset-O ! Following message indicates server is unavailable Jul 25 2005 07:55:16 : %PIX-6-109002: Auth from 50.1.1.1/11007 to 192.168.1.2/23 failed (server 171.69.89.217 failed) on interface outside Jul 25 2005 07:55:16 : %PIX-6-609001: Built local-host outside:171.69.89.217 Jul 25 2005 07:55:16 : %PIX-6-302013: Built outbound TCP connection 5990 for outside:171.69.89.217/49 (171.69.89.217/49) to NP Identity Ifc:172.16.172.164/ 1437 (172.16.172.164/1437) PIX#

• User's Account Locked Out If you configure a lockout feature on the AAA server after a certain number of failed authentication attempts, the user account will be locked out. Hence, subsequent authentication requests will fail. Re-enable the user account so that it can authenticate.

• User reached the MAX session limit If you have a session limit configured on the AAA server, be sure that you do not exit more often than the number of times that is configured for simultaneous authentication for a specific user or a group.

• User Reaches the Proxy Limit If you have too many simultaneous connection requests, you may run short of the proxy limit, which is by default 16. So, if a user makes 17 connection requests at the same time across the firewall, one connection will not be authenticated and the connection will be dropped. You can disable proxies by using the disable parameter. To return to the default proxy-limit value (16), use the no form of this command.

  aaa proxy-limit proxy_limit aaa proxy-limit disable no aaa proxy-limit

Authorization for Cut-Through Proxy

To control the resources users may access once they are authenticated, you need to implement authorization. On the other hand, if you want to allow authenticated users to perform all operations, then authentication is sufficient. Authorization can be performed either with the TACACS+ or RADIUS protocol. Even though a PIX firewall supports a local user database for cut-through proxy authentication, it does not support authorization. Authorization for a cut-through proxy is implemented differently using TACACS+ and RADIUS protocols due to the different nature of the protocols.

Authentication and authorization are performed separately in TACACS+, whereas in RADIUS both authentication and authorization are performed together. Therefore, command authorization is possible with TACACS+, but not with RADIUS.

When an end user who is going through the firewall tries to Telnet/FTP/HTTP/HTTPS to an IP address of a server, the packets are intercepted and authenticated, and then the Telnet/FTP/HTTP/HTTPS are sent to the AAA server as commands, and the IP address after that goes as arguments to the TACACS+ server for the command authorization. For example, if the user enters http 192.168.1.105 on the browser, after successful user authentication, if the authorization is turned on for HTTP traffic for the specific destination, the firewall will send http as a command and 192.168.1.105 as an argument to the TACACS+ server for command authorization.

Authorization can be performed for any type of pass-through IP traffic across the firewall. For example, if you want to authorize the SMTP mail traffic, you can perform authentication from the main server using Telnet/FTP/HTTP/HTTPS, which will cause the firewall to cache the IP address of the mail server. Then, if the authorization is turned on for the mail server using tcp/25 as the service, mail traffic will be authorized. On the TACACS+ server, you need to ensure that tcp/25 is configured as the command and the destination IP address (not the mail server IP address) is configured as argument.

If you want to have more granular control with authorization, configure an access-list on the firewall and then download the name of the access-list from the TACACS+ server. Support for ACL for command authorization is introduced in PIX Firewall Version 5.2.

Authorization with RADIUS protocol for the pass-through traffic is only possible via ACL, which will be discussed in the coming section.

Note

To implement authorization for HTTP in an enterprise network, use URL filtering software such as Websense, N2H2, and so on. AAA Authorization implementation for HTTP authorization is limited based on the IP address not on the DNS, or other important criteria that these URL filtering servers offer.

Configuration steps for the cut-through proxy authorization are discussed in next two sections.

Configuring Cut-through Proxy Authorization using the TACACS+ Protocol

The cut-through proxy authorization configuration discussed in this section is based on Figure 10-2.

Configuring cut-through proxy authorization using the TACACS+ protocol is a two-step process: configuring user and group profiles with command authorization on the TACACS+ server, and enabling authorization on the firewall. For this configuration, Cisco Secure ACS on Windows is used. It supports the TACACS+ protocol.

Work through the steps that follow to configure the Group Profile on the ACS on Windows for Authorization. To illustrate this better, assume that you want the user to be able to use Telnet to access 20.1.1.100, use FTP to access 20.1.1.101, and use HTTP to access any website.

When you are finished with the CS ACS configuration, the next step is to enable authorization on the firewall as shown in Example 10-22.

Example 10-22. Turning on Authorization for Cut-Through Proxy

Caution

In Example 10-22, both ways to enable authorization for cut-through proxy are shown. Use either method, but not both. Otherwise, undesirable results will occur.

Troubleshooting Cut-Through Proxy Authorization using the TACACS+ Protocol

Most authorization problems for the cut-through proxy using the TACACS+ protocol arise because of a lack of understanding and misconfiguration. Careful review of the configuration of both the firewall and the AAA server can eliminate many problems. However, logs and debug commands available on the firewall, and the log on the Cisco Secure ACS software, can help you to nail down any problem. It's important to learn how to use the log and the debug/show command efficiently. It is equally important to be extremely familiar with the AAA server log. Refer to Chapter 13, "Troubleshooting Cisco Secure ACS on Windows" for details on how to use the logging capability on Cisco Secure on Windows software.

By now, you should be familiar with how to turn on the syslog in the debug level. Therefore, go through the debug level log for a successful user authorization session and become comfortable with it, so that you can easily identify a deviation when authorization does not work under a specific condition. Example 10-23 shows syslog when user authentication and authorization is successful.

Example 10-23. Syslog for a Successful Authentication and Authorization Using the TACACS+ Protocol

Now that you are familiar with a successful user authentication and authorization, go through Example 10-24, which shows an example of user authorization failure with TACACS+ protocol, but of successful authentication.

Example 10-24. Syslog for Successful Authentication with Authorization Failure Using the TACACS+ Protocol

The log in Example 10-24 shows that authorization failed for the specific IP and port on the outside interface of the firewall. To understand why, analyze the log on the AAA server. For your setup, you are using Cisco Secure ACS as an AAA server. So, go under "Reports and Activity" and then "Failed Attempt," and find out how the commands are parsing to the AAA server from the firewall. Get the syntax from there and apply the same command and argument on the Group setup.

Following are some important points to check to ensure that authorization is configured correctly:

• Verify that Shell/Exec is turned on under the TACACS+ Settings of the Group profile to which the user belongs.

• Check that the ACL tag defined on the TACACS+ server matches the one defined on the PIX firewall.

• Ensure that traffic is allowed on the ACL that is defined on the PIX firewall, and that the tag is downloaded to the PIX firewall.

• Define the complete syntax of the commands when defining command authorization. If authorization fails, look under the Reports and Activity section for the failed attempt and find out how the commands are sent to the Cisco Secure ACS server. Based on this information, be sure to permit the commands and the arguments.

• Configure the authorization to intercept the traffic for authorization on the PIX firewall. Otherwise, the authorization may be skipped.

• For the inbound cut-through proxy, define the local IP address as the destination, not the translated IP in the include/exclude statement.

• Define the authorization traffic for authentication, because authorization traffic will be denied if authentication is not performed on the traffic.

Configuring Cut-through Proxy Authorization using the RADIUS Protocol

As the RADIUS protocol cannot perform command authorization, it must rely on the access-list to control the traffic going across the firewall. Whereas the TACACS+ protocol can implement authorization using an access-list in addition to command authorization, ACL usage for controlling traffic is very limited; only downloading the ACL name/number is possible. ACL usage to control the traffic using the RADIUS protocol is very flexible. In addition to having the ability to download the ACL name from the RADIUS server, it is also possible to define and download the contents of the ACL using the RADIUS protocol. The following list explains methods available to implement authorization with ACL using the RADIUS protocol:

• Downloading the name of the ACL from RADIUS Server

• Defining access-lists on the firewall without applying any interface

• Defining the ACL number under the user profile on the RADIUS server

• Upon successful authentication, downloading the number from the AAA server to activate the ACL for controlling users' access to different network resources.

Note

Unlike TACACS+, you do not need to enable authorization on the firewall to download the ACL from the RADIUS Server. This is because the ACL name or the content of the ACL is downloaded from the RADIUS Server as part of the Access Accept packet of the RADIUS Protocol. There is no need to turn on authorization.

On the Cisco Secure ACS Server, access-list name or number can be defined in one of the two following ways.

• Using vendor-specific Attribute Beginning with PIX Firewall Version 5.2, you can define the ACL using the vendor-specific attribute. A vendor-specific attribute for the ACL can be set under the Group setup by checking the box 009\001 AV_Pair (vendor-specific), and defining acl=115 in the Cisco/RADIUS rectangular box.

• Using Standard IETF attribute Beginning with PIX Version 6.0.1, the ACL name/number can be defined using a standard Internet Engineering Task Force (IETF) RADIUS attribute 11 (Filter-Id).

  Access-list 115 needs to be defined on the firewall.

• Downloading the contents of the ACL from the RADIUS Server

  If you have a large number of ACLs to manage for different users, the number of ACLs can outgrow the capacity of the memory of the firewall. In addition, if you have more than one firewall using the same policy to control the traffic, every firewall must be locally configured with the same ACL. So the authorization implementation defining the ACL on the firewall may not be scalable. To address this issue, you can define the ACL content on the AAA server and then download the ACL based on user authentication to one, or more than one, firewall. You must be running PIX Firewall Version 6.2 or above to implement this feature.

Note

Downloading contents of the ACL from the AAA server is not possible with the TACACS+ protocol. With TACACS+, only downloading the ACL name or number is possible, as discussed before.

You can define the ACL on the Cisco Secure ACS server in one of the following ways:

• Using a Vendor-Specific Attribute

  You can use a Cisco vendor-specific attribute to define the ACL that may be downloaded to the firewall. To set this up on the CS ACS Server, edit the group to which the users belong, and go under RADIUS (Cisco IOS/PIX) settings. Choose the [009\001] cisco-av-pair check box, and define the contents of the ACL that is displayed in a rectangular box, as shown in Figure 10-3, to allow only Telnet access and deny all other traffic.

  ip:inacl#=permit tcp any any eq telnet ip:inacl#=deny ip any any

  Figure 10-3. Defining ACL Using the Cisco IOS/PIX RADIUS Attribute.

  Finally click the Submit + Restart button.

• Using Shared Profile Components

  If you have multiple groups that require the same ACL to be applied, defining the same ACL for different groups is not a very scalable solution. Beginning with Cisco Secure ACS 3.0, you can configure a Shared Profile Component, which is a template for the ACL that can be applied to different user groups.

  Note

  Configuring a Shared Profile component may not be possible with RADIUS servers from other vendors. Defining a Shared Profile Component is possible only if you are running Cisco Secure ACS version 3.0 and above.

  Before you can configure a Shared Profile Component on the CS ACS, you must enable that option. Go to the Interface Configuration > Advance Options page. Then check either the User-Level Downloadable ACLs or Group-Level Downloadable ACLs checkbox or both checkboxes, based on your requirements. Finally, click on Submit.

  Next you need to define the Shared Profile Component (ACL template), which can be mapped to different groups. To accomplish this task, click on Shared Profile Components, then Downloadable IP ACLs, and then Add. This will display the Downloadable IP ACLs window, where you need to define the Name and, optionally, Description. To define ACL contents for this ACL, click on Add, which will bring up the Downloadable IP ACL Content page (see Figure 10-4), where you need to provide a Name. In the ACL Definitions rectangle box, define the actual ACE in the ACL which follows the same syntax as the extended ACL of the firewall. For example, to allow only Telnet traffic, the syntax for the ACL Definitions will resemble the following:

  Permit tcp any any eq telnet

  Figure 10-4. Shared Profile Components for Downloadable IP ACL Configuration

  Click the Submit button of the Downloadable IP ACL Content page, which will bring you back to the Downloadable IP ACLs page. Here be sure that the newly created ACL Contents selects the proper firewall (NAS) for the Network Access Filtering. If you choose not to filter it based on a specific firewall, choose All-AAA-Clients, which is selected by default.

  Once the downloadable IP ACL is configured, you can apply this to as many users' groups as you want. To apply Downloadable IP ACL, go to Group Setup >. Select the Group, and then click on Edit Settings, which will display the Group Setup page. In the Group Setup page, from the Jump To drop-down box, choose Downloadable ACLs, which will take you to the Downloadable ACLs settings. Check the Assign IP ACL: checkbox, and from the drop-down box next to it, choose the Downloadable IP ACL name you have just created. Then click the Submit+Restart button.

• Firewall Configuration If you have configured the access-group command to apply ACLs to interfaces, be sure to use the per-user-override keyword. Otherwise, traffic for a user session must be permitted by both the interface ACL and the user-specific ACL. With the per-user-override keyword, the user-specific ACL determines what is permitted, not the interface ACL. The complete syntax for the access-group with the per-user-override argument is as follows:

  access-group access-list {in | out} interface interface_name [per-user-override]

Troubleshooting Cut-Through Proxy Authorization using the RADIUS Protocol

To determine the currently authenticated users, the host IP to which they are bound, and any cached IP and port authorization information, use the show uauth command as follows:

PIX# show uauth [username]

If you do not specify a username, all authenticated users' information will be displayed.

Example 10-25 shows the output of the show uauth command when no user is authenticated.

Example 10-25. Output of the show uauth Command With No Users Authentication

When a user is authenticated, user authentication and authorization information can be found with the show uauth command, as shown in Example 10-26.

Example 10-26. Output of the show uauth Command When Users Are Authenticated/Authorized

Once you verify the user authentication, the next step is to verify whether the ACL tag is downloaded correctly to the PIX firewall. If PIX is configured with an ACL (not applied to any interface) and if the ACL tag is downloaded from the AAA server upon successful user authentication, the ACL should be applied for the authenticated users. This can be verified with the command shown in Example 10-27.

Example 10-27. show access-list Command Output After Successful Download of ACL Name/Number for Allowing Telnet

If the ACL content is downloaded from the RADIUS Server, the downloaded ACL looks like that shown in Example 10-28.

Example 10-28. Output of the show access-list Command When the Content of the ACL Is Downloaded to the PIX from the RADIUS Server

If the show uauth or show access-list command does not show you the output as expected, you can run the debug and analyze the log to see the reason for failure on the PIX firewall as on the Cisco Secure ACS Server.

Example 10-29 shows the syslog for user authentication for only Telnet access and denial of other traffic when ACL content is downloaded from the CS ACS to the PIX firewall.

Example 10-29. PIX Debugs for a Valid Authentication and Downloaded Access List

Following are some of the causes for an authorization failure with the RADIUS protocol for a cut-through proxy connection:

• Override keyword is missing from the access-group statement If you have an access-group applied on the interface, you need to ensure that you have the override optional argument configured in the access-group statement. Otherwise, traffic must be allowed by both interface ACL and per-user ACL configured on the PIX firewall. The syntax for the access-group statement is as follows:

  access-group access-list {in | out} interface interface_name [per-user-override]

• There is a mismatched ACL Name/Number between the PIX and CS ACS If you have configured the access-list on the PIX firewall, or the ACL name is defined on the AAA server, ensure that the names match each other. The name is case sensitive. Otherwise, the proper ACL will not be applied for the user traffic, which might result in unexpected behavior.

Accounting for Cut-Through Proxy

You can use the following syntax to configure accounting for cut-through proxy connection:

[no] aaa accounting {include | exclude} service interface-name local-ip local-mask foreign-ip foreign-mask server-tag

For example, to turn on accounting for all inbound traffic, you may execute the following command:

PIX(config)# aaa accounting include any outside 0.0.0.0 0.0.0.0 0.0.0.0 0.0.0.0 AuthInbound

However, it is recommended to use the following syntax to configure cut-through proxy accounting:

PIX(config)# [no] aaa accounting match acl-name interface-name server-tag

The following command shows how to turn on accounting for all inbound traffic using ACL.

PIX(config)# access-list 150 permit ip any any PIX(config)# aaa accounting match 150 outside AuthRadius

If the accounting does not work as expected, you need to analyze the log on both the PIX firewall and on the AAA server. You need to first examine the debug output of the following command to see if the PIX is generating the accounting packets correctly or not:

PIX# debug aaa accounting

Additionally, to see the interaction of accounting records from the PIX firewall to AAA server, you can use either of the following commands depending on the type of protocols you are using:

PIX# debug radius PIX# debug tacacs

If the accounting appears to be working correctly on the PIX firewall, you can analyze the log on the AAA server to see if the problem is with the AAA server. If Cisco Secure ACS is used for AAA, refer to "Troubleshooting Cisco Secure ACS on Windows" for additional details.

Extended Authentication (X-Auth) Issues for Remote Access VPN Connection

Extended Authentication (X-Auth) is an enhancement of IKE version 1 for a Remote Access VPN connection that allows performing both user and group authentication. When X-Auth is configured on the PIX firewall (FWSM does not support VPN), users are prompted to enter username and password, which is either locally stored in the PIX firewall or in any external database, such as Cisco Secure ACS, ACE Server, NT domain and so on. Extended authentication is performed after IKE phase 1 and before phase 2. In phase 1, group authentication takes place, then X-Auth is performed. Right after X-Auth, if authentication is successful, mode configuration takes place, where PIX Firewall pushes configuration parameters for VPN clients, such as DNS Server IP, WINS Server IP, and so on.

Beginning with PIX Firewall Version 5.2, X-Auth includes support for access-list which allows you to control what traffic needs to be encrypted and what traffic should be transmitted in clear text. The minimum VPN client version requirement is 2.5. This feature is called split-tunneling. If you have a requirement to allow the VPN client to connect to the internal network of the VPN Gateway using the secured tunnel, but want the rest of the traffic to go to the Internet in clear text from the VPN client, you need to configure split-tunneling.

Beginning with PIX Firewall Version 6.2, the download of access-control lists (ACLs) from an AAA server to the PIX firewall is supported. This enables the configuration of a per-user ACL on an AAA server to provide per-user ACL authorization. It is then downloadable through the AAA server to the PIX firewall. This feature is supported for RADIUS servers only, not with TACACS+ servers.

Configuration Steps

With PIX Firewall versions earlier than 7.0, to apply x-auth, use the following command:

[no] crypto map map-name client [token] authentication aaa-server-name [LOCAL]

With username, you can define local users on the PIX firewall. To use the local user database, use the key word LOCAL. Otherwise, define a AAA server and assign the same name defined in the preceding command.

Beginning with PIX Version 7.0, you can define attributes for users who are locally created on the PIX firewall.

Work through the steps that follow to configure the X-Auth for PIX Version 7.0 and above.

Troubleshooting Techniques

Troubleshooting AAA for the Remote Access VPN is very similar to of the troubleshooting described in the sections entitled "Firewall Management with AAA Troubleshooting" and "Cut-Through Proxy with AAA Troubleshooting," with some exceptions. As the connection type is VPN using IPsec, in addition to AAA troubleshooting, you also need to troubleshoot the VPN problem. To debug Remote Access VPN, you may want to refer to Chapter 7, "Troubleshooting IPsec VPN on PIX Firewalls." Following are the two commands that you need to use to troubleshoot IPsec VPN issues:

PIX# debug crypto ipsec 5 PIX# debug crypto isakmp 5 PIX# debug crypto engine 5 PIX#

To troubleshoot AAA issues for the VPN connection, look at the log on VPN Client Log Window, debug on the PIX firewall, and the log on the AAA server.

To turn on logging to the debug level on the VPN client, go to Start > Programs > Cisco Systems VPN Client > VPN Client. Then click on Log Settings and set User Authentication log to High. It is recommended that you also set all other classes to high if the tunnel is failed and you are uncertain about the reason of failure. Refer to Chapter 7, "Troubleshooting IPsec VPN on PIX Firewalls."

To turn on debug level syslogging, refer to the "syslog" subsection of section entitled "Diagnostic Commands and Tools" in this chapter. Run the following command to get details about AAA problems:

PIX# debug aaa authentication PIX# debug aaa authorization PIX# debug aaa accounting PIX# debug aaa vpn 5 PIX# debug radius

If you have authentication problems with the external RADIUS server, you might want to turn on local user database authentication to eliminate the possibility that the external AAA server is causing the problem.

If local user database authentication works well, you need to analyze the log on the AAA sever. For Cisco Secure ACS troubleshooting, you need to analyze the package.cab file, which is discussed in Chapter 13, "Troubleshooting Cisco Secure ACS on Windows."