General Troubleshooting Techniques

Cisco UE offers several types of tools to help you isolate problems on the system:

• Viewing system information and status (using a variety of show commands)
• Event logging to log files including information, warning, error, and fatal messages
• A tracing facility showing detailed information on system events, information, and decisions (similar to Cisco IOS debug information)

Viewing Generic System Information

This book provides you with information about Cisco UE so that you can troubleshoot and fix most of the common problems you might encounter. In case you still have to contact the Cisco Technical Assistance Center (TAC) for additional support, however, it is helpful to provide the Cisco TAC with some basic information about your system. This basic information includes the software version, the software license, the current configuration of the Cisco CME and Cisco UE components, and some information about the current usage of the system. This section describes how to view that basic information.

Example 19-1 uses the show version command to provide information about the hardware form factor of the Cisco UE module, the model type of the router that is hosting the Cisco UE, the speed with which CPU is running, and additional hardware-specific information. One thing to note is that the AIM-CUE runs at 150 MHz if it is hosted on 2600XM or 2691 series access routers. In other routers, it runs at 300 MHz.

Example 19-1. Displaying the System Version

CUE#show version CPU Model: Celeron (Coppermine) CPU Speed (MHz): 154.926 CPU Cache (KByte): 128 Chassis Type: C2600 Chassis Serial: FSJC0400673 Module Type: AIM Module Serial: FHH07330001

Example 19-2 uses the show software version command to display the versions of the Cisco UE software components installed on the system. Identifying the installed software versions is important when troubleshooting interworking issues. For example, some Cisco UE features might be supported only with a certain version of Cisco CME or Cisco UE.

Example 19-2. Displaying the Software Version

CUE#show software version Installed Packages: - Bootloader (Primary) 1.0.17 - Global 2.0.0.12 - Voice Mail 2.0.0.12 - Bootloader (Secondary) 2.0.0 - Core 2.0.0.12 - Auto Attendant 2.0.0.12 Installed Languages: - US English 2.0.0.0

Example 19-3 shows the license installed on the system by using the show software licenses command.

Example 19-3. Displaying Licenses

CUE#show software licenses Core: - application mode: CCME - total usable system ports: 4 Voicemail/Auto Attendant: - max system mailbox capacity time: 480 - max general delivery mailboxes: 15 - max personal mailboxes: 50 Languages: - max installed languages: 1 - max enabled languages: 1

Example 19-4 uses show voicemail usage to show the current usage of the different resources in the system, including the number of personal mailboxes, general delivery mailboxes (GDMs), and the voice mail capacity allocation.

Example 19-4. Displaying the System's Current Usage

CUE#show voicemail usage personal mailboxes: 50 general delivery mailboxes: 0 orphaned mailboxes: 0 capacity of voicemail (minutes): 480 allocated capacity (minutes): 350.0 message time used (seconds): 15227 message count: 856 average message length (seconds): 17.78855140186916 greeting time used (seconds): 0 greeting count: 0 average greeting length (seconds): 0.0 total time used (seconds): 15227 total time used (minutes): 253.78334045410156 percentage used time (%): 53

The show voicemail limits command, shown in Example 19-5, shows the limits currently enforced on the voice mail system (based on the system's license). These limits include the default mailbox size, default recording limit for callers and subscribers, and message expiration duration.

Example 19-5. Displaying System Voice Mail Limits

CUE#show voicemail limits Default Mailbox Size (seconds): 420 Default Caller Message Size (seconds): 60 Maximum Recording Size (seconds): 900 Default Message Age (days): 30 System Capacity (minutes): 480 Default Prompt Language: en_US Operator Telephone: 0

Logging in Cisco UE

Cisco UE provides a full-fledged logging facility. Messages are logged to the console, to a Syslog server, or to the messages.log file on the disk or flash. Cisco UE has four levels of logging. Table 19-1 shows the Cisco UE logging levels mapped to Syslog logging levels.

If you are monitoring straightforward issues, such as whether the Cisco UE system reloaded, you might want to look at the Syslog messages before you start troubleshooting by turning on traces. These messages provide a brief problem description, a possible reason for the event, and other relevant information about a problem. These messages might be displayed on the console, written to a Syslog server, or stored in the Cisco UE messages.log file, depending on the system's configuration. Example 19-6 shows different ways of turning on logging in Cisco UE. Logging to the console displays the log messages on the console when they occur.

Example 19-6. Logging Levels and Configuration

cue(config)#log console ? errors Error messages filter Filter events from syslog info Information messages warning Warning messages cue(config)#log console info Logging to a server cue(config)#log server 1.4.53.1

If you are using a Network Module Cisco UE (NM-CUE), the log messages are also written to a file called messages.log on the hard disk. If you want offline access to the file, you can use command-line interface (CLI) commands to copy the messages.log file to an FTP server. Example 19-7 shows the log files on the system.

Example 19-7. Viewing Log Files on Cisco UE

cue#show logs linux_session.log dmesg syslog.log shutdown.log atrace.log debug_server.log memmon.log klog.log messages.log

The following steps show you how to copy the log file to an external server:

Tracing Techniques

Cisco UE provides an extensive CLI-based tracing and debugging facility. It provides tracing commands for most of the system's important components, such as voice mail, the Customer Response Solutions (CRS) component, the web GUI interface, the Session Initiation Protocol (SIP) call processing component, and voice mail networking. Tracing uses the concepts of module, entity, and activity. For example, consider the following command:

cue#trace ccn managerappl dbug

In this example, trace is the command, ccn is the module that represents the CRS component, and managerappl (which stands for application manager) is the entity within the ccn module. The dbug parameter is the activity to be traced.

By default, trace output is stored in a memory buffer. You can use commands to view the trace buffer contents on the console. You also have the option of configuring the trace output to write to a file called atrace.log. If you are using an NM-CUE, writing trace output to the disk is enabled by default. On the Advanced Integration Module (AIM-CUE) hardware, tracing is disabled by default. You should enable it with care, because the AIM-CUE flash card has space limitations and a finite lifetime in terms of the number of write cycles before you must replace it. Doing indiscriminate tracing to your flash card shortens its life unnecessarily. Instead, if you are using an AIM-CUE, it is recommended that you use an external FTP server as the destination device for writing trace files.

trace Command Summary

Table 19-2 lists the most important trace commands. You can find the complete list of commands for tracing, modules, entities, and activities in the "Cisco UE System Administration and CLI Guides" available on Cisco.com.

Turning on Traces

Depending on the problem area you are facing, you turn on traces for specific modules and entities. You might often have to turn on traces for several different modules and entities simultaneously to see how each module handles a particular data item or event. You might have to clear the trace buffer between different trace activities to make the troubleshooting output more readable and manageable, especially because the console connection speed is hard-coded to 9600 baud.

Each trace message in the output contains a time stamp of when the trace was generated. It also contains the module and entity that issued the trace output line so that when you read the trace, it is easy to understand which trace line was originated by which module. For example, in the following trace line, ACCN is the module (and stands for the CCN/CRS component), and VBRO is the entity (and stands for the voice browser).

2699 10/10 12:23:30.115 ACCN VBRO 0 Task:18000000079 Invoke: http://localhost/voicemail/vxmlscripts/login.vxml

For most problems, turning on trace for a couple of modules and entities is sufficient, and the trace output is a manageable volume. More complex problems might require tracing to be turned on for many modules and entities, and the trace output can become cumbersome.

If a trace is turned on for only a couple of modules, it is easy to view the trace in real time. Thus, you turn on the trace, clear the trace buffer, issue the show trace buffer tail command, and make the call to trigger the system events you want to trace. As the call enters the Cisco UE application, you see trace output printed on the console. You can stop the trace output by pressing Ctrl-C, as shown in Example 19-8. The trace is cut off the moment you press Ctrl-C.

Example 19-8. Viewing a Trace in Real Time

cue#trace ccn VbrowserCore dbug cue#clear trace cue#show trace buffer tail Press to exit... DefaultCompilationConfig is now set to JVM en_US 2043 09/25 15:56:22.305 ACCN VBRW 0 Task:20000000089 enterLevel: SESSION_LEVEL 2043 09/25 15:56:22.306 ACCN VBRW 0 callContact.getSessionID() = 6000000059 2043 09/25 15:56:22.306 ACCN VBRW 0 Task:20000000089 VoiceBrowser sessionid: 6000000059, taskid: 20000000089 2043 09/25 15:56:22.306 ACCN VBRW 0 callContact.getCallingNumber() = 5702 2043 09/25 15:56:22.306 ACCN VBRW 0 callContact.getCallingNumber() = 5702 2043 09/25 15:56:22.306 ACCN VBRW 0 callContact.getDNIS() = null 2043 09/25 15:56:22.306 ACCN VBRW 0 callContact.getCalledNumber() = 5800 2043 09/25 15:56:22.307 ACCN VBRW 0 callContact.getDNIS() = null 2043 09/25 15:56:22.307 ACCN VBRW 0 callContact.getCalledNumber() = 5800 2043 09/25 15:56:22.307 ACCN VBRW 0 callContact.getANIIIDigits() = null 2043 09/25 15:56:22.307 ACCN VBRW 0 callContact.getUserToUserInfo() = null 2043 09/25 15:56:22.307 ACCN VBRW 0 callContact.getRDNIS() = null 2043 09/25 15:56:22.307 ACCN VBRW 0 callContact.getLastRedirectedNumber() = null 2043 09/25 15:56:22.307 ACCN VBRW 0 callContact.getSessionID() = 6000000059 2043 09/25 15:56:22 cue#

If you run into a situation where you have to turn on tracing for many modules and entities, or even trace all, it is better not to attempt to watch the trace output in real time, because it becomes very difficult to follow. All the trace output is written to a file called atrace.log. On the NM-CUE, this file is written by default. On an AIM-CUE, use the log trace local enable command to allow the trace buffer to be written to flash. You can also use log trace server so that trace output is written to an FTP server instead of a flash on your Cisco UE, as shown in Example 19-9.

Example 19-9. Writing Trace Output to a File

cue(config)#log trace server url ftp://userid:password@192.168.0.1/trace cue(config)#log trace server enable

After you make the preceding configuration, the trace file is created on the specified server.

Copying a Trace File to an FTP Server

After you make the test call to trigger the events you want to capture to a trace file, you must copy the atrace.log file to an FTP server, as shown in Example 19-10.

Example 19-10. Displaying and Copying a Trace File

cue#show logs linux_session.log dmesg syslog.log shutdown.log atrace.log debug_server.log memmon.log klog.log messages.log cue#copy log atrace.log url ftp://user:password@1.3.61.16/atrace.log

The atrace.log file is a binary file that you must decode to view in a text editor. You can get the utility to decode the trace file from the Cisco TAC. The utility consists of three files:

• atrace_decode (an executable)
• trace.tcmd
• trace.tmsg

Decode the atrace.log file as shown in Example 19-11. This creates an output file (dtmf_issue.log, in this example) that you can now view in a text editor. Currently, atrace_decode runs only on a Linux platform.

Example 19-11. Decoding an atrace.log Trace File

[test@CUE-core ftp]$ ./atrace_decode Usage: ./atrace_decode format = 0 for short, 1 for long [test@CUE-core ftp]$./atrace_decode atrace.log 1 dtmf_issue.log ./trace.tcmd ./trace.tmsg

Troubleshooting System Time Issues and NTP

Many of the Cisco UE features depend on a reliable clock. The voice message envelope information is one important example. As discussed in Chapter 13, "Cisco IPC Express General Administration and Initial System Setup," Cisco UE's clock is managed entirely by Network Time Protocol (NTP) and cannot be set manually.

Another Cisco UE feature that requires clock information is the trace and log messages that carry time stamps. If the clock on the Cisco UE system is not set correctly, troubleshooting might become very difficult, because log and trace messages cannot be correlated with real time.

You should configure an NTP source for Cisco UE that is the clock's source. An NTP source can be the host Cisco CME router (although this is not recommended because the low-end routers do not have a very reliable onboard clock source) or any other clock source in your network. Sometimes, for many reasons, the clock on Cisco UE might not be synchronized to its configured source. You can verify the status of NTP and the location of the clock's source with CLI commands. Example 19-12 shows these commands.

Example 19-12. NTP-Related Commands

NTP not working due to wrong configuration: CUE#show ntp status NTP reference server : 1.3.6.190 Status: reject Time difference (secs): 0.0 Time jitter (secs): 4.0 NTP in Sync: CUE#show ntp status NTP reference server : 1.4.13.190 Status: sys.peer Time difference (secs): 0.02110639284364879 Time jitter (secs): 0.019472182961180806 Trace commands available to debug NTP synchronization issues: CUE#trace ntp ? all Every entity and activity ntp Entity CUE#trace ntp ntp ? all Every activity clkadj Activity clkselect Activity clkvalidity Activity clockstats Activity event Activity loopfilter Activity loopstats Activity packets Activity peerstats Activity