Sending Logging Messages to a Log Server

Problem

You want to collect and save system logging messages over long periods of time, but you don't want to constantly manage the disk space availability on your routers.

Solution

You can set up a log server on your network that has many gigabytes of storage space and then redirect the router's logging messages to that server.

[edit system syslog] aviva@router1# set host 172.17.12.30 any info aviva@router1# set host 172.17.12.30 explicit-priority

Discussion

This configuration redirects all logging messages to the file server 172.17.12.30. You can also specify the hostname instead of the IP address. The file server must be running a standard syslogd utility. You find the system logging messages in the /var/log/messages file on the server, unless the server has been configured to save them some-place else:

aviva-server1%: tail /var/log/messages Mar 23 09:27:29 server1 /kernel: linux: syscall mmap2 is obsoleted or not implemented (pid=12624) %INTERACT-5-UI_DBASE_LOGIN_EVENT: User 'aviva' entering configuration mode Mar 23 17:48:40 router1-fxp0.mycompany mgd[4098]: %INTERACT-5-UI_COMMIT: User 'aviva' performed commit: no comment Mar 23 17:48:44 router1-fxp0.mycompany xntpd[4860]: %NTP-5: ntpd 4.0.99b Sat Mar 12 07:43:39 GMT 2005 (1) Mar 23 17:48:44 router1-fxp0.mycompany xntpd[4860]: %NTP-5: using kernel phase-lock loop 2001 Mar 23 17:48:44 router1-fxp0.mycompany xntpd[4860]: %NTP-5: using kernel phase-lock loop 2041 Mar 23 17:48:45 router1-fxp0.mycompany mgd[4098]: %INTERACT-5-UI_DBASE_LOGOUT_EVENT: User 'aviva' exiting configuration mode

The output shows both the server's and the router's logging messages intermixed in the logging file. You can identify the messages from the router because the field after the timestamp identifies the router by IP address and router port or, if the server can resolve the IP address, by its DNS name. The router messages in this output show router1-fxp0.mycompany. The router name is router1, and the messages were sent to the log server over port (or interface) fxp0, which is the router's out-of-band management interface. The JUNOS software has a tendency to send logging messages out the interface with the shortest path to the syslog server. This can cause all sorts of problems, depending on how your log server and firewall filters are set up. To circumvent these problems, specifically include the interface from which to send the messages:

[edit system syslog] aviva@router1# set source-address 192.168.15.42/32

This command sets the messages to go out 192.168.15.42, which is the router's loopback address. You normally use the router's loopback address when sending system logging messages, but you can use any interface dedicated to management. With this configuration, logging messages from the router will include the resolved hostname of the IP address for lo0 (for example, lo.router1.mycompany.com) rather than showing router1-fxp0.mycompany.

Sometimes you want to have more information to identify the source of the message than just the router name. You can specify a text string that is also included in the logging message:

[edit system syslog] aviva@router1# set host 172.17.12.30 log-prefix M20-JUNOS-cookbook

The messages from your router now contain this string:

Mar 23 12:01:57 server1 named[45618]: zoneref: Masters for slave zone "mycompany.com" REFUSED transfer Mar 23 20:15:46 router1-fxp0.mycompany M20-JUNOS-cookbook: xntpd[5633]: %NTP-5: ntpd 4.0.99b Sat Mar 12 07:43:39 GMT 2005 (1)

How you specify the router identifier string is a little bit different from how you specify other strings in JUNOS statements. You can use all alphanumeric and special characters except equals signs and colons, but you cannot include spaces, even if you enclose them in quotation marks.

The syslog utility running on your server understands just the standard syslog message facilities. Many of the JUNOS system logging facilities map to the standard syslog ones, but some are JUNOS specific. For example, the JUNOS ftp facility maps to LOG_FTP and kernel maps to LOG_KERNEL, but INTERACT and PFE don't map to anything in syslog. By default, the JUNOS software maps the facilities to a syslog alternate facility. Just as in the Unix syslog utility, the JUNOS software has eight alternate facilities, local0 through local7. Table 5-3 shows the default mappings of the JUNOS-specific facilities to alternate facilities.

To have the server process messages from the JUNOS-specific logging facility, you direct messages having these alternate facilities to a file on the server. On a FreeBSD system, you define this mapping in the /etc/syslog.conf file. To place the interactive messages in a logging file, you could include the following line in the /etc/syslog.conf file:

local7.* /var/log/router-command-messages

When you are collecting logs from a number of routers on the same server, the server cannot distinguish among the different routers and places all messages that have the same facility in the same file. This can get rather messy when you are trying to sort out which messages came from which routers, so you should send each router's messages to its own file. To set this up on the router, choose an alternate facility:

[edit system syslog] aviva@router1# set host 172.17.12.30 facility-override local0

This command causes all messages sent to the remote host to be flagged with the standard local0 facility. On the server, you map to a file in the /etc/syslog.conf file:

local0.* /var/log/M20-JUNOS-cookbook-messages

A check of the file shows the system log messages from the router:

aviva-server1%: tail -4 M20-JUNOS-cookbook-messages Mar 24 00:45:40 router1-fxp0.mycompany M20-JUNOS-cookbook: mgd[5257]: %INTERACT-6-UI_CMDLINE_READ_LINE: User 'aviva', command 'edit system syslog ' Mar 24 01:00:00 router1-fxp0.mycompany M20-JUNOS-cookbook: CRON[8784]: %CRON-6: (root) CMD (newsyslog) Mar 24 01:01:00 router1-fxp0.mycompany M20-JUNOS-cookbook: CRON[8787]: %CRON-6: (root) CMD (adjkerntz -a) Mar 24 01:08:04 router1-fxp0.mycompany M20-JUNOS-cookbook: mgd[5257]: %INTERACT-6-UI_CMDLINE_READ_LINE: User 'aviva', command 'edit host server1 '

You can run system logging management software on the central log server to help analyze the collected log messages. One widely used product is syslog-ng (http://www.balabit.com/products/syslog_ng), which filters logging messages based on source IP address and separates messages from different sources into different files instead of placing them into one file. This is particularly useful for network operators who aggregate messages from several devices. Another widely used tool is swatch (simple watcher; http://swatch.sourceforge.net), which actively scans logfiles entries as soon as syslogd receives them and reports what is happening in real time. swatch can also take action when it encounters certain log messages.