12    Administering the Basic System Event Channels

This chapter explains how system events are logged and describes how to configure the basic system event logging channels. Information on managing log files is also included.

The following topics are discussed in this chapter:

• Section 12.1 explains your options for monitoring system events.

• Section 12.2 describes how to set up event monitoring.

• Section 12.3 describes how to recover and read event logs after the system has crashed.

• Section 12.4 explains your options for managing log files.

12.1    Understanding the Basic Event-Logging Facilities

The operating system uses three mechanisms to log system events:

• The system event-logging facility, also known as syslogd. Refer to syslogd(8) for information on the initialization options and syslog.conf(4) for information on configuration options. See syslog.auth(4) for information on remote logging.

• The binary event-logging facility also known as binlogd. Refer to binlogd(8) for information on the initialization options and binlog.conf(4). See binlog.auth(4) for information on remote logging.

• The Event Manager (EVM) provides an integrated approach to administering system events and errors. See EVM(5) for an introduction to EVM, and see Chapter 13 for information on configuring and using EVM.

You can review events detected and recorded by syslogd and binlogd using the Event Manager (EVM), DECevent, or the error report formatter, uerf.

EVM is the recommended method of administering system events. Refer to Chapter 13 for information on configuring EVM. The EVM viewer, evmviewer, provides a graphical interface for selecting, filtering, and displaying system events. See the EVM(5) and evmviewer(8) reference pages for more information.

System events are often returned in a binary format. To render such events in a readable text format you must use a translation tool such as:

• The service tools provided with your service contract contain event analysis tools such as Compaq Analyze. Refer to http://www.support.compaq.com/svctools/webes/index.html for more information. Recent processor models produce binlogd events using a header format that differs from the format produced by earlier platforms. The newer format events are known as Common Event Header (CEH) events. If your system does not produce CEH events you cannot use Compaq Analyze to translate them, and you must install the DECevent formatter utility, /usr/sbin/dia.

• A limited use license for DECevent is provided in the distribution kit as described in the Installation Guide. Refer to the DECevent Translation and Reporting Utility guide and dia(8) for more information.

• The uerf command utility. See uerf(8).

  Note

  The uerf command utility does not support CEH events and will be retired in a future release. You should migrate your event management procedures to EVM as soon as possible.

The log files created by the event-logging facilities are protected and owned by root, and belong to the adm group. You must have the proper authority to examine the files.

The following sections describe the event-logging facilities.

12.1.1    System Event Logging

The primary event-logging facility uses the syslog function to log system-wide events in ASCII format. The syslog function uses the syslogd daemon to collect the messages that are logged by the various kernel, command, utility, and application programs. The syslogd daemon logs the messages to a local file or forwards the messages to a remote system, as specified in the /etc/syslog.conf file.

When you install the operating system, the /etc/syslog.conf file is created and specifies the default event-logging configuration. The /etc/syslog.conf file specifies the file names that are the destination for the event messages, which are in ASCII format. Section 12.2.1.1 discusses the /etc/syslog.conf file. Refer also to syslog.conf(4).

The /etc/syslog.auth file specifies which remote hosts are allowed to forward syslog messages to the local host. For system security, only messages coming from remote hosts listed in this file are logged by the syslogd daemon. If the /etc/syslog.auth file is not present, then event forwarding from all remote hosts is enabled.

The /etc/syslog_evm.conf file specifies which syslogd messages are forwarded from the syslogd daemon to EVM, in the form of EVM events. Those syslogd messages are posted to the EVM daemon, evmd, by syslogd if the syslogd forwarding function is turned on with the -e option. Event forwarding is always on by default. Use the -E option to turn it off if required. Events are posted with the EVM name of sys.unix.syslog.facility.

Refer to syslog.auth(4) and syslog_evm.conf(4) for more information.

12.1.2    Binary Event Logging

The binary event-logging facility detects hardware and software events in the kernel and logs the detailed information in binary format records. Some events that are logged by the binary event-logging facility are also logged by the syslog function in a less detailed message.

The binary event-logging facility uses the binlogd daemon to collect various event-log records. The binlogd daemon logs these records to a local file or forwards them to a remote system, as specified in the /etc/binlog.conf default configuration file, which is created when you install your system. Section 12.2.1.3 discusses the /etc/binlog.conf file.

You use DECevent (or Compaq Analyze) to translate binary events to ASCII reports from entries in the system's binary event log files. Invoke DECevent by entering the dia command at the command line. Entering the command without any options causes DECevent to immediately access and translate the contents of the event log files, displaying the events as shown in Example 12-1. Events will scroll up the terminal screen until all events are displayed or you press [Ctrl/c].

Example 12-1:  Sample Translated Event

**** V3.3  ****************** ENTRY  4 ************************    [1]
Logging OS                              2[OS]    [2]
System Architecture                     2.
Alpha Event sequence number             440. 
Timestamp of occurrence                 22-MAR-2001 18:24:31    [3] 
Host name                               Host Name
 
System type register      x0000001B     AlphaServer 800 or 1000A 
Number of CPUs (mpnum)    x00000001 
CPU logging event (mperr) x00000000 
 
Event validity    [4]                 1. O/S claims event is valid
Event severity                         5. Low Priority
Entry type                             301. Shutdown ASCII
Message Type                           -1.    - (minor class) 
SWI Minor class                        9. 
ASCII Message SWI Minor sub class      2. Shutdown  ASCII Message 
System halted by root:  System going down @ 6:24PM on 22 Mar 
Please log off in good time [5]

1. The number of the event in the translated log. Note that the number might be based on the selection or filtering of events. [Return to example]

2. Identification of the operating system ([OS]) and system architecture. [Return to example]

3. The timestamp (date and system clock time) that indicates when the event occurred and the name of the system on which it occurred (<host name>). [Return to example]

4. Information about the validity, severity, and type of event. In this case, an informational message that the system was shutting down. [Return to example]

5. The actual message logged by the event, which might also have been displayed to a terminal or console at the time the event occurred. [Return to example]

For information about administering the DECevent utility, see the following documentation:

• DECevent Translation and Reporting Guide

• dia(8)

Compaq Analyze is a rules-based hardware fault management diagnostic application that provides error event analysis and translation. The multi-event correlation analysis feature of Compaq Analyze provides you with the capability to analyze events stored in the binary system event log or other specified binary log files. When Compaq Analyze is installed, you can launch its GUI interface directly from the SysMan Station by clicking on the Host Icon and selecting Compaq Analyze from the Tools menu.

12.2    Configuring Event Logging

You can change the default configuration by modifying the configuration files as described in this section. For example, you can change the configuration so that only important, system-critical events are logged and informational events are ignored. You can choose to concentrate on certain subsystems, such as mail or print services, and control how and where event messages are logged. The optimum method of monitoring system events is to use Event Manager (EVM), as described in Chapter 13. EVM enables you to consolidate and filter events.

To enable system and binary event-logging, the special files must exist and the event-logging daemons must be running. Refer to Section 12.2.3 and Section 12.2.4 for more information.

The file /var/adm/syslog.dated and other files in /var/adm directory are context-dependent symbolic links (CDSLs), which facilitate joining single systems into clusters. The CDSL for the syslog directory is /var/cluster/members/member0/adm/syslog.dated. Take care not to break symbolic links when working with these files. Refer to Chapter 6 for more information on CDSLs.

12.2.1    Editing the Configuration Files

If you do not want to use the default system or binary event-logging configuration, you can edit the /etc/syslog.conf or /etc/binlog.conf configuration file to specify how the system should log events. In the files, you specify the following data:

• The facility, which is the source of a message or the part of the system that generates a message

• The priority, which is the message's level of severity

• The destination for messages.

The following sections describe how to edit the configuration files.

12.2.1.1    Editing the syslog.conf File

If you want the syslogd daemon to use a configuration file other than the default, you must specify the file name with the syslogd -f config_file command.

The following is an example of the default /etc/syslog.conf file:

#
# syslogd config file
#
# facilities: kern user mail daemon auth syslog lpr binary
# priorities: emerg alert crit err warning notice info debug
#
# [1]  [2]                            [3]
kern.debug               /var/adm/syslog.dated/kern.log
user.debug               /var/adm/syslog.dated/user.log
daemon.debug             /var/adm/syslog.dated/daemon.log
auth.crit;syslog.debug   /var/adm/syslog.dated/syslog.log
mail,lpr.debug           /var/adm/syslog.dated/misc.log
msgbuf.err               /var/adm/crash.dated/msgbuf.savecore
kern.debug               /var/adm/messages
kern.debug               /dev/console
*.emerg                  *

Each /etc/syslog.conf file entry has the following entry syntax:

1. Specifies the facility, which is the part of the system generating the message. [Return to example]

2. Specifies the severity level. The syslogd daemon logs all messages of the specified severity level plus all messages of greater severity. For example, if you specify level err, all messages of levels err, crit, alert, and emerg or panic are logged. [Return to example]

3. Specifies the destination where the messages are logged. This might be a log file or a device such as /dev/console. [Return to example]

The syslogd daemon ignores blank lines and lines that begin with a number sign (#). You can specify # as the first character in a line to include comments in the /etc/syslog.conf file or to disable an entry.

The facility and severity level are separated from the destination by one or more tab characters or spaces.

You can specify more than one facility and its severity level by separating them with semicolons. In the preceding example, messages from the auth facility of crit severity level and higher and messages from the syslog facility of debug severity level and higher are logged to the /var/adm/syslog.dated/syslog.log file.

You can specify more than one facility by separating them with commas. In the preceding example, messages from the mail and lpr facilities of debug severity level and higher are logged to the /var/adm/syslog.dated/misc.log file.

You can specify the following facilities:

You can specify the following severity levels, which are listed in order of highest to lowest severity:

You can specify the following message destinations:

You can specify that the syslogd daemon create daily log files. To create daily log files, use the following syntax to specify the pathname of the message destination:

/var/adm/syslog.dated/ {file}

The file variable specifies the name of the log file, for example, mail.log or kern.log.

If you specify a /var/adm/syslog.dated/file pathname destination, each day the syslogd daemon creates a subdirectory under the /var/adm/syslog.dated directory and a log file in the subdirectory using the following syntax:

/var/adm/syslog.dated/ date / file

Where:

• The date variable specifies the day, month, and time that the log file was created.

• The file variable specifies the name of the log file you specified in the /etc/syslog.conf file.

The syslogd daemon automatically creates a new date directory every 24 hours, when you boot the system, or when the syslogd daemon is restarted or reconfigured. You can get the latest logs from the /var/adm/syslog.dated/current directory. The current directory is a symbolic link to the latest date directory.

For example, to create a daily log file of all mail messages of level info or higher, edit the /etc/syslog.conf file and include a line similar to the following:

mail.info		/var/adm/syslog.dated/mail.log
 

If you specify the previous line in the /etc/syslog.conf, the syslogd daemon creates the following daily directory and file:

/var/adm/syslog.dated/11-Jan-12:10/mail.log

12.2.1.2    Configuring syslog to Use EVM

By default, syslogd is configured with the -e option to forward events to EVM. (See Section 12.2.4). You can select which syslog events are forwarded to EVM by modifying the syslog_evm.conf file. If the file does not exist, or if it exists but contains no subscription entries, no syslog messages are posted to EVM.

The default syslog_evm.conf file contains entries similar to those shown in Example 12-2, which excludes the informational file header.

Example 12-2:  Sample syslog_evm.conf File Entries

[1]  [2]
*.emerg
# above forwards all emergency events to EVM  [3]
kern.info+   [4]
user.notice+
mail.notice+
daemon.notice+
auth.notice+
syslog.notice+

1. The first part of each line item specifies which facility generated the message, such as kern for kernel. An asterisk (*) indicates that all facilities are selected. In this case, *.emerg ensures that all messages of emergency priority are forwarded to EVM.

   You can choose which events are forwarded by creating an entry for a facility, or removing an existing entry. Entries are based on the keywords in the facility table in Section 12.2.1.1. [Return to example]

2. The second part of each item specifies the priority of messages, based on the keywords in the severity level table in Section 12.2.1.1. [Return to example]

3. You can add comments, preceded by a number sign (#). However, you cannot mix forwarding entries and comments in the same line [Return to example]

4. The plus sign (+) appended to a priority indicates that the specified priority and all higher priority messages are forwarded. If you want to choose individual severity levels for a facility (such as warning, critical and emergency, create a line for each priority. [Return to example]

For more information, refer to syslog_evm.conf(4) and Chapter 13.

12.2.1.3    Editing the binlog.conf File

If you want the binlogd daemon to use a configuration file other than the default, specify the file name with the binlogd -f config_file command. The binlogd daemon forwards all events to EVM. You can filter and select binlog events using EVM utilities, as described in Chapter 13.

You can forward binlogd events to a remote host. Refer to the binlogd(8) for information on the remote logging options. The -R and -r optionsa are important because you use them to control the creation of an inet port for remote access.

The following is an example of a /etc/binlog.conf file:

#
# binlogd configuration file
#
# format of a line:   event_code.priority         destination
#
# where:
# event_code - see codes in binlog.h and man page, * = all events
# priority   - severe, high, low, * = all priorities
# destination - local file pathname or remote system hostname
#
#
*.*			/usr/adm/binary.errlog
dumpfile		/usr/adm/crash/binlogdumpfile
102.high		/usr/adm/disk.errlog
[1]  [2]                   [3]

Each entry in the /etc/binlog.conf file, except the dumpfile event class entry, contains three fields:

1. Specifies the event class code that indicates the part of the system generating the event. [Return to example]

2. Specifies the severity level of the event. Do not specify a severity level if you specify dumpfile for an event class. [Return to example]

3. Specifies the destination where the binary event records are logged. [Return to example]

The binlogd daemon ignores blank lines and lines that begin with a number sign (#). You can specify # as the first character in a line to include comments in the file or to disable an entry.

The event class and severity level are separated from the destination by one or more tab characters or spaces.

You can specify the following event class codes:

You can specify the following severity levels:

You can specify the following destinations:

12.2.2    syslog Security and Remote Messages

Unless the domain host name of a remote host is entered in the local /etc/syslog.auth file, the local system will not log any syslog messages from that remote host. If you intend to make syslogd secure on your system, and you have configured or intend to configure other hosts to forward syslog messages to the system, complete the following steps:

1. Use the su command to become the superuser (root).

2. Create the /etc/syslog.auth file using a text editor. This file must be owned by root and have permissions set to 0600.

3. Add the names of any remote hosts that are allowed to forward syslog messages to the local system. Host names must meet the following criteria:

   • Each remote host name should appear in a separate line in the /etc/syslog.auth file. (Lines beginning with the # character are comments and are ignored.)

   • A host name must be a complete domain name such as trout.fin.huk.com.

   • If a domain host name is given, it must either appear in the local /etc/hosts file or the local system must resolve it through a name server (such as BIND).

   • A host name can have at most as many characters as defined by the MAXHOSTNAMELEN constant in the /sys/include/sys/param.h file, although each line in the /etc/syslog.auth file is limited to 512 characters.

A plus sign (+) by itself allows event forwarding from all hosts. A host name can also be preceded by a minus sign (-) to expressly prohibit that host from forwarding events. If the /etc/syslog.auth file is not present on the system, then forwarding from all hosts is enabled.

Specify the -R option when starting the daemon if you do not want the syslogd daemon to create an inet port to listen for events being sent by remote hosts. To make this the default mode of operation, edit the startup command line in the /sbin/init.d/syslog file. Using the -R option also means that the syslogd daemon cannot forward events to other systems.

Refer to the syslog.auth(4) and syslogd(8) reference pages for additional information.

12.2.3    Creating the Special Files

The syslogd daemon cannot log kernel messages unless the /dev/klog character special file exists. If the /dev/klog file does not exist, create it as follows:

/dev/MAKEDEV /dev/klog

Also, the binlogd daemon cannot log local system events unless the /dev/kbinlog character special file exists. If the /dev/kbinlog file does not exist, create it as follows:

/dev/MAKEDEV /dev/kbinlog

Refer to MAKEDEV(8) for more information.

12.2.4    Starting and Stopping the Event-Logging Daemons

The syslogd and binlogd daemons are automatically started by the init program during system startup. However, you must ensure that the daemons are started. You can also specify options with the command that starts the daemons.

12.2.4.1    The syslogd Daemon

You must ensure that the init program starts syslogd daemon. If the syslogd daemon does not start, or if you want to specify options with the command that starts the syslogd daemon, you must edit the /sbin/init.d/syslog file. When you edit the file, you must either include or modify the syslogd command line. You can also invoke the command manually.

The command that starts the syslogd daemon has the following syntax:

/usr/sbin/syslogd [-b rcvbufsz] [-d] [ -e | -E ] [-f config_file] [-m mark_interval] [-p path] [ -r | -R ] [-s]

The initialization of the daemon uses only the -e option by default. The -e option configures the daemon to automatically forward events to EVM. You can verify the current syslogd configuration using the ps command as follows:

# /sbin/ps agx | grep syslogd

261 ??    S    0:00:10  usr/sbin/syslogd -e

Refer to syslogd(8) for information on the command options.

Note

You must ensure that the /var/adm directory is mounted, or the syslogd daemon will not work correctly.

The syslogd daemon reads messages from the following:

• The domain socket /dev/log file, which is automatically created by the syslogd daemon.

• An Internet domain (UDP) socket, which is specified in the /etc/services file. For security reasons, you might want to either disable this socket using the -R option or specify authorized hosts in the /etc/syslog.conf file.

• The device special /dev/klog file, which logs only kernel messages.

Messages from other programs use the openlog, syslog, and closelog calls.

When the syslogd daemon is started, it creates the /var/run/syslog.pid file, where the syslogd daemon stores its process identification number. Use the process identification number to stop the syslogd daemon before you shut down the system.

During normal system operation, the syslogd daemon is called if data is put in the kernel syslog message buffer, located in physical memory. The syslogd daemon reads the /dev/klog file and gets a copy of the kernel syslog message buffer. The syslogd daemon starts at the beginning of the buffer and sequentially processes each message that it finds. Each message is prefixed by facility and priority codes, which are the same as those specified in the /etc/syslog.conf file. The syslogd daemon then sends the messages to the destinations specified in the file.

To stop the syslogd event-logging daemon, use the following command:

# kill `cat /var/run/syslog.pid` 

Using the following command, you can apply changes to the /etc/syslog.conf configuration file without restarting the daemon:

# kill -HUP `cat /var/run/syslog.pid` 

12.2.4.2    The binlogd Daemon

You must ensure that the init program starts the binlogd daemon. If the binlogd daemon does not start, or if you want to specify options with the command that starts the binlogd daemon, you must edit the /sbin/init.d/binlog file and either include or modify the binlogd command line. Note that you can also invoke the command manually. The binlogd command supports the following options

/usr/sbin/syslogd [-d] [-f config_file] [ -r | -R ]

Refer to binlogd(8) for information on command options.

The binlogd daemon reads binary event records from the following:

• An Internet domain socket (binlogd, 706/udp), which is specified in the /etc/services file. For security reasons, you might want to either disable this socket using the -R option. You can also specify authorized hosts in the /etc/binlog.conf file.

• The /dev/kbinlog special file.

When the binlogd daemon starts, it creates the /var/run/binlogd.pid file, where the binlogd daemon stores its process identification number. Use the process identification number to stop or reconfigure the binlogd daemon.

During normal system operation, the binlogd daemon is called if data is put into the kernel's binary event-log buffer or if data is received on the Internet domain socket. The binlogd daemon then reads the data from the /dev/kbinlog special file or from the socket. Each record contains an event class code and a severity level code. The binlogd daemon processes each binary event record and logs it to the destination specified in the /etc/binlog.conf file.

To stop the binlogd daemon, use the following command:

# kill `cat /var/run/binlogd.pid` 

Using the following command, you can apply changes to the /etc/binlog.conf configuration file without restarting the daemon:

# kill -HUP `cat /var/run/binlogd.pid` 

12.2.5    Configuring the Kernel Binary Event Logger

To configure the kernel binary event logger, modify the default keywords and rebuild the kernel. You can:

• Scale the size of the kernel binary event-log buffer to meet your system needs.

• Enable and disable the binary event logger and the logging of kernel ASCII messages into the binary event log.

The /sys/data/binlog_data.c file defines the binary event-logger configuration. The default configuration specifies a buffer size of 24K bytes, enables binary event logging, and disables the logging of kernel ASCII messages. You can modify the configuration by changing the values of the binlog_bufsize and binlog_status keywords in the file.

The binlog_bufsize keyword specifies the size of the kernel buffer that the binary event logger uses. The size of the buffer can be between 8 kB (8192 bytes) and 1 MB (1048576 bytes). Small system configurations, such as workstations, can use a small buffer. Large server systems that use many disks might need a large buffer.

The binlog_status keyword specifies the behavior of the binary event logger. You can specify the following values for the binlog_status keyword:

0 (zero)

Disables the binary event logger.

BINLOG_ON

Enables the binary event logger.

BINLOG_ASCIION

Enables the logging of kernel ASCII messages into the binary event log if the binary event logger is enabled. This value must be specified with the BINLOG_ON value as follows: int binlog_status = BINLOG_ON | BINLOG_ASCII;

After you modify the /sys/data/binlog_data.c file, you must rebuild and boot the new kernel.

12.3    Recovering Event Logs After a System Crash

You can recover unprocessed messages and binary event-log records from a system crash when you reboot the system.

The msgbuf.err entry in the /etc/syslog.conf file specifies the destination of the kernel syslog message buffer msgbuf that is recovered from the dump file. The default /etc/syslog.conf file entry for the kernel syslog message buffer file is as follows:

msgbuf.err            /var/adm/crash/msgbuf.savecore
 

The dumpfile entry in the /etc/binlog.conf file specifies the file name destination for the kernel binary event-log buffer that is recovered from the dump file. The default /etc/binlog.conf file entry for the kernel binary event-log buffer file is as follows:

dumpfile              /usr/adm/crash/binlogdumpfile
 

If a crash occurs, the syslogd and binlogd daemons cannot read the /dev/klog and /dev/kbinlog special files and process the messages and binary event records. When you reboot the system, the savecore command runs and, if a dump file exists, recovers the kernel syslog message and binary event-log buffers from the dump file. After savecore runs, the syslogd and binlogd daemons are started.

The syslogd daemon reads the syslog message buffer file, checks that its data is valid, and then processes it in the same way that it normally processes data from the /dev/klog file, using the information in the /etc/syslog.conf file.

The binlogd daemon reads the binary event-log buffer file, checks that its data is valid, and then processes the file in the same way that it processes data from the /dev/kbinlog special file, using the information in the /etc/binlog.conf file.

After the syslogd and binlogd daemons are finished with the buffer files, the files are deleted.

12.4    Managing Log Files

On a well maintained system, the size of the various log files should not become a problem as you will:

• Carefully select only those events that you want to log

• Monitor the logs for error conditions that result in many postings

• Regularly archive and back up your important event logs

The /var/spool/cron/crontabs/root file contains the following model entry for managing log files:

0 2 * * 0 /usr/lbin/logclean /var/adm/wtmp > /dev/null
 
 

You can use the cron daemon to specify that other log files be deleted. However, you should also take care that important log files are stored or archived according to your local site requirements.

The following is an example of a crontab file entry that cleans up the older logs in the /var/adm/syslog.dated directory:

5 1 * * * find /var/adm/syslog.dated -type d -mtime +5 -exec rm -rf '{}' \;
 

This entry causes all directories under the /var/adm/syslog.dated directory that were modified more than 5 days ago to be deleted, along with their contents, every day at 1:05. Refer to Chapter 3 and crontab(1) for more information.

12.5    Startup Log Messages in /var/adm/messages

The size of the message buffer used to store boot-log messages is controlled by the msgbuf_size kernel attribute. The minimum default value for this attribute is 8K bytes, for systems with up to 128 MB of physical memory. For systems with greater than 128 MB of physical memory, the value of msgbuf_size is calculated and set automatically at 64 bytes for every 1MB of memory. For example, in a system with 512 MB the value is 512 * 64 = 32,768, which is equivalent to 32K bytes.

For large systems with many adapters and devices, the default value might be insufficient, causing messages to be dropped from the /var/adm/messages file. For large-memory systems that have few devices, the value can be too high and you might want to reclaim the buffer space.

If your system's boot-log record is incomplete, or if you want to reduce the assigned value to reclaim the buffer space, use the following procedure to modify the value of the msgbuf_size attribute:

1. Invoke the dxkerneltuner graphical user interface from the command line.

2. Select the generic subsystem.

3. Set the new boot time value of the msgbuf_size subsystem.

4. Click the Apply button to implement the change, and exit from the dxkerneltuner utility.

You can also use the sysconfig and sysconfigdb commands to implement this change, as described in Chapter 4.