12    Administering the Basic System Event Channels

This chapter explains how system events are logged and describes how to configure the basic system event logging channnels. Information on managing log files is also included.

The following topics are discussed in this chapter:

• Section 12.1 explains your options for monitoring system events.

• Section 12.2 describes how to set up event monitoring.

• Section 12.3 describes how to recover and read event logs after the system has crashed.

• Section 12.4 explains your options for managing log files.

The Event Manager (EVM) provides an integrated approach to administering system events and errors. Refer to the EVM(5) reference page for an introduction to EVM, and see Chapter 13 for information on configuring and using EVM.

12.1    Understanding the Basic Event-Logging Facilities

The operating system uses two basic mechanisms to log system events:

• The system event-logging facility

• The binary event-logging facility

The events detected and recorded by these facilities can be reviewed using the Event Manager (EVM), DECevent, or the error report formatter, uerf.

EVM is the recommended method of administering system events. Refer to Chapter 13 for information on configuring EVM. The EVM viewer, evmviewer, provides a graphical interface for selecting, filtering, and displaying system events. See the EVM(5) and evmviewer(8) reference pages for more information.

Refer to the DECevent documentation, and the uerf(8) reference page for more information on these utilities.

Note

The uerf command utility will be retired in a future release. You should migrate your event management procedures to EVM as soon as possible.

The log files that the system and binary event-logging facilities create are protected and owned by root, and belong to the adm group. You must have the proper authority to examine the files.

The following sections describe the event-logging facilities.

12.1.1    System Event Logging

The primary event-logging facility uses the syslog function to log system-wide events in ASCII format. The syslog function uses the syslogd daemon to collect the messages that are logged by the various kernel, command, utility, and application programs. The syslogd daemon logs the messages to a local file or forwards the messages to a remote system, as specified in the /etc/syslog.conf file.

When you install the operating system, the /etc/syslog.conf file is created and specifies the default event-logging configuration. The /etc/syslog.conf file specifies the file names that are the destination for the event messages, which are in ASCII format. Section 12.2.1.1 discusses the /etc/syslog.conf file. Refer also to the syslog.conf(4) reference page.

The /etc/syslog.auth file specifies which remote hosts are allowed to forward syslog messages to the local host. For system security, only messages coming from remote hosts listed in this file are logged by the syslogd daemon. If the /etc/syslog.auth file is not present, then event forwarding from all remote hosts is enabled.

The /etc/syslog_evm.conf file specifies which syslog messages are forwarded from the syslog daemon to EVM, in the form of EVM events. Those syslog messages are posted to the EVM daemon, evmd, by syslogd if the syslogd forwarding function is turned on with the -e option. Events are posted with the EVM name of sys.unix.syslog.facility.

Refer to the syslog.auth(4) and syslog_evm.conf(4) reference pages for more information.

12.1.2    Binary Event Logging

The binary event-logging facility detects hardware and software events in the kernel and logs the detailed information in binary format records. Some events that are logged by the binary event-logging facility are also logged by the syslog function in a less detailed summary message.

The binary event-logging facility uses the binlogd daemon to collect various event-log records. The binlogd daemon logs these records to a local file or forwards the records to a remote system, as specified in the /etc/binlog.conf default configuration file, which is created when you install your system.

The event management utility of choice is EVM in place of DECevent (dia) or the uerf error log reporting facility. You can still examine the binary event log files using the dia command or the uerf command. Both commands translate the records from binary format to ASCII format.

The DECevent utility is an event translation utility that you can use to produce ASCII reports from entries in the system's binary event log files. The DECevent utility is invoked by entering the dia command at the command line. Entering the command without any options will cause DECevent to immediately access and translate the contents of the event log files, displaying the events as shown in Example 12-1. Events will scroll up the terminal screen until you press Ctrl/C.

Example 12-1:  Sample Translated Event

*************************** ENTRY   48 ********************* [1]
Logging OS                        2. <OS> [2]
System Architecture               2. Alpha
Event sequence number             2.
Timestamp of occurrence              04-MAY-1999 09:50:49 [3]
Host name                            <host name>
 
System type register      x0000001B  AlphaServer 800 or 1000A
Number of CPUs (mpnum)    x00000001
CPU logging event (mperr) x00000000
 
Event validity                    1. O/S claims event is valid [4]
Event severity                    5. Low Priority
Entry type                 250. Generic ASCII Info Message Type
 
SWI Minor class                   9. ASCII Message
SWI Minor sub class               4. Informational
 
ASCII Message              Test for EVM connection of binlogd [5]

1. The number of the event in the translated log. Note that the number might be based on the selection or filtering of events. [Return to example]

2. Identification of the operating system (<OS>), and system architecture. [Return to example]

3. The timestamp (date and system clock time) that indicates when the event occurred, and the name of the system on which it occurred (<host name>). [Return to example]

4. Information about the validity, severity, and type of event. In this case, a low priority informational message that the EVM utility was testing for a connection to the binlog event logger. [Return to example]

5. The actual message logged by the event, which might also have been displayed to a terminal or console at the time the event occurred. [Return to example]

For information about administering the DECevent utility, see the following documentation:

• DECevent Translation and Reporting Guide

• dia(8)

Section 12.2.1.3 discusses the /etc/binlog.conf file.

12.2    Configuring Event Logging

The best method of monitoring system events is to use Event Manager (EVM), as described in Chapter 13. EVM uses the default system and binary event-logging configuration to filter events. You can change this default configuration by modifying the configuration files as described in this section. For example, you can change the configuration so that only important, system-critical events are logged and informational events are ignored. You can opt to concentrate on certain subsystems, such as mail or print services, and control how and where event messages are logged.

To enable system and binary event-logging, the special files must exist and the event-logging daemons must be running. Refer to Section 12.2.3 and Section 12.2.4 for more information.

As for many other system files, /var/adm/syslog.dated and other files in /var/adm are context-dependent symbolic links (CDSLs), which facilitate joining single systems into clusters. The CDSL for the syslog directory is /var/cluster/members/member0/adm/syslog.dated. Take care not to break symbolic links when working with these files. Refer to Chapter 6 for more information on CDSLs.

12.2.1    Editing the Configuration Files

If you do not want to use the default system or binary event-logging configuration, edit the /etc/syslog.conf or /etc/binlog.conf configuration file to specify how the system should log events. In the files, you specify the facility, which is the source of a message or the part of the system that generates a message; the priority, which is the message's level of severity; and the destination for messages.

The following sections describe how to edit the configuration files.

12.2.1.1    The syslog.conf File

If you want the syslogd daemon to use a configuration file other than the default, you must specify the file name with the syslogd -f config_file command.

The following is an example of the default /etc/syslog.conf file:

#
# syslogd config file
#
# facilities: kern user mail daemon auth syslog lpr binary
# priorities: emerg alert crit err warning notice info debug
#
# [1]    [2]                              [3]
kern.debug               /var/adm/syslog.dated/kern.log
user.debug               /var/adm/syslog.dated/user.log
daemon.debug             /var/adm/syslog.dated/daemon.log
auth.crit;syslog.debug   /var/adm/syslog.dated/syslog.log
mail,lpr.debug           /var/adm/syslog.dated/misc.log
msgbuf.err               /var/adm/crash.dated/msgbuf.savecore
kern.debug               /var/adm/messages
kern.debug               /dev/console
*.emerg                  *

Each /etc/syslog.conf file entry has the following entry syntax:

1. Specifies the facility, which is the part of the system generating the message. [Return to example]

2. Specifies the severity level. The syslogd daemon logs all messages of the specified severity level plus all messages of greater severity. For example, if you specify level err, all messages of levels err, crit, alert, and emerg or panic are logged. [Return to example]

3. Specifies the destination where the messages are logged. [Return to example]

The syslogd daemon ignores blank lines and lines that begin with a number sign (#). You can specify # as the first character in a line to include comments in the /etc/syslog.conf file or to disable an entry.

The facility and severity level are separated from the destination by one or more tabs.

You can specify more than one facility and its severity level by separating them with semicolons. In the preceding example, messages from the auth facility of crit severity level and higher and messages from the syslog facility of debug severity level and higher are logged to the /var/adm/syslog.dated/syslog.log file.

You can specify more than one facility by separating them with commas. In the preceding example, messages from the mail and lpr facilities of debug severity level and higher are logged to the /var/adm/syslog.dated/misc.log file.

You can specify the following facilities:

You can specify the following severity levels, which are listed in order of highest to lowest severity:

You can specify the following message destinations:

You can specify in the /etc/syslog.conf file that the syslogd daemon create daily log files. To create daily log files, use the following syntax to specify the pathname of the message destination:

/var/adm/syslog.dated/ {file}

The file variable specifies the name of the log file, for example, mail.log or kern.log.

If you specify a /var/adm/syslog.dated/file pathname destination, each day the syslogd daemon creates a subdirectory under the /var/adm/syslog.dated directory and a log file in the subdirectory using the following syntax:

/var/adm/syslog.dated/ date / file

Where:

• The date variable specifies the day, month, and time that the log file was created.

• The file variable specifies the name of the log file you previously specified in the /etc/syslog.conf file.

The syslogd daemon automatically creates a new date directory every 24 hours, when you boot the system, or when the syslogd daemon is restarted. The current directory is a link to the latest date directory. To get the latest logs, you only need to reference the /var/adm/syslog.dated/current directory.

For example, to create a daily log file of all mail messages of level info or higher, edit the /etc/syslog.conf file and specify a line item similar to the following:

mail.info		/var/adm/syslog.dated/mail.log
 

If you specify the previous line item in the /etc/syslog.conf, the syslogd daemon could create the following daily directory and file:

/var/adm/syslog.dated/11-Jan-12:10/mail.log

12.2.1.2    Configuring syslog to use EVM

By default, syslogd is configured to forward events to EVM with the -e option. (See Section 12.2.4). You can select which syslog events are forwarded to EVM by modifying the syslog_evm.conf file. If the file does not exist, or if it exists but contains no subscription entries, no syslog messages are posted to EVM.

The default syslog_evm.conf file contains entries similar to those shown in Example 12-2, which excludes the informational file header.

Example 12-2:  Sample syslog_evm.conf File Entries

[1]    [2]
*.emerg
# above subscribes to all emergency events from syslog [3]
kern.info+ [4]
user.notice+
mail.notice+
daemon.notice+
auth.notice+
syslog.notice+

1. The first part of each line item specifies which facility (a part of the system) generated the message, such as kern for kernel. An asterisk (*) indicates that all facilities are selected. In this case, *.emerg ensures that all messages of emergency priority are forwarded to EVM.

   You can select which events are forwarded by creating an entry for a facility, or removing an existing entry. Entries are based on the keywords in the facility table in Section 12.2.1.1. [Return to example]

2. The second part of each item specifies the priority of messages, based on the keywords in the severity level table in Section 12.2.1.1. [Return to example]

3. You can add comments, preceded by a number sign (#). However, you cannot mix subscription entries and comments in the same line [Return to example]

4. The plus sign (+) appended to a priority indicates that the specified priority and all higher priority messages should be forwarded. If you want to choose individual severity levels for a facility (such as warning, critical and emergency, create a line for each priority. [Return to example]

For more information, refer to syslog_evm.conf(4) and Chapter 13.

12.2.1.3    The binlog.conf File

If you want the binlogd daemon to use a configuration file other than the default, specify the file name with the binlogd -f config_file command. Note that EVM subscribes to binlog events by default, and any configuration options you select will affect what events are available to EVM. You can filter and select binlog events using EVM utilities, as described in Chapter 13.

The following is an example of a /etc/binlog.conf file:

#
# binlogd configuration file
#
# format of a line:   event_code.priority         destination
#
# where:
# event_code - see codes in binlog.h and man page, * = all events
# priority   - severe, high, low, * = all priorities
# destination - local file pathname or remote system hostname
#
#
*.*			/usr/adm/binary.errlog
dumpfile		/usr/adm/crash/binlogdumpfile
102.high		/usr/adm/disk.errlog
[1]    [2]                     [3]

Each entry in the /etc/binlog.conf file, except the dumpfile event class entry, contains three fields:

1. Specifies the event class code that indicates the part of the system generating the event. [Return to example]

2. Specifies the severity level of the event. Do not specify a severity level if you specify dumpfile for an event class. [Return to example]

3. Specifies the destination where the binary event records are logged. [Return to example]

The binlogd daemon ignores blank lines and lines that begin with a number sign (#). You can specify # as the first character in a line to include comments in the file or to disable an entry.

The event class and severity level are separated from the destination by one or more tabs.

You can specify the following event class codes:

You can specify the following severity levels:

You can specify the following destinations:

12.2.2    syslog Security and Remote Messages

Unless the domain host name of a remote host is entered in the local file, /etc/syslog.auth, the local system will not log any syslog messages from that remote host. If you intend to use the secure version of syslogd on a system, and you have configured or intend to configure other hosts to forward syslog messages to the system, complete the following steps:

1. Use the su command to become the superuser (root).

2. Create the file /etc/syslog.auth using a text editor. This file must be owned by root and have permissions set to 0600.

3. Add the names of any remote hosts that are allowed to forward syslog messages to the local system. Host names must meet the following criteria:

   • Each remote host name should appear in a separate line in /etc/syslog.auth. (Lines beginning with the # character are comments and are ignored.)

   • A host name must be a complete domain name such as trout.fin.huk.com.

   • If a domain host name is given, it must either appear in the local /etc/hosts file or the local system must resolve it through a name server (such as BIND).

   • A host name can have at most as many characters as defined by the MAXHOSTNAMELEN constant in /sys/include/sys/param.h, although each line in the /etc/syslog.auth file is limited to 512 characters.

   Note that the syslogd daemon's forwarding option (-e) is used by default, so remote messages logged to syslog are forwarded to EVM.

A plus character (+) by itself allows event forwarding from all hosts. A host name can also be preceded by a minus character (-) to expressly prohibit that host from forwarding events. If the file /etc/syslog.auth is not present on the system, then forwarding from all hosts is enabled.

Refer to the syslog.auth(4) and syslogd(8) reference pages for additional information.

12.2.3    Creating the Special Files

The syslogd daemon cannot log kernel messages unless the /dev/klog character special file exists. If the /dev/klog file does not exist, create it as follows:

/dev/MAKEDEV /dev/klog

Also, the binlogd daemon cannot log local system events unless the /dev/kbinlog character special file exists. If the /dev/kbinlog file does not exist, create it as follows:

/dev/MAKEDEV /dev/kbinlog

Refer to the MAKEDEV(8) reference page for more information.

12.2.4    Starting and Stopping the Event-Logging Daemons

The syslogd and binlogd daemons are automatically started by the init program during system startup. However, you must ensure that the daemons are started. You can also specify options with the command that starts the daemons. Refer to the init(8) reference page for more information.

12.2.4.1    The syslogd Daemon

You must ensure that the syslogd daemon is started by the init program. If the syslogd daemon is not started or if you want to specify options with the command that starts the syslogd daemon, you must edit the /sbin/init.d/syslog file and either include or modify the syslogd command line. Note that you can also invoke the command manually.

The command that starts the syslogd daemon has the following syntax:

/usr/sbin/syslogd [-d] [-e] [-s] [-m mark_interval] [-p path] [-f config_file] [-b rcvbufsz]

By default the initialization of the daemon uses only one command option, the -e option, which configures the daemon to automatically forward events to EVM. You can verify the current syslogd configuration using the ps command as follows:

# /sbin/ps agx | grep syslogd

261 ??    S    0:00:10  usr/sbin/syslogd -e

Refer to the syslogd(8) reference page for information on the command options.

Note

You must ensure that the /var/adm directory is mounted, or the syslogd daemon will not work correctly.

The syslogd daemon reads messages from the following:

• The domain socket /dev/log file, which is automatically created by the syslogd daemon

• An Internet domain socket, which is specified in the /etc/services file

• The special file /dev/klog, which logs only kernel messages

Messages from other programs use the openlog, syslog, and closelog calls.

When the syslogd daemon is started, it creates the /var/run/syslog.pid file, where the syslogd daemon stores its process identification number. Use the process identification number to stop the syslogd daemon before you shut down the system.

During normal system operation, the syslogd daemon is called if data is put in the kernel syslog message buffer, located in physical memory. The syslogd daemon reads the /dev/klog file and gets a copy of the kernel syslog message buffer. The syslogd daemon starts at the beginning of the buffer and sequentially processes each message that it finds. Each message is prefixed by facility and priority codes, which are the same as those specified in the /etc/syslog.conf file. The syslogd daemon then sends the messages to the destinations specified in the file.

To stop the syslogd event-logging daemon, use the following command:

# kill `cat /var/run/syslog.pid` 

Using the following command, you can apply any changes to the /etc/syslog.conf configuration file without shutting down the system:

# kill -HUP `cat /var/run/syslog.pid` 

12.2.4.2    The binlogd Daemon

You must ensure that the init program starts the binlogd daemon. If the binlogd daemon does not start, or if you want to specify options with the command that starts the binlogd daemon, you must edit the /sbin/init.d/binlog file and either include or modify the binlogd command line. Note that you can also invoke the command manually.

The command that starts the binlogd daemon has the following syntax:

/usr/sbin/binlogd [-d] [-fconfig_file]

Refer to the binlogd(8) reference page for information on command options.

The binlogd daemon reads binary event records from the following:

• An Internet domain socket (binlogd, 706/udp), which is specified in the /etc/services file

• The /dev/kbinlog special file

When the binlogd daemon starts, it creates the /var/run/binlogd.pid file, where the binlogd daemon stores its process identification number. Use the process identification number to stop or reconfigure the binlogd daemon.

During normal system operation, the binlogd daemon is called if data is put into the kernel's binary event-log buffer or if data is received on the Internet domain socket. The binlogd daemon then reads the data from the /dev/kbinlog special file or from the socket. Each record contains an event class code and a severity level code. The binlogd daemon processes each binary event record and logs it to the destination specified in the /etc/binlog.conf file.

To stop the binlogd daemon, use the following command:

# kill `cat /var/run/binlogd.pid` 

Using the following command, apply any changes to the /etc/binlog.conf configuration file without shutting down the system:

# kill -HUP `cat /var/run/binlogd.pid` 

12.2.5    Configuring the Kernel Binary Event Logger

To configure the kernel binary event logger, modify the default keywords and rebuild the kernel. You can

• Scale the size of the kernel binary event-log buffer to meet your system needs.

• Enable and disable the binary event logger and the logging of kernel ASCII messages into the binary event log.

The /sys/data/binlog_data.c file defines the binary event-logger configuration. The default configuration specifies a buffer size of 24K bytes, enables binary event logging, and disables the logging of kernel ASCII messages. You can modify the configuration by changing the values of the binlog_bufsize and binlog_status keywords in the file.

The binlog_bufsize keyword specifies the size of the kernel buffer that the binary event logger uses. The size of the buffer can be between 8 kilobytes (8192 bytes) and 1MB (1048576 bytes). Small system configurations, such as workstations, can use a small buffer. Large server systems that use many disks might need a large buffer.

The binlog_status keyword specifies the behavior of the binary event logger. You can specify the following values for the binlog_status keyword:

0 (zero)

Disables the binary event logger.

BINLOG_ON

Enables the binary event logger.

BINLOG_ASCIION

Enables the logging of kernel ASCII messages into the binary event log if the binary event logger is enabled. This value must be specified with the BINLOG_ON value as follows: int binlog_status = BINLOG_ON | BINLOG_ASCII;

After you modify the /sys/data/binlog_data.c file, you must rebuild and boot the new kernel.

12.3    Recovering Event Logs After a System Crash

You can recover unprocessed messages and binary event-log records from a system crash when you reboot the system.

The msgbuf.err entry in the /etc/syslog.conf file specifies the destination of the kernel syslog message buffer msgbuf that is recovered from the dump file. The default /etc/syslog.conf file entry for the kernel syslog message buffer file is as follows:

msgbuf.err            /var/adm/crash/msgbuf.savecore
 

The dumpfile entry in the /etc/binlog.conf file specifies the file name destination for the kernel binary event-log buffer that is recovered from the dump file. The default /etc/binlog.conf file entry for the kernel binary event-log buffer file is as follows:

dumpfile              /usr/adm/crash/binlogdumpfile
 

If a crash occurs, the syslogd and binlogd daemons cannot read the /dev/klog and /dev/kbinlog special files and process the messages and binary event records. When you reboot the system, the savecore command runs and, if a dump file exists, recovers the kernel syslog message and binary event-log buffers from the dump file. After savecore runs, the syslogd and binlogd daemons are started.

The syslogd daemon reads the syslog message buffer file, checks that its data is valid, and then processes it in the same way that it normally processes data from the /dev/klog file, using the information in the /etc/syslog.conf file.

The binlogd daemon reads the binary event-log buffer file, checks that its data is valid, and then processes the file in the same way that it processes data from the /dev/kbinlog special file, using the information in the /etc/binlog.conf file.

After the syslogd and binlogd daemons are finished with the buffer files, the files are deleted.

12.4    Maintaining Log Files

On a well maintained system, the size of the various log files should not become a problem as you should carefully select the events that you want to log, monitoring the logs for error conditions that result in many postings, and regularly archiving your important event logs. The /var/spool/cron/crontabs/root file contains some default entries for managing some log files. as follows:

0 2 * * 0 /usr/lbin/logclean /var/adm/cron/log > /dev/null
0 2 * * 0 /usr/lbin/logclean /var/adm/wtmp > /dev/null
0 2 * * 0 /usr/lbin/logclean /var/adm/messages > /dev/null

You can use the cron daemon to specify that other log files be deleted. However, you should also take care that important log files are stored or archived according to your local site requirements.

The following is an example of a crontab file entry to clean up the older logs in /var/adm/syslog.dated

5 1 * * * find /var/adm/syslog.dated -type d -mtime +5 -exec rm -rf '{}' \;
 

This file entry causes all directories under /var/adm/syslog.dated that were modified more than 5 days ago to be deleted, along with their contents, every day at 1:05. Refer to the crontab(1) reference page for more information.

12.4.1    Startup Log Messages in /var/adm/messages

The size of the message buffer used to store boot-log messages is controlled by the msgbuf_size kernel attribute. The minimum default value for this attribute is 8K, for systems with up to 128 MB of physical memory. For systems with greater than 128 MB of physical memory, the value of msgbuf_size is calculated and set automatically at 64 bytes for every 1MB of memory. For example, in a system with 512 MB the value is 512 * 64 = 32,768, which is equivalent to 32KB.

For large systems with many adapters and devices, the default value might be insufficient, causing messages to be dropped from the /var/adm/messages file. For large-memory systems that have few devices, the value can be too high and you might want to reclaim the buffer space.

If your system's boot log record is incomplete, or if you want to reduce the assigned value to reclaim the buffer space, use the following procedure to modify the value of the msgbuf_size attribute:

1. Invoke the dxkerneltuner graphical user interface from the command line.

2. Select the generic subsystem.

3. Set the new Boot Time Value of the msgbuf_size subsystem.

4. Press the Apply button to implement the change, and exit from the dxkerneltuner utility.

You can also use the sysconfig and sysconfigdb commands to implement this change, as described in Chapter 4.

Note

In a cluster, the clu_create command and clu_add_member script invokes the sysconfigdb command to set msgbuf_size to 16K, overriding the automatic scaling algorithm used in standalone systems. Consult the Cluster Administration guide for an appropriate cluster setting for msgbuf_size.