13    Event Management Using EVM

The Event Manager, EVM, is a comprehensive event management system. In addition to providing traditional event handling facilities, EVM unifies its own events and events from other channels to provide a single source of information, simplifying the task of monitoring system activity. EVM includes a graphical event viewer and a full set of command-line tools. It is integrated into the SysMan Menu application suite and the SysMan Station.

The following topics are covered in this chapter:

13.1    EVM Overview

A critical part of a UNIX system administrator's job is to monitor the state of the system, and to be aware and ready to take action when certain unusual conditions occur, such as when a disk fills or a processor begins reporting hardware errors. It is also important to verify that certain routine tasks run successfully each day, and to review certain system configuration values. Such conditions or task completions are described as system events.

An event is only an indication that something interesting has occurred - an action has been taken, some condition has been met, or it is time to confirm that an application is still operational. A particular event might be interesting to the administrator or to some other class of system user. If it is a system event it could also be significant to other system entities, such as:

Entities interested in events can either be part of the local system or of a remote system.

When a system component has something interesting to report, it makes the information available through an event channel. An event channel is any facility used to publish or retrieve event information, and can be a simple log file, an event management system, or a program that can be run to obtain a snapshot of status information. An event management system is an active event channel and as such, it provides services for distributing, storing, and retrieving event information.

UNIX supports a number of channels through which system components can report event and status information, and it is the administrator's responsibility to regularly check the information available at each channel to be sure that the system is operating normally. The UNIX system logger, syslog, and the binary error logger, binlog, are familiar examples of event management systems. They provide simple event distribution facilities for other components to use, and their daemons actively manage the event information they receive. By contrast, the cron daemon's log file, /var/adm/cron/log, is an example of a passive event channel. The cron daemon writes new event information to the end of its file, and takes no special action to notify interested entities when it does so.

Apart from syslog and binlog, there are a number of other logs in various locations around the system, each of which might require some level of monitoring. The Event Manager, EVM, provides a single point of focus for multiple event channels by combining events from all sources into a single event stream. The system administrator can either monitor the combined stream in real time or view historical events retrieved from storage. EVM's viewing facilities include a graphical event viewer, which is integrated with the SysMan Menu and SysMan Station, and a full set of command-line utilities, which allow events to be filtered, sorted, and formatted in a variety of ways. EVM can also be configured to perform automatic notification of selected conditions.

Rather than replacing the familiar event channels, such as syslog and binlog, EVM encapsulates them. These channels remain in place, and continue to handle the same set of events as they always did. However, EVM makes the existing channels much more accessible.

EVM provides you with the following features:

13.1.1    Understanding EVM Events

An EVM event is a binary package of data that can contain any or all of a set of standard data items, including a name, a timestamp and information about the poster, and variable data, which is named and supplied by the poster. For example, an event reporting the failure of a device might hold variables containing the pathname and type of the device. Events are typically created and posted by an EVM posting client, and distributed to other clients by the EVM daemon. A process might also create an event and store it in a file, or send it directly to another process, rather than post it. Other processes can then read or receive the event and extract the information it contains.

Although the EVM logger captures posted events and stores them in a system log file, you can easily capture your own set of events using the logger or the evmwatch monitoring utility and store them in your own file for later analysis. Figure 13-1 shows a graphical representation of an event.

Figure 13-1:  Event Model

In Figure 13-1, the Event Contents box shows some items that might be included in the event, such as the process identifier (PID) and the name of the host system on which the event was generated. The Event Actions box shows some of the actions that can be performed on any event.

Because an event is a package of binary data, you cannot view it directly with a text viewer. However, EVM includes command-line utilities which understand the format of the event, and which can be used to perform basic operations at the UNIX command prompt or in shell scripts. The EVM commands can be used to retrieve events from storage, sort them into any useful order, format them for display, watch for new events being posted, or post new events.

The EVM utilities are designed to be used together in pipelines. For example, you might pipe a set of events from a file into the evmsort utility, and pipe the output into the formatting utility, evmshow, then pipe the output of that command into more, or redirect it to a file.

Once the event file has been converted to text form, you can use other standard utilities to analyze it. For example, you might display just the event names, and then pipe the display into sort -u and wc -l to determine how many different types of events are in the file.

13.1.2    EVM Components

This section describes how the different parts of EVM interact. It also describes the system files used to run EVM and any files created by EVM during normal operations. A model of the system is shown in Figure 13-2.

Figure 13-2:  EVM Component Model

In Figure 13-2, client components involved in posting are shown at the left, EVM system components in the center, and client components involved in subscribing and retrieval are at the right. Active event channels are those that post directly to EVM. Passive event channels do not post events and must be polled for information. These channels are depicted by the log files handled by the monitor scripts.

The primary component of EVM is the evmd daemon, which is initialized when the system is booted to run level two. For event management to function during system startup, the initialization of the daemon and its child processes is synchronized as follows:

The EVM logger program, evmlogger, runs as a resident process. It is configured to subscribe to a selected set of events, and to store them in managed log files for later retrieval. The logger is also configured by default to:

The resident channel manager process, evmchmgr, is configured to run periodic channel-monitoring scripts, which might post events. It also runs the daily log cleanup functions.

The get server process, evmget_srv, is a transient (demand) process that executes event retrieval scripts for the various event channels. It is run by the daemon whenever a user runs the evmget command.

Entities on the left side of the model create posting connections to the daemon in order to post events. As it receives events from the posters, the daemon merges them with corresponding event templates from its template database, and distributes them to its listening clients.

On the right side of the model, evmwatch and other application programs that need to receive event information as it happens create listening connections to the daemon, and pass filter strings to it to specify their event subscriptions. The evmget command, which a user can run to retrieve historical event information from log files, creates a service connection, and passes a filter string to specify the set of events to be retrieved. The daemon then runs an instance of the get server to handle the request. The e-mail and pager actions are examples of forwarding commands, which can be executed by the logger in response to the occurrence of certain events.

13.1.2.1    EVM Command-Line Utilities

EVM provides a number of command-line utilities both for administering the EVM system itself and for use in posting or obtaining events. Table 13-1 describes the general user commands. Detailed information is available from the reference pages.

Table 13-1:  EVM Command-Line Utilities

Command Description

evmget

Retrieves stored events from a configured set of log files and event channels, using channel-specific retrieval functions.

evmpost

Accepts a file or stream of text event sources and posts them to the EVM daemon for distribution

evmshow

Accepts one or more EVM events and outputs them in the requested format.

evmsort

Reads a stream of events and sorts them according to supplied criteria.

evmwatch

Subscribes to events specified and outputs them as they arrive.

Table 13-2 lists the administrative commands which are normally invoked during system initialization. The individual command reference pages discuss other conditions under which the command should be used.

Table 13-2:  EVM Administrative Utilities

Command Description

evmchmgr

The EVM channel manager is started automatically by the EVM daemon and should not be run manually. It executes the periodic functions defined for any channel.

evmd

The EVM daemon receives events from posting clients and distributes them to subscribing clients that have indicated that they are interested in receiving them. The daemon is a critical system facility which is started automatically at system startup time and should never be terminated.

evmlogger

The EVM logger is started automatically by the EVM daemon. This command receives events from the daemon and writes them to each of the logs whose filter string they match. This command also serves as an event forwarding agent that can be used to take action when required.

evmreload

This command posts control events, which instruct EVM components to reload their configuration files.

When you modify the configuration file you must use this command to load the new configuration into EVM.

evmstart

This command starts the EVM daemon. It is intended for use by the system startup scripts, but it also can be used to restart EVM if it has been terminated for any reason.

evmstop

This command stops the EVM daemon, preventing events from being posted or subscribed to. It is intended for use by the system shutdown scripts. Because EVM is required to be running for many system functions to operate correctly, this command should not be used under normal circumstances.

13.1.2.2    EVM Application Programming Interface

The EVM API library, libevm.so, contains an extensive range of event management functions. This library enables programmers to design programs that interface with EVM. The API functions enable programs to post events, send requests and notifications to the daemon, or receive responses and information from the daemon. Use of these interfaces is described in the Programmer's Guide, and the EVM(5) reference page lists all the available API functions.

13.1.2.3    EVM System Files

The following system files are created or used by EVM:

13.1.3    Related Utilities

The following existing subsystems or optional components also provide event handling capabilities:

13.2    Administering EVM

The role of the administrator in running EVM involves the following principal activities:

For information on using EVM, see Section 13.3.

13.2.1    Configuring EVM

Configuring EVM means establishing and maintaining its configurable resident components, which are:

Each component recognizes a configuration file that directs its operations.

When you install the operating system, EVM is automatically configured to run with default configuration options that should be suitable for most installations. However, you should change the configuration for your system if, for example, an event channel is to be added or modified, the log file archive and expiration options need to be changed, or an alternate logging directory is established.

EVM is preconfigured to use both DECevent and Compaq Analyze to translate binary logger (binlogd) events.

Whenever the configuration changes because a new file is loaded, or because a change is made, the configuration should be reestablished by running the evmreload command. See the evmreload(8) reference page for information on this command.

Configuration files are described in the following sections and in the corresponding reference pages.

13.2.1.1    EVM Daemon Configuration

The EVM daemon reads its configuration file, /etc/evmdaemon.conf, at system startup and whenever you issue a reload request using the evmreload command. For a complete description of the contents and syntax of the configuration file, see the evmdaemon.conf(4) reference page. Example 13-1 shows some sample entries in a daemon configuration file.

Example 13-1:  Sample EVM Daemon Configuration File Entries

# Event template directory:
sourcedir "/usr/share/evm/templates" [1]
 
# Start the EVM Logger                     [2]
start_sync "/usr/sbin/evmlogger -o /var/run/evmlogger.info \
                        -l /var/evm/adm/logfiles/evmlogger.log"
# Start the EVM Channel Manager            [2]
start_sync "/usr/sbin/evmchmgr -l /var/evm/adm/logfiles/evmchmgr.log"
 
# Event retrieval service definition:
service                                    [3]
   {
     name          event_get
     command       "/usr/sbin/evmget_srv"
   }
 
# Set up an activity monitor.
activity_monitor                           [4]
   {
     name            event_count
     period          10
     threshold       500
     holdoff         240
   }
 
remote_connection false                    [5]
 
 

  1. This statement identifies the top of the directory hierarchy for all event template files. [Return to example]

  2. These commands start the evmlogger and the evmchmgr components as synchronized clients, ensuring that both clients will have completed their subscription requests before the daemon accepts any events from posting clients. The command-line options for these commands define the clients' log files and, in the case of the logger, an output file that will be used to make operational details available to the evmlog event channel functions. [Return to example]

  3. These statements define the event_get event retrieval service, which is used by the evmget command to retrieve events. [Return to example]

  4. These statements define an activity monitor. If 500 or more events are received during any ten minute period, the daemon posts a high-priority event to alert the system administrator. Monitoring (counting) of events is suspended for the hold-off period of four hours (240 minutes). [Return to example]

  5. This line sets the remote_connection to false to disable connection to this system by remote EVM clients. Refer to the evmdaemon.conf(4) reference page and to Section 13.2.2 for information about the security implications of changing this value. [Return to example]

If you make any changes to the configuration file you must run evmreload(8) to make the changes known to the EVM daemon.

13.2.1.2    EVM Channel Configuration

An event channel is a source of event information. The channel configuration file, /etc/evmchannel.conf, defines a set of event channels and the functions that operate on them, for use by the channel manager, by the evmshowcommand, and by the event retrieval process. For a complete description of the contents and syntax of the channel configuration file, see the evmchannel.conf(4) reference page. Example 13-2 shows sample channel configuration entries.

Example 13-2:  Sample EVM Channel Configuration File

# Global path for channel functions
path   /usr/share/evm/channels [1]
 
# Time-of-day at which daily cleanup function will run
cleanup_time 02:00:00                              [2]
 
# ==================================
# Event channel:  EVM log
# ==================================
channel
  {[3]
    name        evmlog[4]
    path        /usr/share/evm/channels/evmlog
    events      *[5]
    fn_get      "evmlog_get"
    fn_details  "evmlog_details"
    fn_explain  "evmlog_explain"
    fn_monitor  "evmlog_mon"
    fn_cleanup  "evmlog_cleanup 7 31"[6]
    mon_period  15:00  # Monitor every 15 minutes[7]
   }
 

  1. This line declares the default path for all channel functions. In this case it is the /usr/share/evm/channels directory. This path is prefixed to the names of any channel functions defined in this file that do not begin with a slash (/) character, unless the channel group supplies its own path value. [Return to example]

  2. This line defines a daily 2:00am cleanup for all channels. [Return to example]

  3. This line specifies a configuration group that defines an event channel. [Return to example]

  4. This line specifies that the name of the channel is evmlog. [Return to example]

  5. In this line, the asterisk (*) indicates that the channel provides default event handling, meaning that its functions will be invoked to provide details and explanations for any events whose names do not match the events value of any other channel. [Return to example]

  6. This line defines the arguments to this function that specify when the cleanup occurs. In this example, log files older than seven days are compressed and log files older than thirty-one days are deleted. [Return to example]

  7. This line sets the monitoring period, causing the /usr/share/evm/channels/evmlog/evmlog_mon function to be invoked every 15 minutes. [Return to example]

13.2.1.3    EVM Logger Configuration

The EVM logger handles storage and forwarding of events, according to entries in its configuration file, /etc/evmlogger.conf. For a complete description of the contents and syntax of this file, see the evmlogger.conf(4) reference page. Example 13-3 shows sample entries in a logger configuration file. An example of possible customization of the logger would be to direct output to a terminal in addition to a log file.

Example 13-3:  Sample EVM Logger Configuration File Entries

# Main log file: eventlog
{                                               [1]
name        evmlog                              [2]
logfile     /var/evm/evmlog/evmlog.dated        [3]
type        binary                              [4]
maxsize     512  # Kbytes                       [5]
 
# Uncomment the following "alternate" line and set the
# logfile path to specify an alternate logfile in case
# of write failures. The path must specify an existing
# directory.
# alternate  /your_alternate_fs/evmlog/evmlog.dated  [6]
 
# Log all events with priority >= 200, except binlog events:
filter  "[prio >= 200] & (! [name @SYS_VP@.binlog])" [7]
 
# Suppress logging of duplicate events:
suppress                                             [8]
{   filter      "[name *]"
    period      30        # minutes 
    threshold   3         # No. of duplicates before suppression
    }
}   
# Forward details of high-priority events to root:
   forward {                                         [9]
   name      priority_alert                         [10]
# Don't forward mail events through mail
   filter  "[prio >= 600] & ![name @SYS_VP@.syslog.mail]" [11]
 
   suppress                                               [12]
{  filter  "[name *]"
   period  120         # minutes
   threshold   1       # No. of duplicates before suppression
   }      
 
# This evmshow command writes a subject line as the
# first line of output, followed by a detailed display
# of the contents of the event. The resulting message is
# distributed by mail(1).
command "evmshow -d -t 'Subject: EVM ALERT [@priority]: @@' | \
mail root"                                                [13]
   }

  1. This line specifies the configuration group that defines an event log. [Return to example]

  2. This line specifies that the evmlog event channel uses the name to get configuration information about the log. [Return to example]

  3. This line specifies that the log files are stored in the directory /var/evm/evmlog directory. Each day, when the log for that day is first written, the dated suffix is replaced by the date in the format yyyymmdd. [Return to example]

  4. This line specifies that the type of events written to this log are binary EVM events, rather than formatted (ASCII text) events. [Return to example]

  5. This line specifies the maximum size of the log file in kilobytes (KB). In this case, if the size of the current log file exceeds 512 KB the logger will close it and begin a new log file, with a sequentially numbered suffix (for example, _2) appended to the file name. [Return to example]

  6. If this line is uncommented and the sample path is replaced by the pathname of an existing write-enabled directory, an alternate log file is opened in this directory if the primary directory becomes write-disabled. [Return to example]

  7. This line establishes the filtering conditions for events, determining which events are logged by this event log. Refer to EvmFilter(5) for details of EVM filter syntax. [Return to example]

  8. These statements define the suppression parameters for this event log. In this case, suppression of a particular event begins if three or more duplicate events are received within 30 minutes. Suppression of duplicate events saves space in the log file. Refer to the evmlogger.conf(4) reference page for a detailed description of event suppression. [Return to example]

  9. This line establishes conditions for forwarding events to the root user. An event forwarder executes a specified command string when selected events occur. It is useful for notifying the system administrator when a significant error occurs. [Return to example]

  10. In this line, name identifies the forwarder. [Return to example]

  11. This line establishes filtering for the events. As with an event log definition, the filter string specifies the set of events that are handled by this forwarder. To prevent an event loop from occurring if the mailer posts high-priority events, mail events are explicitly excluded from this forwarder. [Return to example]

  12. These lines suppress multiple forwarding of events. The suppression mechanism for a forwarder is similar to that for an event log. Here, the purpose is to prevent the command from being sent multiple times in a short period due to the same event being posted repeatedly. In the example, a particular event is only forwarded once every two hours. [Return to example]

  13. This line defines the command that execute when an event is handled by the forwarder. The event is piped into the command's stdin stream. The result of this command is shown in the comments preceding the command line. [Return to example]

If you make any changes to the logger configuration file you must run the evmreloadcommand to make the changes known to the logger.

13.2.2    Security Considerations

Security is an important consideration when dealing with events, for the following reasons:

Traditionally, security has been maintained over event information by restricting read access to log files and limiting certain posting operations to the rootuser. Because the EVM daemon and event retrieval facilities provide alternate means of access to all events, both as they are posted and after they are logged, the daemons also provide a way to limit access, so that events are seen only by authorized users. You can enable access control by providing authorization facilities and using authentication techniques. You must also be careful to avoid compromising security when writing executable functions to be used in the EVM environment. Refer to the Programmer's Guide for more information about protecting channel functions.

13.2.2.1    User Authentication

The EVM daemon authenticates the identities of all local system users before accepting any connection request. There is currently no authentication of remote users. See Section 13.2.2.3 for information about remote connections.

13.2.2.2    User Authorization

Access to events is controlled by the EVM authorization file, /etc/evm.auth.

Users can be authorized individually and by group to do the following:

By default, all events are protected. Event rights are granted by supplying, for each event class, a list of users who have the specified right or who are explicitly denied rights. An indicator (+ or -) that is not followed by a user list implicitly grants or denies the right to all users. The root user has implicit posting and access rights to all events unless explicitly denied them. Example 13-4 shows sample entries in an authorization file. See the evm.auth(4) reference page for additional details.

Example 13-4:  Sample EVM Authorization File Entries

# ===================
#      EVENTS
# ===================
 
event_rights {                             [1]
   class        @SYS_VP@.evm.control   # EVM control events
   post         root
   access       +
}
 
event_rights {                             [2]
   class        @SYS_VP@.evm.msg.admin  # EVM admin message
   post         root
   access       "root, group=adm"
}
 
event_rights {                             [3]
   class        @SYS_VP@.evm.msg.user         # EVM user message
   post         +
   access       +
}
 
 
# ===================
#      SERVICES
# ===================
 
service_rights {                           [4]
   service      event_get
   execute      +
}

  1. The class of events having names beginning sys.unix.evm.control can be posted only by root, but accessed by all users. The entry @SYS_VP@ is a macro which will be replaced with sys.unix when the file is read. [Return to example]

  2. The class of events having names beginning sys.unix.evm.msg.admin can only be posted by root, but might be accessed by root or anybody in the admin group. [Return to example]

  3. The class of events having names beginning sys.unix.evm.msg.user can be posted or accessed by all users. [Return to example]

  4. All users can execute the event_get service. [Return to example]

If you make any changes to the authorization file you must run the evmreload command to make the changes known to the EVM daemon.

13.2.2.3    Remote Access

Although EVM provides facilities that allow clients to connect to an EVM daemon running on a remote system, it does not currently provide a means of authenticating these clients, and consequently cannot make restricted event information or services available to them. By default, remote access is disabled in the daemon's configuration file, /etc/evmdaemon.conf, and you should only enable it if your system is running in a fully secure environment.

If remote access is enabled, remote clients can connect without authentication, although they are only permitted to take actions that are completely unrestricted in the remote system.

13.2.3    Managing Log Files

The EVM channel manager, evmchmgr, provides log management capability through the channel fn_cleanup function, which can be defined for any channel through the channel configuration file, evmchannel.conf. See Section 13.2.1.2 for additional information on this file.

By default, channel cleanup functions are run when EVM is started, and then at 2:00 A.M. every day. You can change the time of day by editing the cleanup_time value in the channel configuration file. When a cleanup is scheduled, the channel manager scans the event channel list, and executes the fn_cleanup command for each channel identified in the file.

The evmlog cleanup function, evmlog_cleanup, takes two arguments as follows:

The function uses the find utility to locate and compress (zip) all logs older than the archive period, and to delete any archived files older than the delete period. You can change the period values by editing the function definition in the channel configuration file. See Section 13.2.1.2 for a description of this file. Setting either of these values to zero disables the corresponding function. Logs are compressed for archive using the gzip command. See the gzip(1) reference page.

The default channel configuration also provides a similar cleanup function for the SysMan Station message log files, through the misclog event channel. The syslog and binary error log channels can be managed through crontab entries. Because the binary error log file is typically not managed on a daily basis, the channel's cleanup function posts a daily EVM event reporting the size of the log. If the log is growing significantly you should review the log entries and, if necessary, use the cleanup options in binlogd to initiate a cleanup. See the binlogd(8) reference page for more information.

Note that evmget does not retrieve evmlog events stored in archived (zipped) logs. To retrieve events from archived logs you must first uncompress them with the gunzip command. See the gunzip(1) reference page for information on unzipping archive files.

13.2.4    Event Templates

An event template is a centrally held description of an event. The template is used for the following purposes:

Event template definitions are held in template files, which are text files stored in directories subordinate to (or linked to) the system template directory, /usr/share/evm/templates. Place any installation-specific or third-party event templates in the local directory /var/evm/adm/templates, which is linked into the system directory. Each time an event is posted, the EVM daemon looks in its internal template database for a template event whose name matches the posted event. It then retrieves any centralized data items held in the template event, and combines them with the items the program supplied when it posted the event, to yield a merged event for distribution to subscribers.

Template files require specific ownership and permissions to be recognized by EVM. See the evmtemplate(4) reference page for details. Refer to the Programmer's Guide for details of installation of new event template files.

13.2.5    Installing New EVM Clients

You can add new events to the event set as new applications are installed and as new administrative scripts are developed to use the facilities. As events are added it might be necessary to modify EVM configuration and authorization files, and to add new templates. See Section 13.2.1 for a discussion of the various configuration files. See Section 13.2.2.2 for information on changing the authorization for new clients.

To add new event templates, create new template files as described in Section 13.2.4 and add them to the /var/evm/adm/templates directory or to a subdirectory. Refer to the evmtemplate(4) reference page for details of the required ownership and permissions of a template file.

Refer to the Programmer's Guide for additional information about developing EVM client applications.

13.2.6    Configuring binlog Event Translation Utilities

DECevent is a rules-based translation and reporting utility that provides event translation for binary error log events. EVM uses DECevent's translation facility, dia, to translate binary error log events into human-readable form. Some newer processors do not support DECevent and can only support Compaq Analyze.

Compaq Analyze is a rules-based hardware fault management diagnostic tool that provides error event analysis and translation. The multi-event correlation analysis feature of Compaq Analyze provides the capability to analyze events stored in the system's event log file and to analyze events from other systems, including other operating systems such as OpenVMS and Windows NT.

Although the EVM infrastructure directly recognizes events only its own EVM format, an event posted through another channel, such as binlogd, can be passed to EVM by creating a wrapper EVM event and inserting the lower-level event into the EVM event as variable data. The whole package can then be passed to EVM without EVM having any knowledge of the content or format of the variable.

The binary logger daemon, binlogd, uses this approach to make its own events available through EVM. When binlogd receives an event from the operating system it first stores the event in its own log file and distributes it to its own clients, in its traditional manner. It then creates an EVM event named sys.unix.binlog, and adds a variable called binlog_event, containing the binlogd event data. Finally, it posts the package to the EVM daemon for further distribution. The EVM daemon deals with the package as it would any EVM event, and has no direct knowledge of the contents of the binlog_event variable.

When you request a detailed view of an event, either by running the evmshow -d command from the command line or by selecting the Details... button in the event viewer's event summary window, EVM runs the detailed display program defined for the event in the /etc/evmchannel.conf file. The resulting display always begins with an explanation of the event and a detailed view of its contents. If the event is a binlogd event, this display is followed by a translation of the contents of the binlog_event variable. This translation can be useful if you are troubleshooting a system problem. Example 13-5 shows a detailed display of a binlogd event, including a DECevent translation:

Example 13-5:  A binlogd Event Showing the DECevent Translation

=================== Binary Error Log event ====================
EVM event name: sys.unix.binlog.op.shutdown
 Binary error log events are posted through the binlogd
 daemon, and stored in the binary error log file,
 /var/adm/binary.errlog. This event is posted by the shutdown(8),
 halt(8), and reboot(8) commands when the system is being shut
 down. The message includes details of the user who initiated
 the shutdown.
===============================================================
Formatted Message:
    System shutdown msg:  System rebooted by root:
Event Data Items:
    Event Name        : sys.unix.binlog.op.shutdown
    Priority          : 200
    Timestamp         : 26-Jan-2000 20:54:36
    Host IP address   : 16.69.224.11
    Host Name         : kopper
    Format            : System shutdown msg: $message
    Reference         : cat:evmexp.cat:300
Variable Items:
    subid_class = 301
    message = "System rebooted by root:"
    binlog_event = [OPAQUE VALUE: 96 bytes]
============================ Translation =====================
Logging OS                       2. Digital UNIX 
System Architecture              2. Alpha 
Event sequence number          752. 
Timestamp of occurrence             26-JAN-2000 20:54:36   
Host name                           kopper 
System type register      x0000000F  AlphaStation 600 or 500
Number of CPUs (mpnum)    x00000001 
CPU logging event (mperr) x00000000 
Event validity                   1. O/S claims event is valid
Event severity                   5. Low Priority 
Entry type                     301. Shutdown ASCII Message Type
SWI Minor class                  9. ASCII Message 
SWI Minor sub class              2. Shutdown 
ASCII Message                    System rebooted by root:
===============================================================

EVM obtains the binlogd event translation by passing the event to either DECevent or Compaq Analyze. If neither of these programs is available, or if the translation attempt fails for any reason, the translation area of the display shows a message indicating the failure.

Several factors govern the type of binlogd event translation that is available on any given system:

If your system will use DECevent for binlogd event translation, or will use a Compaq Analyze server running on the local system, you do not need to change the standard configuration. If you plan to use a Compaq Analyze server running on a remote system, you will need to edit the /etc/evmchannel.conf file. In a default installation, the fn_details line for the binlog event channel is configured as follows: 

fn_details    "binlog_details -decevent -ca localhost"
 

This line instructs EVM to use DECevent to provide translations if it is available; otherwise EVM will attempt to connect to a Compaq Analyze server running on the local host. If neither of these options is successful, no translation is done. It is advisable to leave these options in place as the first two items in the list, but if you have other systems running the Compaq Analyze server you can opt to append further -ca items. In the following example, EVM will try in turn DECevent, Compaq Analyze on the local system, Compaq Analyze on the remote system gandalf, and finally Compaq Analyze on the remote system tigger. (This line has been broken at the backslash (\) to fit the page, and appears as a single line in the file). 

fn_details  "binlog_details -decevent -ca localhost -ca gandalf \
-ca tigger"

After you edit the file you should run the evmreload -c command to make the EVM channel manager aware that the file has been updated.

Note that EVM does not start the Compaq Analyze server; it must already be running on the selected system for the translation to succeed. The server is usually started automatically when the system is initialized. For more information, log onto a system that has Compaq Analyze installed and refer to the desta(8) reference page.

See Section 13.3.5 for procedures that enable you to test whether either translation utility is available on your system.

13.3    Using EVM in System Administration

The ability of EVM to monitor multiple event sources and combine them into a single event stream makes it a very useful means of monitoring system activity. By default, the logger is configured to send mail to root when events with a priority of 600 (alert) or greater are posted, but you should take care to check the full event log on a daily basis, using the event viewer or command-line utilities. You can also configure the logger to take other actions, such as sending a pager message, according to any criteria you choose, and you can monitor events at your terminal as they occur, using the evmwatch command.

13.3.1    Monitoring Events Using Commands or Scripts

The evmwatch command is a subscribing EVM client you can use to monitor events from the command line or from within a shell script. Example 13-6 shows its use from the command line while Example 13-8 shows its use from within a script.

See the evmwatch(1) reference page for a complete description of this command. See the EvmFilter(5) reference page for a description of the filter syntax used with the evmwatch command. Both evmwatch and evmshow have numerous options that you will find useful in monitoring your system. Refer to the evmwatch(1) and evmshow(1) reference pages for details of the command options.

Example 13-6:  Monitoring Events from the Command Line Using evmwatch

# evmwatch -f '[priority >= 300]' | [1] evmshow \
-t "@timestamp [@priority] @@"[2]

  1. This command subscribes for all events with a priority of at least 300, and as each one arrives it is written to stdout and piped to the next command. [Return to example]

  2. This command formats the events into human-readable form. In this example the timestamp, priority, and formatted message text is displayed for each event. [Return to example]

You can use the evmwatch -i command option to retrieve a list of all registered events, and pipe the output using the evmshow command to display the event templates in any desired format. For example: 

# evmwatch -i | evmshow -t "@name [@priority] @format" -x

The -i option asks EVM for a list of all event templates; that is, the events that are registered and could be posted. Templates are returned as if they are posted events, and can be piped into the evmshow command for display. The evmshow command show-template (-t option) displays the name of the event, the priority, and the message format. The -x option causes each summary line to be followed by an explanation of the event.

These events have not actually occurred, so the command sequence requested the event's message format, rather than an expanded message. As a result, the summary lines display the messages with names of variables, rather than their values. For example you might see the following summary line: 

AdvFS: Balance error on AdvFS domain $domain

In this example, $domain is a variable that is replaced with the name of the domain when the event is posted.

If you display an event with the evmshow command using @@ in your show template, it provides you with a formatted list of explanations, where available. You can customize the output in many ways as shown in Example 13-7.

Example 13-7:  Customizing Formatted Output

# evmwatch -i -f "[name sys.unix.fs.advfs]" | \[1]
   evmsort -s "@priority-" | \[2]
    evmshow -x -t "@name [@priority] @format"[3]

  1. Ask only for AdvFS event templates [Return to example]

  2. Sort into priority order (highest first) [Return to example]

  3. Display the output using a filter so that only certain events are returned. [Return to example]

Example 13-8 shows how the commands are used within a shell script. The example monitors for a custom event from a remote application, ensuring that the application stays active.

Example 13-8:  Monitoring Events from a Shell Script

#! /bin/sh
# Monitor a remote application for a heartbeat pulse event
 
# Pulse due every 5 minutes, so timeout if nothing in 6 minutes
timeout=360
 
heartbeat_event=myco.myprod.myapp.remote.pulse                 [1]
evmwatch -f "[ name $heartbeat_event ]" -w $timeout >/dev/null [2]
if [ $? -ne 0 ]                                                [3]
then
    echo "event { name myco.myprod.myapp.remote.nopulse }" |
         evmpost                                               [4]
    exit 1
fi
 
 

  1. This line establishes the name of the pulse event. The event is posted only if its name has been specified in an event template file [Return to example]

  2. In this line, the -f option identifies the filter used to select events. The -w option causes the evmwatch command to reset its internal timer when the event is received. [Return to example]

  3. This line is an if condition that tests the return code from evmwatch, taking the then action only if the evmwatch command should return a non-zero exit value. See the evmwatch(1) reference page for more information [Return to example]

  4. This line specifies the event to be posted if an error occurs. The event name must have been declared in an event template file. All of the values in the event are taken from the template. [Return to example]

During the monitoring of system operations, users or system administrators might need to post additional events to explain some occurrence, to mark some time point, or for any other reason. The evmpost command is used to post events. See the evmpost(1) reference page for a description of this command.

13.3.2    Reviewing Event Activity

To assist in the diagnostic processes, EVM provides a graphical event viewer, evmviewer, reporting utilities, and a set of command-line utilities. These features are described in the following sections.

13.3.2.1    Viewing Events

Launch the evmviewer utility ifrom the View Events option on the SysMan Menu. Application controls available with the viewer provide the following options:

See the sysman(8) and evmviewer(8) reference pages for more information on using these applications. See the online help associated with the event viewer for information on using the viewer options.

13.3.2.2    Generating Reports

Using EVM command-line utilities in conjunction with other commands, you can generate printed reports or create formatted displays to use for problem analysis and resolution, system accounting, or other purposes. You can also create programs using the API functions cited in Section 13.1.2.2 in order to use the EVM capabilities for analysis and accounting purposes.

You can create reports that are based on the event logs archived on any host, providing that security requirements for data access are met. The evmget command retrieves the events from various logs and the evmshow command formats the event data into readable form. You can also pipe the output of the evmshow command to any other available formatting commands, or store the output in files for additional processing.

Using the evmsort command, you can sort event data using an event data item as a sort key. By default, the evmsort command sorts events in in ascending order by timestamp, that is, the oldest events first. The sort sequence is controlled by the sort specification command-line option. The specification is a character string of the following form: 

@key_item[+|-] [:@key_item[+|-]

The key_item is the name of any EVM standard data item. The optional trailing character indicates ascending (+) or descending (-) sort. Multiple items can be specified in the string by separating them with colon characters (:). The left-most key_item in the specification is the most significant key. See the evmshow(1) reference page for more information on this command.

The primary tool for formatting information in a report produced by the evmshow command is the show template. A show template is a string that contains event data-item specifiers of the form @item_name[%width], where item_name is the name of any standard event data item, such as timestamp or priority and [%width] indicates the display field width as a number of characters. If a show template is supplied, evmshow replaces each specifier in the template with the value of the corresponding data item from the event, producing an output string containing only the required items. Any characters in the show template that are not part of an item specifier are displayed unchanged in the output.

The special specifier is represented by two at symbols (@@) and is replaced with the formatted event message text. If an item specified in the show template is not present in the event, a minus sign (-) is output in its place. An at symbol (@) not associated with a valid item name is unchanged in the output. If it is necessary to have an at symbol (@) in the output, use a backslash before the at symbol (\) as an escape sequence. For example; \@compaq.com

You can set the environment variable EVM_SHOW_TEMPLATE to any string to be used as a show template for each iteration of the evmshow command.

For example (assuming ksh syntax): 

EVM_SHOW_TEMPLATE="@timestamp @@"

The show template variable is explained in the evmshow(1) reference page. If you set the EVM_SHOW_TEMPLATEvariable as shown in the preceding example, the default format for any event output from the evmshow command will be the event's timestamp (@timestamp) followed by the formatted event message (@@). Events generally contain a format data item, which is the raw event message, and you can use @format instead of @@. However, in most cases the format includes references to variable values that are also contained in the event; when you specify @@ in the template the evmshow command replaces those references with the values of the variables and a complete message is returned. If you do not specify a template, a default of @@ (with no timestamp) is used.

Once you have the show template defined, you can monitor events from your login terminal in real time, using the following command: 


# evmwatch | evmshow &

You can customize this further by changing the template and by adding a filter to avoid returning events that you do not want to see. For example the following value of the EVM_SHOW_TEMPLATEvariable also displays the event priority: 

EVM_SHOW_TEMPLATE="@timestamp [@priority] @@"

You can further control the selected events at the command line. For example, the following command shows events of a priority greater than or equal to 300: 

# evmwatch -f "[priority >= 300]" | evmshow

Using the the -t option with the evmshowcommand, you can override the default show template and specify any valid template file.

Example 13-9 and Example 13-10 provide samples of of the EVM command-line utilities used with the pipe command (|) to produce reports. These commands normally appear on a single line after the command prompt. In the example, line breaks are indicated by a backslash (\).

Example 13-9:  Using EVM Commands to Select Events and Display a Report

# evmget -f '[priority >= 300]'   | \      [1]
   evmsort -s "priority-:timestamp" | \    [2]
    evmshow -t "@priority  @name  @@" | \  [3]
      more                                     [4]
 
 

  1. This command retrieves only those events with a priority of 300 or higher. [Return to example]

  2. This command sorts the events with the highest priority event first. Events with the same priority are sorted oldest first. [Return to example]

  3. This command formats each event for display, showing the priority, the name, and the formatted message. [Return to example]

  4. This command specifies that the events are displayed on the screen. Additional formatting or analysis statements could replace this line. [Return to example]

Example 13-10:  Using EVM Commands to Format Events and Print a Report

 

# evmget                          | \     [1]
   evmsort -s "@name:@timestamp-"   | \   [2]
    evmshow -t "@name%40  @timestamp" | \ [3]
      lpr                                     [4]
 
 

  1. This command retrieves all events. [Return to example]

  2. This command sorts the events by name. Events with the same name are sorted with the most recent events first. [Return to example]

  3. This command formats the events for display, showing the name in a column with a minimum width of 40 characters, and the timestamp. [Return to example]

  4. This command prints the events on the default printer. [Return to example]

13.3.3    Responding to Event Activity

The response to event activity is a procedural action determined by your site-specific needs and conditions. This response can range from activating alarms or paging responsible personnel, through more mundane reactions such as making a log entry or ignoring an expected occurrence of a regular activity. It is also possible to configure the event processing sequence to perform a series of dependent tasks, using an event output by one task as the trigger to activate the next process. EVM provides an interface to the response activity through the logging facility. The available options are event storage and event forwarding.

The EVM logger, evmlogger, started automatically by the EVM daemon, is responsible for the following:

The logger is an ordinary EVM client that is controlled through a configuration file. The default is /etc/evmlogger.conf. See the evmlogger.conf(4) reference page for additional information on this file. See the evmlogger(8) reference page for additional information on the command.

13.3.3.1    Displaying Detailed Event Information

While many events are posted as part of normal activity, and are simply informational in nature, others will require specific action. In many cases you can get additional information about the event by displaying a detailed view, although note that many event types simply carry text information from another source, and the explanation will be generic.

You can display a detailed view of an event through the event viewer by selecting the event of interest and selecting the "Details..." button. To display details from the command line you will need to isolate the event or events of interest, and then pipe them into the evmshow -dcommand (to show full details of the event) or the evmshow -x command (to show just the explanation text). The simplest way to isolate an event is to use its event-id; however if the system has been restarted recently the event-id may not be unique, and might need to be qualified with the timestamp. Example 13-11 illustrates the commands and the output generated by the commands.

Example 13-11:  Displaying an Event Explanation from the Command Line

# evmget | evmshow -t "@timestamp @event_id%4 @@"  [1]
 
...
15-Apr-1999 14:19:06 0   EVM daemon: Configuration completed
15-Apr-1999 14:19:06 1   EVM daemon: Initialization completed
15-Apr-1999 14:19:06 2   EVM logger: Logger started
15-Apr-1999 14:19:06 3   EVM: Mark event - initial
15-Apr-1999 14:19:06 5   EVM logger: Started eventlog
/var/evm/evmlog/evmlog.19990415
...
 
# evmget -f '[id = 3] & [time 1999:4:15:*:14:19:06]' |  \ [2]
  evmshow -t "@timestamp @@" -x                     [3]
 
15-Apr-1999 14:19:06 EVM: Mark event - initial
    The EVM "mark initialization" event is posted by the
    evmlog event channel's "fn_monitor" function when it
    is first started by the EVM channel monitor. This
    event indicates the point at which EVM began to post
    mark events; the monitor function should run
    periodically thereafter to post normal "mark" events. 
 
 
 

  1. This command retrieves and displays all events, in a format that includes the event_id for each event in a column that is four characters wide. [Return to example]

  2. This command retrieves only the event that has an event_id of 3, and that was posted at 14:19:06 on April 15th, 1999.

    The timestamp information is necessary in order to exclude any other events that might have the same event_id. [Return to example]

  3. When run with the -x option, evmshow displays the explanation for the event. Use of the -t option in conjunction with -x displays formatted information before the explanation, which is useful confirmation that you have selected the expected event. [Return to example]

13.3.3.2    Handling Events Automatically

If you want to automate the handling of selected events, you can configure the EVM logger to forward the event by executing a command. For example, you can mail the event information to a paging service, or invoke an event-handling application program.

By default, the logger is configured to mail high priority events to the root user. You can use that default forwarding command as an example for developing your own actions. See Section 13.2.1.3 and the evmlogger.conf(4) reference page for more information.

When configuring the logger to forward an event, note that:

13.3.3.3    Logging Events

All events meeting the specifications of an eventlog statement in the configuration file are written to the event log as binary events. See Section 13.1.2.3 for the default location of this file and the naming conventions.

As shown in Example 13-3, you can include a suppress group specification in an eventlog statement in the configuration file. When you include such a statement, events meeting the suppression criteria are not entered in the log. One instance of the event is stored, with additional data indicating the number of events and the time of the first and last occurrence of the event. See the evmlogger.conf(4) reference page for the explanation of this criterion.

13.3.3.4    Displaying Events on a Terminal

If a terminal device is indicated as the logfile in the configuration file, all events meeting the filter specifications of an eventlog statement are formatted for display on the terminal. (See Section 13.2.1.3 for a discussion of the configuration file.)

You can also use the evmshow command to display events retrieved from log files as described in Section 13.3.2.

13.3.3.5    Forwarding Events

Events can be forwarded as specified in the configuration file. (See Section 13.2.1.3 for a discussion of the configuration file.)

All events meeting the filter specifications of a forward statement in the configuration file are written to the standard input (stdin) of the command specified in the statement. The command can be the name of a shell script, a single UNIX command, a series of UNIX commands (pipeline), or any other executable statement. The following operations are typically specified as a forwarding action:

13.3.4    Posting Quick Messages

The evmpost command provides two quick message formats, one for use by system administrators and one for use by all users. For system administrators the format of the command is: 

# evmpost [-a msg] [-p priority]

For general system users the format is: 

# evmpost[-u msg] [-p priority]

The use of these options results in the posting of an event with the event name sys.unix.evm.msg.admin or sys.unix.evm.msg.user.

13.3.5    Troubleshooting EVM

If you suspect that EVM is not operating correctly, the first step is to check the message files in /var/evm/adm/logfiles. Messages in these files are also displayed through the EVM viewer and evmget, as part of the misclog event channel.

The following list describes some common problems and the initial steps you should take in trying to resolve such problems:

Kernel events are not being posted.

Check the EVM daemon log file for errors using the following command: 


# more /var/evm/adm/logfiles/evmdaemon.log

Check for the presence of the kernel interface pseudodevice using the following command: 

# ls -l /dev/kevm

If this pseudodevice is not present, create it using the following command: 

# dsfmgr -vF

A subscribing application fails to receive expected events.

Verify that the poster is authorized to post these events by checking the authorization file using the following command: 


# more /etc/evm.auth

Verify that the event is registered using the following command: 

# evmwatch -i -f `[nameevent_name]' | \
  evmshow -t "@name"

If the events are still not shown, run evmreload and check again. If they are still not visible, check that the template files are correctly installed.

Verify that the subscriber is authorized to access these events, using the following command: 

# more /etc/evm.auth

Verify that the expected events are actually being posted using the following command: 


# evmwatch | evmshow -t "@name @@"

Run the program that posts the event, and check that the preceding evmwatch command displays them correctly.

A posting program is unable to post events.

Verify that the EVM daemon is running, using the following command: 


# ps -aef | grep evmd

Verify that the poster is authorized to post these events by checking the authorization file using the following command: 


# more /etc/evm.auth

Verify that the event is registered using the following command: 

# evmwatch -i -f `[nameevent_name]' | \
  evmshow -t "@name"

If the events are still not shown, run the evmreload command and check again. If they are still not visible, check that the template files are correctly installed.

Expected syslog or binlog events are not visible through EVM.

You must either be logged in as root or belong to the adm group in order to access syslog and binlog events.

Check that the binlogd and syslogd daemons are running, using the ps command.

Check that the /etc/syslog_evm.conf file is configured to forward the events you expect to see.

Use the following commands to test communication with syslog and binlog: 


# evmwatch | evmshow &
# logger "test syslog message"
# logger -b "test binlog message"

Event retrieval through evmget or the event viewer is slow.

Check the sizes of all log files, particularly the evmlog files (/var/evm/evmlog), the binary error log (/var/adm/binary.errlog) and the SMS daemon log files (/var/adm/sysman/sysman_station/logs).

Be sure to use the -L option of the ls command when looking at file sizes, to make sure that you see the file itself and not a symbolic link or CDSL.

See binlogd(8) for details of binary log size management, but note that EVM retrieves events from the archive log file, so starting a new log might not immediately reduce the number of events available to EVM. You can use cron to perform a regular binlog archive. You can reduce the sizes of the evmlog files by changing configuration values in the logger and channel configuration files (/etc/evmlogger.conf and /etc/evmchannel.conf).

Expected events are not being logged.

Check the event priority. Only events with a priority of 200 or higher are logged by the EVM logger.

Cannot post or subscribe to events through a remote daemon.

Check that remote access is configured and if it is not:

(Be sure to consider the security implications of enabling a remote connection.)

Binlog events are not being translated.

Use the following procedures to troubleshoot the absence of a translation utility:

  1. Run the following command: 

    
# usr/sbin/dia

    If DECevent is installed, this command displays the translated contents of the current binary error log file, /var/adm/binary.errlog.

  2. If the dia command is not found, use the following command to test the status of the DECevent software subset (the distribution kit): 

    # setld -i | grep OSFDIA

    This command will tell you if the subset OSFDIABASE*** DECEvent Base Kit (Translation/Analysis) is installed or not. If it is not installed, mount the installation media and use the setld command to install the subset. Refer to the setld(8) reference page.

Check for the presence of Compaq Analyze as follows:

  1. Use the following command to see if the Compaq Analyze director service is running on the local host: 

    # ps agx | grep desta

  2. If you do not see the running desta daemon, the Compaq Analyze utility might still be installed but not running or properly configured. To test whether if the utility is installed on your system, look for the binaries using the following command: 

    
# ls /usr/opt/compaq/svctools/bin/desta*

  3. If you do not find the binaries, install Compaq Analyze from the distribution media using the setld command.

13.3.6    Updating Compaq Analyze

You can download the latest version of Compaq Analyze (and other WEBES tools) from the following location:http://www1.service.digital.com/svctools/webes/index.html.Download the kit from the website, saving it to /var/tmp/webes. Unpack the kit using a command similar to the following: 

# tar -xvf <tar file name>

Use the following command to install the Compaq Web Based Enterprise Service Suite: 

# setld -l /var/temp/webes/kit

During the installation, you can safely select the default options. However, you might not want to install all the optional WEBES tools. Only Compaq Analyze is used by EVM. Refer to the separate Compaq Analyze documentation and ca(8) reference page for more information. Note that recent Alpha EV6 processors are supported only by Compaq Analyze and not DECevent