14    Administering Crash Dumps

This chapter describes how you configure the system crash dump environment and how you save and store crash dumps and their associated data. Crash dumps are a snapshot of the running kernel, taken automatically when the system shuts down unexpectedly. Crash dumps are most often referenced when you contact your technical support representatives to analyze and correct problems that result in a system crash. However, if you are an experienced system administrator or developer you might be familiar with techniques of crash dump analysis and you might want to take and analyze your own dump files.

The following topics are discussed in this chapter:

14.1    Overview of Crash Dumps

When a system shuts down unexpectedly, it writes all or part of the data in physical memory to swap space on disk (the virtual memory space). Such shut down events are often referred to as system crashes or panics. The stored data and status information is called a crash dump. Crash dumps are different from application-related core dumps, after which the system usually keeps running. After a crash dump, the system is shut down to the console prompt (>>>) and must be rebooted when the problem is identified and resolved.

During the reboot process, the system moves the crash dump into a file and copies the kernel executable image to another file. Together, these files are the crash dump files and are often required for analysis when a system crashes, or during the development of custom kernels (debugging). You might need to supply a crash dump file to your technical support organization to analyze system problems.

To administer dumps, you must understand how crash dump files are created. You must also reserve space on disks for the crash dump and crash dump files. The amount of space you reserve depends on your system configuration and the type of crash dump you want the system to perform.

14.1.1    Related Documentation and Utilities

Crash dumps make use of the on-disk virtual memory swap space. Administering swap space is described in Chapter 3. System event managment is described Chapter 12 which describes the binlogd and syslogd event management channels.

The following documentation contains information on crash dumps or related topics:

14.1.2    Files Used During Crash Dumps

By default, the savecore command copies a crash dump file into /var/adm/crash, although you can redirect crash dumps to any file system that you designate. As for many other system files, /var/adm/crash is a context-dependent symbolic link (CDSL) which facilitates joining systems into clusters. The CDSL for this directory is /var/cluster/members/member0/adm/crash. Within this directory, the following files are created or used:

14.2    Crash Dump Creation

After a system crash, you normally reboot your system by issuing the boot command at the console prompt. During a system reboot, the /sbin/savecore script invokes the savecore command. This command moves crash dump information from the swap partitions into a file and copies the kernel that was running at the time of the crash into another file. You can analyze these files to help you determine the cause of a crash. The savecore command also logs the crash in system log files.

You can invoke the savecore command from the command line. For information about the command syntax, see the savecore(8) reference page.

14.2.1    Crash Dump File Creation

When the savecore command begins running during the reboot process, it determines whether a crash dump occurred and whether the file system contains enough space to save it. (The system saves no crash dump if you shut it down and reboot it; that is, the system saves a crash dump only when it crashes.)

If a crash dump exists and the file system contains enough space to save the crash dump files, the savecore command moves the crash dump and a copy of the kernel into files in the default crash directory, /var/adm/crash. (You can modify the location of the crash directory.) The savecore command stores the kernel image in a file named vmunix.n, and by default it stores the (compressed) contents of physical memory in a file named vmzcore.n.

The n variable specifies the number of the crash. The number of the crash is recorded in the bounds file in the crash directory. After the first crash, the savecore command creates the bounds file and stores the number 1 in it. The command increments that value for each succeeding crash.

The savecore command runs early in the reboot process so that little or no system swapping occurs before the command runs. This practice helps ensure that crash dumps are not corrupted by swapping.

14.2.2    Crash Dump Logging

Once the savecore command writes the crash dump files, it performs the following steps to log the crash in system log files:

  1. Writes a reboot message to the /var/adm/syslog/auth.log file.

    If the system crashed due to a panic condition, the panic string is included in the log entry.

    You can cause the savecore command to write the reboot message to another file by modifying the auth facility entry in the syslog.conf file. If you remove the auth entry from the syslog.conf file, the savecore command does not save the reboot message.

  2. Attempts to save the kernel message buffer from the crash dump.

    The kernel message buffer contains messages created by the kernel that crashed. These messages might help you determine the cause of the crash.

    The savecore command saves the kernel message buffer in the /var/adm/crash/msgbuf.savecore file, by default. You can change the location to which savecore writes the kernel message buffer by modifying the msgbuf.err entry in the /etc/syslog.conf file. If you remove the msgbuf.err entry from the /etc/syslog.conf file, savecore does not save the kernel message buffer.

    Later in the reboot process, the syslogd daemon starts up, reads the contents of the msgbuf.err file, and moves those contents into the /var/adm/syslog/kern.log file, as specified in the /etc/syslog.conf file. The syslogd daemon then deletes the msgbuf.err file. For more information about how system logging is performed, see the syslogd(8) reference page.

  3. Attempts to save the binary event buffer from the crash dump.

    The binary event buffer contains messages that can help you identify the problem that caused the crash, particularly if the crash was due to a hardware error.

    The savecore command saves the binary event buffer in the /usr/adm/crash/binlogdumpfile file by default. You can change the location to which savecore writes the binary event buffer by modifying the dumpfile entry in the /etc/binlog.conf file. If you remove the dumpfile entry from the /etc/binlog.conf file, savecore does not save the binary event buffer.

    Later in the reboot process, the binlogd daemon starts up, reads the contents of the /usr/adm/crash/binlogdumpfile file, and moves those contents into the /usr/adm/binary.errlog file, as specified in the /etc/binlog.conf file. The binlogd daemon then deletes the binlogdumpfile file. For more information about how binary error logging is performed, see the binlogd(8) reference page.

14.2.3    Swap Space

When the system creates a crash dump, it writes the dump to the swap partitions. The system uses the swap partitions because the information stored in those partitions has meaning only for a running system. Once the system crashes, the information is useless and can be safely overwritten.

Before the system writes a crash dump, it determines how the dump fits into the swap partitions, which are defined in the /etc/sysconfigtab file. For example, the following fragment of the /etc/sysconfigtab entry shows three swap partitions available: 

vm:
   swapdevice=/dev/disk/dsk0b, /dev/disk/dsk3h, /dev/disk/dsk13g
   vm-swap-eager=1

The following list describes how the system determines where to write the crash dump:

Each crash dump contains a header, which the system always writes to the end of the primary swap partition. The header contains information about the size of the dump and where the dump is stored. This information allows savecore to find and save the dump at system reboot time.

The way that a crash dump is taken can be controlled by several kernel attributes, as follows:

In most cases, compressed dumps will fit on the primary swap partition and you will not find it necessary to modify these variables. The next section describes dump_sp_threshold, which is relevant in understanding how a crash dump is created. The use of the remaining kernel attributes controls the content of the dump. These attributes are described in Section 14.3.

Controlling the Use of Swap Partitions

You can configure the system so that it fills the secondary swap partitions with dump information before writing any information (except the dump header) to the primary swap partition. The attribute that you use to configure where crash dumps are written first is the dump_sp_threshold attribute.

The value in the dump_sp_threshold attribute indicates the amount of space you normally want available for swapping as the system reboots. By default, this attribute is set to 4096 blocks, meaning that the system attempts to leave 2 MB of disk space open in the primary swap partition after the dump is written.

Figure 14-1 shows the default setting of the dump_sp_threshold attribute for a 40 MB swap partition. (Note that 40 MB is atypical of a real swap partition size on most systems, but the principle is the same.)

Figure 14-1:  Default dump_sp_threshold Attribute Setting

The system can write 38 MB of dump information to the primary swap partition shown in Figure 14-1. Therefore, a 30 MB dump fits on the primary swap partition and is written to that partition. However, a 40 MB dump is too large; the system writes the crash dump header to the end of the primary swap partition and writes the rest of the crash dump to secondary swap partitions, if available.

Setting the dump_sp_threshold attribute to a high value causes the system to fill the secondary swap partitions before it writes dump information to the primary swap partition. For example, if you set the dump_sp_threshold attribute to a value that is equal to the size of the primary swap partition, the system fills the secondary swap partitions first. (Setting the dump_sp_threshold attribute is described in Section 14.3.1.). Figure 14-2 illustrates how a crash dump is written to secondary swap partitions on multiple devices.

Figure 14-2:  Crash Dump Written to Multiple Devices

If a noncompressed crash dump fills partition e in Figure 14-2, the system writes the remaining crash dump information to the end of the primary swap partition. Note that the system fills as much of the primary swap partition as is necessary to store the entire dump. The dump is written to the end of the primary swap partition to attempt to protect it from system swapping. However, the dump can fill the entire primary swap partition and might be corrupted by swapping that occurs as the system reboots.

If a compressed crash dump needs space on the primary swap partition, it always begins 2 MB into the partition, taking what space it requires.

Estimating Crash Dump Size Using dumpsys

To estimate the size of crash dumps, you can use the dump utility dumpsys, which produces a run time or continuable dump. See Section 14.4.1 for information on using dumpsys. Note that you may have to temporarily create file system space to hold the experimental dumps. You can produce both full and partial dumps using this method. Crash dumps are compressed by default, and you can use the expand_dump utility to produce a noncompressed crash dumps or use the -u option with the dumpsys utility.

Because the crash dumps written to swap are about the same size as their resulting saved crash dump files, you can easily determine how large a crash dump was by examining the size of the resulting crash dump file. For example, to determine how large the first crash dump file created by your system is, issue the following command: 

# ls -s /var/adm/crash/vmzcore.0
20480 vmzcore.0

This command displays the number of 512-byte blocks occupied by the crash dump file. In this case, the file occupies 20,480 blocks, so you know that a crash dump written to the swap partitions would also occupy about 20,480 blocks.

In some cases, a system contains so much active memory that it cannot store a crash dump on a single disk. For example, suppose your system contains 2 GB of memory but only has several 4 GB disks, most of which are dedicated to storing data. Crash dumps for this system may be too large to fit on a single swap partition on a single device. To cause crash dumps to spread across multiple disks, create secondary swap partitions on several disks. The system automatically writes dumps that are too large to fit in the specified portion of the primary swap partition to secondary swap partitions.

14.2.4    Planning Crash Dump Space

Because crash dumps are written to the swap partitions on your system, you allow space for crash dumps by adjusting the size of your swap partitions, thereby creating temporary or permanent swap space. For information about modifying the size of swap partitions, see the swapon(8) reference page.

Note

Be sure to list all permanent swap partitions in the /etc/sysconfigtab file. The savecore command, which copies the crash dump from swap partitions to a file, uses the information in the /etc/sysconfigtab file to find the swap partitions. If you omit a swap partition from /etc/sysconfigtab, the savecore command might be unable to find the omitted partition.

The sections that follow give guidelines for estimating the amount of space required for partial and full crash dumps. In addition, setting the dump_sp_threshold attribute is described.

14.2.5    Planning and Allocating File System Space for Crash Dump Files

Using the information on typical crash dump sizes for your system, you can also plan and allocate the file system space that you need for /var/adm/crash.

For example, suppose you save partial crash dumps. Your system has 96 MB of memory and you have reserved 85 MB of disk space for crash dumps and swapping. In this case, you should reserve 20 MB of space in the file system for storing crash dump files. You need to reserve considerably more space if you want to save files from more than one crash dump. If you want to save files from multiple crash dumps, consider archiving older crash dump files. See Section 14.5 for information about archiving crash dump files.

By default, savecore writes crash dump files to the /var/adm/crash directory. To reserve space for crash dump files in the default directory, you must mount the /var/adm/crash directory on a file system that has a sufficient amount of disk space. (For information about mounting file systems, see Chapter 6 and the mount(8) reference page.) If you expect your crash dump files to be large, you might need to use a Logical Storage Manager (LSM) file system to store crash dump files. For information about creating LSM file systems, see the Logical Storage Manager manual.

If your system cannot save crash dump files due to insufficient disk space, the system returns to single-user mode. This return to single-user mode prevents system swapping from corrupting the crash dump. Once in single-user mode, you can make space available in the crash directory or change the crash directory. One possibility in this situation is to issue the savecore command at the single-user mode prompt. On the command line, specify the name of a directory that contains a sufficient amount of file space to save the crash dump files. For example, the following savecore command writes crash dump files to the /usr/adm/crash2 directory: 

# savecore /usr/adm/crash2

Once savecore has saved the crash dump files, you can bring your system to multiuser mode.

Specifying a directory on the savecore command line changes the crash directory only for the duration of that command. If the system crashes later and the system startup script invokes the savecore script, savecore copies the crash dump to files in the default directory, which is normally /var/adm/crash.

You can control the default location of the crash directory with the rcmgr command. For example, to save crash dump files in the /usr/adm/crash2 directory by default (at each system startup), issue the following command: 

# /usr/sbin/rcmgr set SAVECORE_DIR /usr/adm/crash2

If you want the system to return to multiuser mode, regardless of whether it saved a crash dump, issue the following command: 

# /usr/sbin/rcmgr set SAVECORE_FLAGS M

14.3    Choosing the Content and Method of Crash Dumps

Crash dumps are compressed and partial by default, but can be full, noncompressed, or both. Normally, partial crash dumps provide the information that you need to determine the cause of a crash. However, you might want the system to generate full crash dumps if you have a recurring crash problem and partial crash dumps have not been helpful in finding the cause of the crash.

A partial crash dump contains the following:

A full crash dump contains the following:

You can modify how crash dumps are taken:

These options are explained in the following sections.

14.3.1    Adjusting the Primary Swap Partition's Crash Dump Threshold

To configure your system so that it writes even small crash dumps to secondary swap partitions before the primary swap partition, use a large value for the dump_sp_threshold attribute. As described in Section 14.2, the value you assign to this attribute indicates the amount of space that you normally want available for system swapping after a system crash.

To adjust the dump_sp_threshold attribute, issue the sysconfig command. For example, suppose your primary swap partition is 40 MB. To raise the value so that the system writes crash dumps to secondary partitions, issue the following command: 

# sysconfig -r generic dump_sp_threshold=20480

In the preceding example, the dump_sp_threshold attribute, which is in the generic subsystem, is set to 20,480 512-byte blocks (40 MB). In this example, the system attempts to leave the entire primary swap partition completely open for system swapping. The system automatically writes the crash dump to secondary swap partitions and the crash dump header to the end of the primary swap partition.

The sysconfig command changes the value of system attributes for the currently running kernel. To store the new value of the dump_sp_threshold attribute in the sysconfigtab database, modify that database using the sysconfigdb command. For information about the sysconfigtab database and the sysconfigdb command, see the sysconfigdb(8) reference page.

Note

Once the savecore program has copied the crash dump to a file, all swap devices are immediately available for mounting and swapping. The sharing of swap space only occurs for a short time during boot, and usually on systems with a small amount of physical memory.

14.3.2    Including User Page Tables in Partial Crash Dumps

By default, the system omits user page tables from partial crash dumps. These tables do not normally help you determine the cause of a crash and omitting them reduces the size of crash dumps and crash dump files. However, you may be directed by your technical support person to include user page tables.

If you want the system to include user page tables in partial crash dumps, set the value of the dump_user_pte_pages attribute to 1. The dump_user_pte_pages attribute is in the vm subsystem. The following example shows the command you issue to set this attribute: 

# sysconfig -r generic dump_user_pte_pages = 1

The sysconfig command changes the value of system attributes for the currently running kernel. To store the new value of the dump_user_pte_pages attribute in the sysconfigtab database, modify that database using the sysconfigdb command or use dxkerneltuner.

To return to the system default of not writing user page tables to partial crash dumps, set the value of the dump_user_pte_pages attribute to 0 (zero).

14.3.3    Selecting Partial or Full Crash Dumps

By default, the system generates partial crash dumps. If you want the system to generate full crash dumps, you can modify the default behavior by setting the kernel's partial_dump variable to 0 (zero) as follows: 

# sysconfig -r generic partial_dump=0
partial_dump: reconfigured
# sysconfig -q generic partial_dump
generic:
partial_dump = 0

To return to partial crash dumps, reset the partial_dump variable to 1.

Note that you can also use dxkerneltuner or sysconfigdb to modify kernel entries and preserve the modifications across reboots.

14.3.4    Expected Dump Compression

The expected_dump_compression variable is used to signal how much compression you typically expect to achieve in a dump . By default, the value of expected_dump_compression is set to 500, the median for a minimum allowed value of 0 (zero) and a maximum value of 1000. The following steps describe how you calculate the appropriate expected_dump_compression variable for your system:

  1. Create a compressed dump, using dumpsys, as described in Section 14.4.1. Using the command ls -s, note the size of this dump as value a.

  2. Use expand_dump utility to produce a noncompressed version of the dump. Using the command ls -s, note the size of this dump as value b.

  3. Divide a by b to produce the approximate compression ratio.

  4. Repeat the previous steps several times and choose the largest value of the compression ratio. Multiply the compression ratio by 1000 to produce an expected dump value.

  5. Add 10 percent of the expected dump value to create a value for expected_dump_compression.

Set the kernel's expected_dump_compression variable to the required value using sysconfig as follows: 

# sysconfig -r generic expected_dump_compression=750
expected_dump_compression: reconfigured
# sysconfig -q generic partial_dump
generic:
expected_dump_compression=750

Note that you can also use dxkerneltuner or sysconfigdb to modify kernel entries and preseve the modifications across reboots.

14.3.5    Selecting and Using Noncompressed Crash Dumps

By default, crash dumps are compressed to save disk space, allowing you to dump a larger crash dump file to a smaller partition. This can offer significant advantages on systems with a large amount of physical memory, particularly if you want to tune the system to discourage swapping for real-time operations. On reboot after a crash, the crash dump utility, savecore automatically detects that the dump is compressed, using information in the crash dump header in swap. It then copies the crash dump file from swap to the /var/adm/crash directory. The compressed crash dump files are identified by the letter z in the file name, to distinguish them from noncompressed crash dump files. For example: vmzcore.1.

This type of compressed crash dump file can be used with some debugging tools such as dbx while still compressed, which is not true of the type of compression produced by tools such as compress or gzip. If you need to use a tool that does not support compressed crash dump files, you can convert it to a conventional noncompressed format with the expand_dump utility. The following example shows how you use the expand_dump utility: 

# expand_dump vmzcore.2 vmcore.2

You may want to disable compressed dumps if you are always using tools or scripts that will not work with the compressed format, and it is not convenient to use the expand_dump utility. To do this, use the following sysconfig command: 


# sysconfig -r generic compressed_dump=0

Using this method will only temporarily change the mode of dumping to noncompressed and it will revert to compressed on the next reboot. To make a more permanent change, you can use sysconfigdb to update the value of compressed_dump in the /etc/sysconfigtab file or use dxkerneltuner to modify the value in the generic subsystem.

Refer to the reference pages savecore(8), expand_dump(8), and sysconfig(8) for information on crash dump compression and how to produce a noncompressed crash dump file.

14.4    Generating a Crash Dump Manually

The following sections describes how you can manually create a crash dump file under two conditions:

It is assumed that you have planned adequate space for the crash dump file and set any kernel parameters as described in the preceding sections.

14.4.1    Continuable Dumps on a Running System

When you cannot halt the system and take a normal crash dump, use the dumpsys command to dump a snapshot of memory. Because the system is running while dumpsys takes a snapshot, memory may be changing as it is copied. Analysis of the resulting dump may show things like incomplete linked lists and partially zeroed pages, which are not problems, but reflect the transitory state of memory. For this reason, some system problems cannot be detected by dumpsys and you may need to halt the system and force a crash dump as described in Section 14.4.2. By default, the dumpsys command writes the crash dump in /var/adm/crash.

The text file /var/adm/crash/minfree specifies the minimum number of kilobytes that must be left on the file system after dumpsys copies the dump. By default, this file does not exist, indicating that no minimum is set. To specify a minimum, create the file and store the number of kilobytes you want reserved in it. You can override the setting in /minfree using the -i option. The -s option displays the approximate number of disk blocks that full and partial dumps will require. The exact size can not be determined ahead of time for the following reasons:

The following examples show a dump taken on a system with 512 KB of physical memory. Note that this is a noncompressed crash dump, and dumps will normally be compressed by default. 


# dumpsys -s
Approximate full dump size = 1048544 disk blocks,
 if compressed, expect about 524272 disk blocks.
Approximate partial dump size = 94592 disk blocks,
 if compressed, expect about 47296 disk blocks.

# dumpsys -i /userfiles
Saving 536797184 bytes of image in /userfiles/vmzcore.0
# ls /userfiles
bounds vmzcore.0 vmunix.0
 
 

Refer to the dumpsys(8) reference page for information on the command options. See Kernel Debugging for information on analyzing the continuable crash dump.

14.4.2    Forcing Crash Dumps on a Hung System

You can force the system to create a crash dump when the system hangs. On most hardware platforms, you force a crash dump by following these steps:

  1. If your system has a switch for enabling and disabling the Halt button, set that switch to the Enable position.

  2. Press the Halt button.

  3. At the console prompt, enter the crash command.

Some systems have no Halt button. In this case, follow these steps to force a crash dump on a hung system:

  1. Press Ctrl/p at the console.

  2. At the console prompt, enter the crash command.

If your system hangs and you force a crash dump, the panic string recorded in the crash dump is the following: 

hardware restart

This panic string is always the one recorded when system operation is interrupted by pressing the Halt button or Ctrl/p.

14.5    Storing and Archiving Crash Dump Files

If you are working entirely with compressed (vmzcore.n) crash dump files, they should be sufficiently compressed for efficient archiving. The following sections discuss certain special cases.

Section 14.5.1 describes how to compress files for storage or transmission if:

Section 14.5.2 describes how to uncompress partial crash dump files that have been compressed from vmcore.n files.

14.5.1    Compressing a Crash Dump File

To compress a vmcore.n crash dump file, use a utility such as gzip, compress, or dxarchiver. For example, the following command creates a compressed file named vmcore.3.gz: 

# gzip vmcore.3

A vmzcore.n crash dump file, on the other hand, uses a special compression method that makes it readable by debuggers and crash analysis tools without requiring decompression. A vmzcore.n file is substantially compressed compared to the equivalent vmcore.n file, but not as much as if the latter had been compressed using a standard UNIX compression utility, such as gzip. Standard compression applied to a vmzcore.n file will make the resulting file about 40 percent smaller than the equivalent vmzcore.n file.

If you need to apply the maximum compression possible to a vmzcore.n file, follow these steps:

  1. Uncompress the vmzcore.n file using the expand_dump command (see expand_dump(8)). The following example creates an uncompressed file named vmcore.3 from the file vmzcore.3: 

    
# expand_dump vmzcore.3

  2. Compress the resulting vmcore.n file using a standard UNIX utility. The following example uses the gzip command to create a compressed file named vmcore.3.gz: 

    # gzip vmcore.3

You can uncompress a vmzcore.n file only with the expand_dump command. (Do not use gunzip, uncompress, or any other utility).

After a vmzcore.n file has been uncompressed into a vmcore.n file with expand_dump, you cannot compress it back into a vmzcore.n file.

14.5.2    Uncompressing a Partial Crash Dump File

This section applies only if you are uncompressing a partial crash dump file that was compressed from a vmcore.n file.

If you compress a vmcore.n dump file from a partial crash dump, you must use care when you uncompress it. Using the gunzip or uncompress command with no options results in a vmcore.n file that requires space equal to the size of memory. In other words, the uncompressed file requires the same amount of disk space as a vmcore.n file from a full crash dump.

This situation occurs because the original vmcore.n file contains UNIX File System (UFS) file "holes." UFS files can contain regions, called holes, that have no associated data blocks. When a process, such as the gunzip or uncompress command, reads from a hole in a file, the file system returns zero-valued data. Thus, memory omitted from the partial dump is added back into the uncompressed vmcore.n file as disk blocks containing all zeros.

To ensure that the uncompressed core file remains at its partial dump size, you must pipe the output from the gunzip or uncompress command with the -c option to the dd command with the conv=sparse option. For example, to uncompress a file named vmcore.0.Z, issue the following command: 

# uncompress -c vmcore.0.Z | dd of=vmcore.0 conv=sparse
 
262144+0 records in
 
262144+0 records out