16    Using the Problem Solving Tools

To help you resolve problems with network hardware, the network itself, and various network services, the operating system provides problem solving tools to help you do the following tasks:

• Detect network interface failures

• Test access to network hosts on the Internet network

• Display network statistics

• Display and modify the Internet to Ethernet translation tables

• Display a datagram's route to a network host

• Display headers of packets on the network

• Test a UUCP remote connection

• Monitor a UUCP file transfer

• Display the error log file

• Display the syslogd daemon message files

16.1    Detecting Network Interface Failures

Use the Network Interface Failure Finder (NIFF) utility to detect and report possible failures in network interface cards or their connections.

Once you specify an interface to monitor by using the niffconfig command, the kernel Traffic Monitor Thread (TMT) checks the connectivity of the monitored interface and, if necessary, informs the NIFF daemon (niffd) to generate traffic for the network interface that is determined to have failed. If the niffd daemon cannot get the interface packet counters to increment, signifying that the interface is alive and well, it reports the problem to the Event Manager subsystem.

You can use NIFF as a standalone reporting utility, or you can configure a NetRAIN interface (as discussed in Section 2.4), a virtual interface which incorporates the NIFF technology and provides automatic failover for selected network interfaces on your system.

See niff(7) and niffconfig(8) for more information about the NIFF utility. See Appendix A and netstat(1) for more information about monitoring network interfaces.

16.2    Testing Access to Internet Network Hosts

Use the ping command to test your system's ability to reach a host on the Internet network. The ping command has the following syntax:

/usr/sbin/ping [options... ] hostname

Table 16-1 describes some of the ping command options.

Table 16-1:  Options to the ping Command

The ping command sends an Internet Control Message Protocol (ICMP) echo request to the host specified. When the request is successful, the remote host sends the data back to the local host. If the remote host does not respond to the request, the ping command does not display any results.

To terminate the ping command output, press Ctrl/C. When terminated, the ping command displays statistics on packets sent, packets received, the percentage of packets lost, and the minimum, average, and maximum round-trip packet times.

You can use the output from the ping command to help determine the cause of direct and indirect routing problems such as an unreachable host, a timed-out connection, or an unreachable network.

When using the ping command for fault isolation, first test the local host to verify that it is running. If the local host returns the data correctly, use the ping command to test remote hosts farther and farther away from the local host.

If you do not specify command options, the ping command displays the results of each ICMP request in sequence, the number of bytes received from the remote host, and the round-trip time on a per-request basis.

The following example shows the output from a ping command to a host named host1:

% ping host1
PING host1.corp.com (16.20.32.2): 56 data bytes
64 bytes from 16.20.32.2: icmp_seq=0 ttl=255 time=11 ms
64 bytes from 16.20.32.2: icmp_seq=1 ttl=255 time=3 ms
64 bytes from 16.20.32.2: icmp_seq=2 ttl=255 time=7 ms
64 bytes from 16.20.32.2: icmp_seq=3 ttl=255 time=3 ms
64 bytes from 16.20.32.2: icmp_seq=4 ttl=255 time=7 ms
64 bytes from 16.20.32.2: icmp_seq=5 ttl=255 time=3 ms
[Ctrl/C]
----host1.corp.com PING Statistics---
6 packets transmitted, 6 packets received, 0% packet loss
roundtrip (ms) min/avg/max = 3/5/11 ms

The ping command accepts an IPv4 address, IPv6 address, or node name on the command line. The following example specifies an IPv6 address:

# ping -c 2 5F00:2100:108C:4000:8C40:800:2B2D:2B2
PING (5F00:2100:108C:4000:8C40:800:2B2D:2B2): 56 data bytes
64 bytes from 5F00:2100:108C:4000:8C40:800:2B2D:2B2: icmp6_seq=0
     hlim=58 time=17 ms
64 bytes from 5F00:2100:108C:4000:8C40:800:2B2D:2B2: icmp6_seq=1
     hlim=58 time=17 ms
----5F00:2100:108C:4000:8C40:800:2B2D:2B2 PING Statistics----
2 packets transmitted, 2 packets received, 0% packet loss
round-trip (ms)  min/avg/max = 17/17/17 ms

The command sends appropriate ECHO_REQUEST packets based on the address family being used. In some cases, a single node name might resolve to both an IPv4 and IPv6 address. Use the -V4 or -V6 option specify which address to use.

You can also use the -I flag to force the use of a specific interface. For example:

# ping -I ln0 FE80::800:2B2D:2B2

See ping(8) for more information on this command and its options.

16.3    Displaying Network Statistics

Use the netstat command to display network statistics for sockets, interfaces, and routing tables. The netstat command accepts either the -f inet or -f inet6 flag to limit the data displayed to either IPv4 or IPv6, respectively. For example, the netstat -f inet6 -rn command displays only IPv6 routing table entries, as opposed to the default, which displays both IPv4 and IPv6 entries.

The netstat -s command displays statistics for all protocols, including IPv6 and ICMPv6. See netstat(1) for more information.

16.4    Displaying and Modifying the Internet (IPv4) to MAC Address Translation Tables

You can display and modify the Internet to Media Access Control (MAC) address translation tables used by the Address Resolution Protocol (ARP) to help diagnose direct IPv4 routing problems resulting from the following circumstances:

• A source host has incorrect Ethernet address information for a destination host

• Two hosts have the same IPv4 address

  Although you can work around this problem by modifying the translation tables, it is best to change one host's IPv4 address to permanently resolve the conflict.

Use the arp -a to display the entries in the Internet to MAC address translation tables. To modify the tables, log in as root and use the arp command as follows:

/usr/sbin/arp [options ] hostname

The following example shows the Ethernet address for an IPv4 host named host1. The system response tells you that the Ethernet address for host1 is aa-00-04-00-8f-11.

# /usr/sbin/arp host1
host1 (16.20.32.2) at aa:0:4:0:8f:11 permanent

The following example shows how to temporarily add host9 to the system translation tables:

# /usr/sbin/arp -s host9 0:dd:0:a:85:0 temp

The following example shows how to remove host8 from the system translation tables:

# /usr/sbin/arp -d host8

See arp(8) for more information on this command.

16.5    Displaying a Datagrams's Route to a Network Host

You can display a datagram's route to a network host to manually test, measure, and manage the network.

To display a datagram's route, use the traceroute command with the following syntax:

traceroute [options...] hostname [packetsize]

Table 16-2 describes some of the traceroute command options.

Table 16-2:  Options to the traceroute Command

The traceroute command sends UDP packets (known as probe packets) to an unused port on the remote host, and listens for ICMP replies from IP routers. It sends the probe packets with a small ttl parameter, which specifies the maximum number of hops a packet can take to reach its destination. The traceroute command starts by specifying a ttl of one hop and it increases the ttl by one for each probe packet it sends. It continues sending probe packets until a packet reaches the destination or until the ttl reaches the maximum number of hops.

In response to each probe packet, traceroute can receive one of the following ICMP messages:

• time exceeded

  The IP router that received the probe packet cannot forward it any further due to the ttl value. This message tells you which IP routers are processing the packets.

• port unreachable

  The probe packet reached its indended destination, but could not access the intended port.

When traceroute sends three probe packets (datagrams) for each ttl setting, it displays a line showing the following:

• ttl

• IP address of the host or router that responded

• Round-trip time of each probe datagram/ICMP response

If multiple IP routers respond to the probe, the traceroute command displays the address of each IP router. If the traceroute command does not elicit a response in 3 seconds (the default wait time), an asterisk (*) is displayed for the probe.

The following example shows a successful traceroute command to host2:

% traceroute host2
traceroute to host2 (555.55.5.5), 30 hops max, 40 byte packets
 1  host3 (555.55.5.1) 2 ms 2 ms 2 ms
 2  host5 (555.55.5.2) 5 ms 6 ms 4 ms
 3  host7 (555.55.5.3) 7 ms 7 ms 6 ms
 4  host2 (555.55.5.5) 12 ms 8 ms 8 ms

The traceroute command with the host argument prints the route that packets take to both IPv4 and IPv6 hosts.

See traceroute(8) for more information about this command and its options.

16.6    Displaying Headers of Packets on the Network

You display packet headers on the network when you want to monitor the network traffic associated with a particular network service. This is usually done to determine whether requests are being received or acknowledged, or to determine the source of network requests, in the case of slow network performance.

Use the tcpdump command to display packet headers for a network interface. This command enables you to specify the interface on which to listen, the direction of the packet transfer, and the type of protocol traffic to display. In addition, it enables you to identify the source of the packet. See tcpdump(8) for more information.

Note

In order to use the tcpdump command, the packetfilter option must be configured into the kernel and the system rebooted. See packetfilter(7) for more information.

16.7    Testing a UUCP Remote Connection

Testing a uucp remote connection can help you diagnose certain UUCP problems; for example, to determine why there is a backlog of transfer requests in the queue.

To test a remote connection, do the following:

1. Log in as root.

2. Change to the /usr/lib/uucp directory by using the cd command.

3. Test the remote connection by using the uutry command, using the following syntax:

   uutry system_name

   The system_name variable names the remote system to contact.

4. Examine the debugging output; the last line contains the status of the transaction. If your local system establishes a connection to the remote system, the debugging output contains a good deal of information. You can press Ctrl/C to stop the uutry shell script.

The uutry command has the following characteristics:

• It is a shell script stored in the /usr/lib/uucp directory.

• It contacts a remote system with debugging turned on. If you are using the UUCP scheduler, uusched, to start uucico automatically at specified intervals, the uutry command overrides the retry time interval specified in the /usr/spool/uucp/.Status/system_name file.

  If you use the uutry command frequently, you can put the pathname to the command in the PATH entry in your .profile file.

• It directs debugging information to a file named /tmp/system_name, where system_name is the name of the local system. The uutry command then executes a tail -f command to display the file's contents.

If your local system cannot contact the remote system, do the following:

1. Validate the physical connections between the local and remote systems. At both systems, confirm that the computer is turned on, that all the cables are properly connected, that the ports are enabled, and the modems (if being used) are working. If the remote system is not at your physical location, contact the administrator of the remote system.

2. Verify all configuration files on both systems. Verify that all entries in the Devices, Systems, and Permissions files are correct. If you are using a modem, verify all entries in the Dialers and Dialcodes files.

   If you are using a TCP/IP connection, verify that the configuration files contain the correct TCP entries. Verify that the inetd daemon can start the uucpd daemon. Edit the /etc/inetd.conf file and delete the comment character (#) from the beginning of the line containing the uucp entry. Restart the inetd daemon by using the following command:

   
   # /sbin/init.d/inetd start
   

Always save the debugging output produced by the uutry command until you are certain that the problem is resolved.

The following example shows a successful test of a remote connection to system host6:

# /usr/lib/uucp/uutry host6
   
.
.
.
Conversation Complete: Status SUCCEEDED

The following example shows an unsuccessful test of a remote connection to system host6:

# /usr/lib/uucp/uutry host6
   
.
.
.
mchFind called (host6)
conn (host6)
getto ret -1
Call Failed: CAN'T ACCESS DEVICE
exit code 101
Conversation Complete: Status FAILED

16.8    Monitoring a UUCP File Transfer

Monitoring a uucp file transfer enables you to diagnose other UUCP problems, especially if you can already establish a remote UUCP connection.

To monitor a file transfer, do the following:

1. Verify the status of the files in the spooling directory on your local system by using the uustat -q command.

2. Verify that the local system can contact the remote system by using the uutry system_name command.

3. If the debugging output indicates that the connection was not successful, follow the steps described in Section 16.7 to test the remote connection..

4. Prepare a file for transfer by using the uucp -r command. The -r option instructs uucp to place the file in the queue without starting the uucico daemon.

   Start the file transfer by using the uutry command.

See uutry(1) for additional information on this command.

The following example sends the test1 file to the system host6:

# uucp -r test1 host6! ~/test1
# /usr/lib/uucp/uutry host6

16.9    Viewing the Error Log File

To diagnose kernel and hardware errors, you can look at the system events that occurred prior to the errors. Messages from system events, such as error messages relating to the software kernel and system hardware, and informational messages about system status, startup, and diagnostics, are recorded in the binary error log file, /var/adm/binary.errlog.

Because this log file is in binary format, the operating system offers special utilities, DECevent and Compaq Analyze, that read the binary log file and run the data through a formatter to display the information. See dia(8) and ca(8) for more information about DECevent and Compaq Analyze, respectively.

Note that these utilities are not available in the operating system by default; you must install the Web-Based Enterprise Services (WEBES) kit, a suite of diagnostic utilities, to obtain them. WEBES is available for installation from the Associated Product CD-ROMs or for download from the following URL:

http://www.support.compaq.com/svctools/webes

See the System Administration guide for information about using the Event Viewer to present errors as interpreted by DECevent and Compaq Analyze. Also, see uerf(8) for an alternative to these utilities.

16.10    Viewing the syslogd Daemon Message Files

You can use the syslogd daemon to help diagnose session layer problems such as access control problems for the Internet Protocol Version 4 (IPv4) and Internet Protocol Version 6 (IPv6).

The syslogd daemon starts running when you boot the system and whenever it receives a hangup signal. By default, it records the system messages for these events in a set of files in the /var/adm/syslog.dated directory (as specified in the /etc/syslog.conf file). The system messages can indicate error conditions or warnings, depending on the priority codes they contain.

Although it is possible to review the contents of the system message files from the command line, it is best to use the Event Viewer that is part of the SysMan Menu utility, because it simplifies access to the files and makes it easier for you to find particular problems. To start the Event Viewer, invoke the SysMan Menu as decribed in Section 1.1.1, then select Monitoring and Tuning-->View events. Alternatively, you can invoke the Event Viewer from a command line by entering the following command:

# /usr/bin/sysman event_viewer

Once the Event Viewer is displayed, you can use it to sort the log entries, filter the entries (for a certain event name, priority level, posting host, or date), and obtain more detailed information about individual entries.

For more information about event management and accessing the system log files, see evm(5), syslogd(8), the System Administration guide, and the online help.