When the operating system encounters an irrecoverable error running
a program, the system creates a file named core
and places it in the current directory. The core file is not an
executable file; it is a snapshot of the state of your program at
the time the error occurred. It allows you to analyze the program at
the point it crashed. This chapter describes a technique you can use
to debug core files using the Digital
Ladebug debugger.
Kernel debugging can be very helpful in analyzing kernel crash dumps. For information on kernel debugging, see Chapter 22.
You can use the debugger to examine the program information in a
core file. Use the following ladebug command syntax to
invoke the debugger on a core file:
ladebug executable_file core_file
Specify the name of the executable file (the program that was running at the time the core file was generated) in addition to the core file.
When debugging a core file, use the debugger to obtain a stack trace and the values of a few variables.
The stack trace lists the functions in your program that were active
when the dump occurred. By examining the values of a few program
variables along with the stack trace, you should be able to pinpoint
the program state and the cause of the core dump. (Core files cannot
be executed, and so a rerun, step, cont, etc. will not
work until a run is issued.)
In addition, if the program is multithreaded, you can examine
the native (that is, kernel-level) thread information with the
show thread and thread commands. You can
examine the stack trace for a particular thread or all threadswith
the where thread command. (See Section 14.3. Also see Chapter 19.)
The program shown in Example 14-1 is
almost identical to the program used in Chapter 7 but adds a null pointer reference in the
factorial function. This reference causes the program
to abort and dump the core when it is executed. The dump
command prints the value of the x variable
as a null, and the print *x command reveals that you
cannot dereference a null pointer.
% cat testProgram.c
main() {
int i,f;
for (i=1 ; i<3 ; i++) {
f = factorial(i);
printf("%d! = %d\en",i,f);
}
}
factorial(i)
int i;
{
int *x;
x = 0;
printf("%d",*x);
if (i<=1)
return (1);
else
return (i * factorial(i-1) );
}
% cc -o testProgram -g testProgram.c
% testProgram
Memory fault - core dumped.
% ladebug testProgram core
Welcome to the Ladebug Debugger Version 4.0
------------------
object file name: testProgram
core file name: core
Reading symbolic information ...done
Core file produced from executable testProgram
Thread terminated at PC 0x120000dc4 by signal SEGV
(ladebug) where
>0 0x120000dc4 in factorial(i=1) testProgram.c:13
#1 0x120000d44 in main() testProgram.c:4
(ladebug) dump
>0 0x120000dc4 in factorial(i=1) testProgram.c:13
printf("%d",*x);
(ladebug) print *x
Cannot dereference 0x0
Error: no value for *x
(ladebug)
Ladebug can debug core files generated by multithreaded
applications. For this kind of core file debugging, Ladebug sets
the $threadlevel variable to native in
the following initializations:
load command is issued for core file
debugging.
process command is used for core
file debugging.
$threadlevel variable correctly in the
following case: Process A, not a core file, is attached with the
$threadlevel set to native. Switching
to Process B and then back to Process A resets the variable to
decthreads rather than native.
Example 14-2 presents an example of core file kernel thread debugging.
Welcome to the Ladebug Debugger Version 4.0-10 ------------------ object file name: ../bin/c_threadcore Reading symbolic information ...done (ladebug) record io out (ladebug) unload (ladebug) load ../bin/c_threadcore ../bin/c_threadcore_core Reading symbolic information ...done Core file produced from executable c_threadcore Thread 0xffffffff836934d0 terminated at PC 0x3ff805065c8 by signal SEGV 1 (ladebug) print $threadlevel2 "native" (ladebug) show thread3 Thread # Id State > 1 0xffffffff836934d0 dead 2 0xffffffff83596120 dead (ladebug) thread4 Thread 0x1 (ladebug) where5 >0 0x3ff805065c8 in __kill(0xb, 0x2, 0x3, 0x47c18, 0x4, 0x6) ../../../../../src/usr/ccs/lib/libc/alpha/kill.s:41 #1 0x3ff8055c560 in exc_raise(0x3ffc01e7e78, 0x0, 0x1, 0x0, 0x309a9a07, 0xb97c60) ../../../../../src/usr/ccs/lib/DECthreads/COMMON/ exc_handling.c:649 #2 0x3ff8055c764 in exc_pop_ctx(0x3ffc01e7bd0, 0x0, 0x0, 0x3ff80567b40, 0x0, 0x0) ../../../../../src/usr/ccs/lib/DECthreads/COMMON/ exc_handling.c:805 #3 0x3ff8056e374 in cma__thread_base(0x0, 0x0, 0x0, 0x0, 0x0, 0x1200136e0) ../../../../../src/usr/ccs/lib/DECthreads/COMMON/ cma_thread.c:1623 (ladebug) thread 26 Thread 0x2 (ladebug) where7 >0 0x3ff8051df04 in msg_receive_trap(0xe56b0, 0x0, 0x3ff804b500c, 0x100000000000, 0x3ff804b5058, 0x18157f0d0d) /usr/sde/osf1/build/ goldminos.bld/export/alpha/usr/include/mach/syscall_sw.h:74 #1 0x3ff80514764 in msg_receive(0xe0df0, 0x0, 0x0, 0x240000000000000, 0x61746164702e, 0x3ff800b9fe0) ../../../../../src/usr/ccs/lib/libmach /msg.c:95 #2 0x3ff80574b00 in cma__vp_sleep(0x2801000000, 0x3ff00000001, 0x6, 0x3ffc00841a0, 0x3ff00000000, 0x3ffc00960c0) ../../../../../src/usr/ ccs/lib/DECthreads/COMMON/cma_vp.c:1588 #3 0x3ff8055ae7c in cma__dispatch(0x6, 0x3ffc00841a0, 0x3ff00000000, 0x3ffc00960c0, 0x3ff80554354, 0x3ffc0080468) ../../../../../src/usr/ccs/ lib/DECthreads/COMMON/cma_dispatch.c:998 #4 0x3ff80554354 in cma__int_wait(0x11ffff428, 0x4ac18, 0x3ffc01de9f8, 0x70000003, 0x3ff80485192, 0x3ff80089931) ../../../../../src/usr/ccs/ lib/DECthreads/COMMON/cma_condition.c:2651 #5 0x3ff8056d704 in cma_thread_join(0x11ffff848, 0x11ffffbf0, 0x11ffffbe8, 0x3ff8056e110, 0x0, 0x3ffc01e3de0) ../../../../../src/ usr/ccs/lib/DECthreads/COMMON/cma_thread.c:955 #6 0x3ff80563a2c in pthread_join(0x47c18, 0x400000000000002, 0x11ffffc68, 0x3ffc01de9f8, 0x120013d18, 0x5) ../../../../../src/usr/ ccs/lib/DECthreads/COMMON/cma_pthread.c:2270 #7 0x120013d70 in main() thread_core.c:92 (ladebug) where thread all8 Stack trace for thread 1 #0 0x3ff805065c8 in __kill(0xb, 0x2, 0x3, 0x47c18, 0x4, 0x6) ../../../../ ../src/usr/ccs/lib/libc/alpha/kill.s:41 #1 0x3ff8055c560 in exc_raise(0x3ffc01e7e78, 0x0, 0x1, 0x0, 0x309a9a07, 0xb97c60) ../../../../../src/usr/ccs/lib/DECthreads/COMMON/exc_handling.c:649 #2 0x3ff8055c764 in exc_pop_ctx(0x3ffc01e7bd0, 0x0, 0x0, 0x3ff80567b40, 0x0, 0x0) ../../../../../src/usr/ccs/lib/DECthreads/COMMON/ exc_handling.c:805 #3 0x3ff8056e374 in cma__thread_base(0x0, 0x0, 0x0, 0x0, 0x0, 0x1200136e0) ../../../../../src/usr/ccs/lib/DECthreads/ COMMON/cma_thread.c:1623 Stack trace for thread 2 >0 0x3ff8051df04 in msg_receive_trap(0xe56b0, 0x0, 0x3ff804b500c, 0x100000000000, 0x3ff804b5058, 0x18157f0d0d) /usr/sde/osf1/build/ goldminos.bld/export/alpha/usr/include/mach/syscall_sw.h:74 #1 0x3ff80514764 in msg_receive(0xe0df0, 0x0, 0x0, 0x240000000000000, 0x61746164702e, 0x3ff800b9fe0) ../../../../../src/usr/ccs/lib/libmach/ msg.c:95 #2 0x3ff80574b00 in cma__vp_sleep(0x2801000000, 0x3ff00000001, 0x6, 0x3ffc00841a0, 0x3ff00000000, 0x3ffc00960c0) ../../../../../src/usr/ ccs/lib/DECthreads/COMMON/cma_vp.c:1588 #3 0x3ff8055ae7c in cma__dispatch(0x6, 0x3ffc00841a0, 0x3ff00000000, 0x3ffc00960c0, 0x3ff80554354, 0x3ffc0080468) ../../../../../src/usr/ ccs/lib/DECthreads/COMMON/cma_dispatch.c:998 #4 0x3ff80554354 in cma__int_wait(0x11ffff428, 0x4ac18, 0x3ffc01de9f8, 0x70000003, 0x3ff80485192, 0x3ff80089931) ../../../../../src/usr/ccs/ lib/DECthreads/COMMON/cma_condition.c:2651 #5 0x3ff8056d704 in cma_thread_join(0x11ffff848, 0x11ffffbf0, 0x11ffffbe8, 0x3ff8056e110, 0x0, 0x3ffc01e3de0) ../../../../../src/usr/ccs/lib #6 0x3ff80563a2c in pthread_join(0x47c18, 0x400000000000002, 0x11ffffc68, 0x3ffc01de9f8, 0x120013d18, 0x5) ../../../../../src/usr/ #7 0x120013d70 in main() thread_core.c:92 (ladebug) thread 19 Thread 0x1 (ladebug) printregs10 $r0 [$v0] = 0 $r1 [$t0] = 0 $r2 [$t1] = 4396974517024 $r3 [$t2] = 320024 $r4 [$t3] = 4396974517024 $r5 [$t4] = 0 $r6 [$t5] = 0 $r7 [$t6] = 130 $r8 [$t7] = 0 $r9 [$s0] = 2 $r10 [$s1] = 2 $r11 [$s2] = 0 $r12 [$s3] = 0 $r13 [$s4] = 0 $r14 [$s5] = 0 $r15 [$s6] = 0 $r16 [$a0] = 28613 $r17 [$a1] = 11 $r18 [$a2] = 4396974766704 $r19 [$a3] = 0 $r20 [$a4] = 0 $r21 [$a5] = 4 $r22 [$t8] = 3 $r23 [$t9] = 0 $r24 [$t10] = 3 $r25 [$t11] = 0 $r26 [$ra] = 4395904648544 $r27 [$t12] = 4395904749552 $r28 [$at] = -1 $r29 [$gp] = 4396974751200 $r30 [$sp] = 4396974766800 $r31 [$zero]= 0 $f0 = 0 $f1 = 0 $f2 = 0 $f3 = 0 $f4 = 0 $f5 = 0 $f6 = 0 $f7 = 0 $f8 = 0 $f9 = 0 $f10 = 0 $f11 = 0 $f12 = 0 $f13 = 0 $f14 = 0 $f15 = 0 $f16 = 0 $f17 = 0 $f18 = 0 $f19 = 0 $f20 = 0 $f21 = 0 $f22 = 0 $f23 = 0 $f24 = 0 $f25 = 0 $f26 = 0 $f27 = 0 $f28 = 0 $f29 = 0 $f30 = 0 $f31 = 0 $pc = 0x3ff805065c8 (ladebug) thread 2 Thread 0x2 (ladebug) printregs $r0 [$v0] = -207 $r1 [$t0] = 0 $r2 [$t1] = 0 $r3 [$t2] = 0 $r4 [$t3] = 0 $r5 [$t4] = 0 $r6 [$t5] = 1 $r7 [$t6] = 1 $r8 [$t7] = 0 $r9 [$s0] = 4831834496 $r10 [$s1] = 0 $r11 [$s2] = 4396974730048 $r12 [$s3] = 0 $r13 [$s4] = 0 $r14 [$s5] = 162129586585337856 $r15 [$s6] = 0 $r16 [$a0] = 4831834496 $r17 [$a1] = 0 $r18 [$a2] = 40 $r19 [$a3] = 6 $r20 [$a4] = 0 $r21 [$a5] = 6 $r22 [$t8] = 4 $r23 [$t9] = 2 $r24 [$t10] = 1 $r25 [$t11] = 0 $r26 [$ra] = 4395904354148 $r27 [$t12] = 4395904050272 $r28 [$at] = 293938 $r29 [$gp] = 4396974651024 $r30 [$sp] = 4831834336 $r31 [$zero]= 0 $f0 = 0 $f1 = 0 $f2 = 0 $f3 = 0 $f4 = 0 $f5 = 0 $f6 = 0 $f7 = 0 $f8 = 0 $f9 = 0 $f10 = 3.237908616585193e-319 $f11 = 65536 $f12 = 0 $f13 = 0 $f14 = 1.016984725399622e-319 $f15 = 20584 $f16 = 0 $f17 = 0 $f18 = 0 $f19 = 0 $f20 = 0 $f21 = 0 $f22 = 3.237908616585193e-319 $f23 = 65536 $f24 = 0 $f25 = 0 $f26 = 0 $f27 = 0 $f28 = 0 $f29 = 0 $f30 = 0 $f31 = 0 $pc = 0x3ff8051df04 (ladebug)
$threadlevel is set to native
and you can begin core file kernel thread debugging.
where thread all ,
Ladebug shows the stack trace for all threads. Notice that it
labels the stack trace for each thread.
thread command with the
thread ID, Ladebug changes the thread context to that thread (in
this case, thread 1).
printregs command asks Ladebug to show
the register contents within the current thread context.