How To Debug a System-Wide Hook (102428)

SUMMARY

Debugging a system-wide hook such as a journal hook must be done with the extreme caution. When an application installs such a hook, it effectively takes control of user input. In effect, this disables the interface with the debugger. For example, after installing a journal record hook, you must unhook the record hook when you want to allow the debugger to regain control.

It is not possible to use an interactive debugger to debug an actively installed journal hook using a single machine. It is possible to use a remote debugger, because one interface can be blocked (or recording) while the other one does the debugging.

MORE INFORMATION

System-wide input hook procedures can be thought of as being in three possible states:

   unhooked (not installed)
   suspended
   hooked (installed)

In the unhooked state, the procedure imposes no control over user input. In the hooked state, all user input specifically defined to be handled by this hook passes through this procedure. In the suspended state, all user input specifically defined to be handled by this hook is completely blocked.

In the case of a journal record hook, the suspended state can be achieved when a breakpoint is reached within the hook procedure. When this happens, all user input (system wide, that is) in the form of mouse and keyboard input is blocked, and thus you cannot interact with the debugger or any other application as you normally would. Fortunately, when the user presses the CTRL+ESC or the CTRL+ALT+DEL key combinations, all system-wide hooks are automatically unhooked, returning the system to the unhooked state.

Once this has occurred, it is likely that the application with the journal hook is now in a undefined state (because it had the hook pulled out from underneath it, so to speak). Fortunately, the system will send all applications the WM_CANCELJOURNAL message to indicate that it has removed the hook. A well behaved application can intercept this message and adjust its state accordingly.