How to debug a GINA DLL on a single computer (260901)



The information in this article applies to:

  • Microsoft Win32 Application Programming Interface (API), when used with:
    • the operating system: Microsoft Windows NT 4.0
    • the operating system: Microsoft Windows 2000
    • the operating system: Microsoft Windows XP

This article was previously published under Q260901

SUMMARY

Debugging a Graphical Identification and Authentication (GINA) dynamic-link library (DLL) can be a tedious task. The process usually involves setting up a kernel debug session from a remote computer. This article offers an alternative way of debugging a GINA on Windows NT 4.0 and Windows 2000, which allows for source-level debugging on a single computer.

The debugging method described in this article will not work on Windows XP. For more information, see the section below entitled "Windows XP".

MORE INFORMATION

Typically, an application can use the built-in debug features of the operating system by placing a DebugBreak function call within the code, and by the system starting the just-in-time (JIT) debugger.

Unfortunately, a DebugBreak call in the GINA code does not typically work because the JIT debugging mechanism starts the debugger on the default desktop, whereas GINA runs on the Winlogon desktop. To overcome this limitation, you can use the stub application that is generated by the sample code in this article to start the JIT debugger on the Winlogon desktop.

Just-In-Time debugger settings

The operating system starts the JIT debugger based on settings within the registry. The settings are located under the following key:

HKEY_LOCAL_MACHINE\Software\Microsoft\Windows NT\CurrentVersion\AeDebug
					

A typical AeDebug key may contain the following values:

Auto                 REG_SZ       0
Debugger             REG_SZ       C:\mssdk\Bin\windbg.exe -p %ld -e %ld
UserDebuggerHotKey   REG_DWORD    0x00000000
					

If an application does not handle an exception, the system looks in the AeDebug key and starts the debugger that is specified in the "Debugger" value. If "Auto" is set to 1 (one), the system will automatically start the debugger without user intervention; otherwise, the system will query the user as to whether or not they want to debug the application.

When debugging a GINA DLL, it is necessary for the system to start the debugger on the Winlogon desktop. This can be accomplished by a middle-layer stub application. The system starts this small stub, which in turn, starts the debugger on the Winlogon desktop.

The stub presented in this article is referred to as the Aedebug.exe file. The "Auto" value must be set to 1 (one) so that the stub starts automatically. To make Aedebug.exe more flexible, two additional values, "LaunchDebugger" and "Desktop", can be added to the registry. The LaunchDebugger value tells Aedebug.exe what debugger to start. The Desktop value specifies the desktop on which to create the debugger process.

An effective AeDebug key for debugging a GINA DLL might resemble the following:

Auto                 REG_SZ       1
Debugger             REG_SZ       C:\Aedebug.exe -p %ld -e %ld
UserDebuggerHotKey   REG_DWORD    0x00000000
LaunchDebugger       REG_SZ       C:\mssdk\Bin\windbg.exe
Desktop              REG_SZ       winsta0\Winlogon
					

Now all that needs to be done is to place a DebugBreak() in the GINA code and GINA can be debugged with the debugger of choice.

Important Note If you place a DebugBreak function in DllMain, a deadlock may occur.

Windows XP

Unfortunately, this method will not work on Windows XP because DebugBreak is caught by Winlogon and will cause the computer to restart. A possible way to debug your GINA under Windows XP is to create a named event and wait on the event at the point you want to start debugging. After you attach a debugger, you can programmatically signal the event from another application and debug from that point.

Sample code

The following sample code can be used to create Aedebug.exe:
#include <windows.h>
#include <stdio.h>

CHAR szSubKey[] = "Software\\Microsoft\\Windows NT\\CurrentVersion\\AeDebug";
CHAR szDebugValue[] = "LaunchDebugger";
CHAR szDesktopValue[] = "Desktop";


int main(int argc, char *argv[])
{
   HKEY hKey = NULL;
   CHAR szDebugger[MAX_PATH];
   CHAR szCommandLine[MAX_PATH];
   CHAR szDesktop[256];
   BOOL fResult;
   DWORD dwSize;
   LONG lResult;
   STARTUPINFO si;
   PROCESS_INFORMATION pi;
   int n;

   if (argc < 2)
   {
      return 0;
   }

   __try
   {
      ZeroMemory(&pi, sizeof(pi));

      lResult = RegOpenKeyEx(HKEY_LOCAL_MACHINE, 
                             szSubKey, 
                             0,
                             KEY_ALL_ACCESS, 
                             &hKey);
      if (lResult != ERROR_SUCCESS)
      {
         __leave;
      }

      dwSize = MAX_PATH;
      lResult = RegQueryValueEx(hKey, 
                                szDebugValue, 
                                NULL, 
                                NULL, 
                                (LPBYTE)szDebugger, 
                                &dwSize);
      if (lResult != ERROR_SUCCESS)
      {
         __leave;
      }

      dwSize = 256;
      lResult = RegQueryValueEx(hKey, 
                                szDesktopValue, 
                                NULL, 
                                NULL,
                                (LPBYTE)szDesktop, 
                                &dwSize);
      if (lResult != ERROR_SUCCESS)
      {
         __leave;
      }

      lstrcpy(szCommandLine, szDebugger);

      for (n = 1; n < argc; n++)
      {
         lstrcat(szCommandLine, " ");
         lstrcat(szCommandLine, argv[n]);
      }

      ZeroMemory(&si, sizeof(si));
      si.cb = sizeof(si);
      si.lpDesktop = szDesktop;

      fResult = CreateProcess(NULL, 
                              szCommandLine, 
                              NULL,
                              NULL, 
                              TRUE, 
                              0, 
                              NULL,
                              NULL,
                              &si, 
                              &pi);
      if (!fResult)
      {
         __leave;
      }
   }
   __finally
   {
      if (hKey) RegCloseKey(hKey);
      if (pi.hProcess) CloseHandle(pi.hProcess);
      if (pi.hThread) CloseHandle(pi.hThread);
   }

   return 1;
}
				

Modification Type:MajorLast Reviewed:7/20/2005
Keywords:kbBug kbDebug kbGINA kbhowto kbKernBase kbSecurity KB260901