NT4: 16-Bit Applications Within a VDM May Suddenly Fail to Open a Child Window (265801)

SYMPTOMS

A problem may occur where 16-bit applications that share the same virtual MS-DOS machine (VDM) work correctly for a while and then may suddenly fail to create child windows. Any new application that is started within the same VDM may exhibit the same problem. The affected VDM must be halted and restarted to resolve this problem.

This problem was observed only in computers running Microsoft Windows NT version 4.0 (any Service Pack). The problem has not been observed in computers running Microsoft Windows 95, Microsoft Windows 98, or Microsoft Windows 2000.

CAUSE

The selector for the base code segment of a 16-bit module is also its handle. When the module unloads, the selector/handle is used to destroy its classes by using the same handle as the high word and all zeros as the low word. If this combination happens to be identical to the handle of a 32-bit module (which is also its load address) then the classes registered by the 32-bit module are destroyed.

In a test case, a 16-bit module intermittently gets a handle of 77E7, which gets thunked to 77E70000, which in turn happens to map exactly to the User32.dll handle of 77E70000. Therefore, the module unload call for the 16-bit module inadvertently marks User32.dll classes for destruction. This renders such classes as EDIT, COMBO, and so on, unusable. As a result, child windows are not created.

RESOLUTION

The potential solutions involve preventing 16-bit module handles from coinciding with any existing 32-bit modules that are loaded within the VDM process space. The 16-bit selectors appear to always end in either 7 or F (for example, 017F), and therefore a likely overlap is potentially low.

Following are some potential solutions:

Rebase 32-bit modules that meet this criteria to an empty space whose address high word does not end in 7 or F. In the test case, the User32.dll file was rebased from 77E70000 to 79000000 by using Rebase.exe. Rebase.exe ships with the Win32 platform Software Development Kit (SDK). The problem with this solution is that you must rebase the images after every Windows NT Service Pack upgrade or downgrade.
Reduce the number of selectors that are used by the 16-bit application. The 16-bit Microsoft C or Microsoft Visual C++ compilers generate a code segment for each object module (OBJ) when you compile by using the large memory model. You can use Exehdr.exe, which ships with Microsoft C or Visual C++, to view the number of code segments in a 16-bit executable. You can reduce the number of code segments by compiling with /NT and giving the name of a segment, for example, /NT "CSEG1". For Windows applications, the linker default is /PACKCODE. This combines segments with the same name. The maximum size of a segment is 65,500; the linker issues an error if you exceed this limitation.

MORE INFORMATION

NOTE: To set up the checked version of W32k.sys, the checked version of all of the following files is required, because they are dependent on each other: Win32k.sys, User32.dll, Winsrv.dll, Gdi32.dll, Kernel32.dll, and User.exe.

NOTE: To set up the debug environment for NTVDM (SCT equivalent), the checked version of all of the following files is required: Ntvdm.exe, Wow32.dll, Wowexec.exe, Krnl386.exe, and Dosx.exe. Wowexec.exe starts a local debugger whose output is available to the kernel debugger, if one is attached. This facilitates an easy switch between kernel-mode and user-mode debugging.