VB "Bad DLL Calling Convention" Means Stack Frame Mismatch (85108)

SUMMARY

When you call a dynamic link library (DLL) function from Visual Basic for Windows, the "Bad DLL Calling Convention" error is often caused by incorrectly omitting or including the ByVal keyword from the Declare statement or the Call statement. The ByVal keyword affects the size of data placed on the stack. Visual Basic for Windows checks the change in the position of the stack pointer to detect this error.

When Visual Basic for Windows generates the run time error "Bad DLL Calling Convention," the most common cause when calling API functions is omitting the ByVal keyword from the Declaration of the external function or from the call itself. It can also occur due to including the ByVal keyword when the function is expecting a 4 byte pointer to the parameter instead of the value itself. This changes the size (number of bytes) of the values placed on the stack, and upon return from the DLL, Visual Basic for Windows detects the change in the position of the stack frame and generates the error.

MORE INFORMATION

There are two calling conventions, or inter-language protocols: the Pascal/Basic/FORTRAN calling convention, and the C calling convention. Visual Basic for Windows uses the Pascal calling convention, as do the Microsoft Window API functions and other Microsoft Basic language products. Under the Pascal convention, it is the responsibility of the called procedure to adjust or clean the stack. (In addition, parameters are pushed onto the stack in order from the leftmost parameter to the rightmost.) Because the DLL function is responsible for adjusting the stack based on the type and number of parameters it expects, Visual Basic for Windows checks the position of the stack pointer upon return from the function. If the called routine has adjusted the stack to an unexpected position, then Visual Basic for Windows generates a "Bad DLL Calling Convention" error. Visual Basic for Windows assumes a stack position discrepancy because the DLL function uses the C calling convention. With the C calling convention, the calling program is responsible for adjusting the stack immediately after the called routine returns control.

Steps to Reproduce Behavior

Create a simple DLL using Microsoft Quick C for Windows or any compiler capable of creating Windows DLLs. The following example is in C and written for Quick C for Windows:

STACKING.C

#include <windows.h>

long far pascal typecheck (long a, float b, short far *c, char far *buff)
{
short retcode;
a = a * 3;
retcode = MessageBox(NULL, "I am in the DLL", "BOX", MB_OK);
return (a);
}

STACKING.DEF

LIBRARY      STACKING
EXETYPE     WINDOWS
STUB        'winstub.exe'
STACKSIZE   5120
HEAPSIZE    1024
DATA PRELOAD MOVEABLE SINGLE        ; ADD THESE TWO LINES
CODE PRELOAD MOVEABLE DISCARDABLE   ; TO AVOID WARNINGS.
EXPORTS
    typecheck   @1
     WEP           @2

Add the following code to the general Declarations module in a Visual Basic for Windows form:

Declare Function typecheck Lib "d\stacking.dll" (ByVal a As Long,
      ByVal b As Single, c As Integer, ByVal s As String) As Long

NOTE: The above declaration must be placed on one line.

In the Form_Click event:

Sub Form_Click ()
Dim a As Long   ' Explicitly type the variables.
Dim b As Single
Dim c As Integer
Dim s As String
a = 3                      ' Initialize the variables.
b = 4.5
c = 6
s = "Hello there! We've been waiting for you!"
Print typecheck(a, b, c, s)
End Sub

Running the program as written above will not generate the error. Now add the ByVal keyword before the variable named c in the Visual Basic for Windows Declaration. Run the program. Note that the MessageBox function pops a box first, and then the error box pops up indicating that Visual Basic for Windows checks the stack upon return to see if it has been correctly adjusted. Because the DLL expected a 4-byte pointer and received a 2-byte value, the stack has not adjusted back to the initial frame.

As another test, first remove the ByVal keyword before the variable 'c' that you added in the previous test. Declare the parameter 'a As Any' instead of As Long. Change the type of the variable 'a' in the Form_Click to Integer. Run the program again. Using As Any turns off type checking by Visual Basic for Windows. Because the program passed an integer ByVal instead of the long that the DLL expected, the stack frame is off and the error is generated.