FIX: Method with BOOL* Parameter Type Overwriting Memory in Visual Basic (199315)

SYMPTOMS

Calling a Microsoft Foundation Class (MFC) COM Automation Object method with a Boolean* parameter type destroys the content of a variable in Visual Basic. For instance, see the following code:

   void MyTest::Boo(BOOL FAR* p1);

Where MyTest is an automation object in a MFC DLL (created with Class Wizard; a CCmdTarget-derived class with Creatable by Type ID option), and Boo is a method that has BOOL* as parameter type. Calling Boo in a Visual Basic application overwrites memory in Visual Basic.

RESOLUTION

Use VARIANT_BOOL* data type instead. VARIANT_BOOL is defined in OLE and is language-independent. VARIANT_BOOL is 2 bytes in length. However, the MFC Class Wizard does not show VARIANT_BOOL* in its drop-down list box as a choice for the parameter type. A workaround is to select BOOL* as the parameter's type as usual. Then manually change BOOL FAR* to VARIANT_BOOL FAR* in three files where this parameter is declared: .h/.cpp for automation object class and .odl file of the project. In addition, move the function prototype of the method out of the Class Wizard comments blocks because Class Wizard doesn't recognize the VARIANT_BOOL type. For example see the following code sample: In the .h file of MyTest class:

   class MyTest : public CCmdTarget
   {   
       // ... other stuff

       // Generated OLE dispatch map functions
       //{{AFX_DISPATCH(MyTest)
       // ... other stuff
       //}}AFX_DISPATCH
       afx_msg void Boo(VARIANT_BOOL FAR* p1);// USE VARIANT_BOOL and 
                                              // MOVE OUT OF AFX_DISPATCH BLOCK!
   };

In the .cpp file of MyTest class:

   void MyTest::Boo(VARIANT_BOOL FAR* p1) // USE VARIANT_BOOL!
   {
      // TODO: Add your dispatch handler code here.
   }

In the project's .odl file:

   // ... other stuff
   dispinterface IMyTest
   {
       // ... other stuff

       methods:
          // NOTE: ClassWizard maintains method information here.
          //    Use extreme caution when editing this section.
          //{{AFX_ODL_METHOD(MyTest)
          // ... other stuff
          //}}AFX_ODL_METHOD
      [id(1)] void Boo(VARIANT_BOOL* p1);// USE VARIANT_BOOL and MOVE THIS
                                         // OUT OF AFX_ODL_METHOD BLOCK!
   };

MORE INFORMATION

Steps to Reproduce Behavior

Create a new MFC AppWizard DLL project with all the default settings and Automation support. Name the project AutoObj.
Use Class Wizard to create a new CCmdTarget-derived class with Createable by Type ID option selected. Name this class MyTest.
Use Class Wizard to add a method to the IMyTest interface with a parameter whose type is Bool*. Name the method Boo.

Implement the method as shown below:

   void MyTest::Boo(BOOL FAR* p1)
   {
      // TODO: Add your dispatch handler code here
      *p1 = TRUE;
   }

Compile it and register AutoObj.dll.

Start Visual Basic then Create a Standard EXE project and add a command button to the form and use the following code to test the Automation object you created above:

   Private Sub Command1_Click()
      Dim a As Integer
      Dim b As Boolean
      a = 123
      b = False
      Set test = CreateObject("AutoObj.MyTest")
      test.Boo b
      Debug.Print a, b    '<--- a is 0 - MEMORY IS TRASHED!
   End Sub

Notice that the value of variable "a" is changed to "0" after the Boo method on the test object has been called. Print output can be seen in immediate window.