Previous | Index | Next 

[PRB] VB6 applications that use window subclassing or other API callback methods throw a CallbackOnCollectedDelegate exception

VB6 applications typically implement window subclassing by using the SetWindowLong API function to replace the address of a window’s default procedure with the address of a method defined in the application. Such a method receives four arguments – the handle of the window, the message number, plus two 32-bit integers whose meaning depends on the specific message – and returns a 32-bit integer. This is the typical VB6 code that implements this technique:

Private Declare Function SetWindowLong Lib "user32" Alias "SetWindowLongA" ( _
	ByVal hWnd As Long, ByVal ndx As Long, ByVal newValue As Long) As Long
Private Declare Function CallWindowProc Lib "user32" Alias "CallWindowProcA" ( _
	ByVal lpPrevWndFunc As Long, ByVal hWnd As Long, ByVal Msg As Long, _
	ByVal wParam As Long, ByVal lParam As Long) As Long
' This is used with the SetWindowLong API function.
Const GWL_WNDPROC = -4

Dim saveHWnd As Long        ' The handle of the subclassed window.
Dim oldProcAddr As Long     ' The address of the original window procedure

Sub StartSubclassing(ByVal hWnd As Long)
    saveHWnd = hWnd
    ' replace address of window procedure
    oldProcAddr = SetWindowLong(hWnd, GWL_WNDPROC, AddressOf WndProc)
End Sub

Sub StopSubclassing()
    ' restore original window procedure
    SetWindowLong saveHWnd, GWL_WNDPROC, oldProcAddr
End Sub

Function WndProc(ByVal hWnd As Long, ByVal uMsg As Long, _
         ByVal wParam As Long, ByVal lParam As Long) As Long
    ' Send the message to the original window procedure, and prepare to 
    ' return Windows the return value from the original procedure.
    WndProc = CallWindowProc(oldProcAddr, hWnd, uMsg, wParam, lParam)
    
    ' process the message
    Select Case uMsg
        ' (omitted)
    End Select
End Function

When the AddressOf operator is used in a call to a Declare method (SetWindowLong, in this case), VB Migration Partner defines a delegate class that matches the target procedure’s signature (WndProc, in this case) and defines an overload of the Declare method with a delegate parameter instead of an integer parameter:

Public Delegate Function SetWindowLong_CBK(ByVal hWnd As Integer, ByVal uMsg As Integer,_
       ByVal wParam As Integer, ByVal lParam As Integer) As Integer

Friend Module SubclassingAPI
    Private Declare Function SetWindowLong Lib "user32" Alias "SetWindowLongA" _
        (ByVal hWnd As Integer, ByVal ndx As Integer, _
        ByVal newValue As Integer) As Integer
    Private Declare Function SetWindowLong Lib "user32" Alias "SetWindowLongA" _
            (ByVal hWnd As Integer, ByVal ndx As Integer, _
            ByVal newValue As SetWindowLong_CBK) 
    ' ...
    Public Sub StartSubclassing(ByVal hWnd As Integer)
        saveHWnd = hWnd
        oldProcAddr = SetWindowLong(hWnd, GWL_WNDPROC, AddressOf WndProc)
    End Sub
    ' ...
End Module

This converted code would work correctly, except for a subtle detail: the VB code creates a SetWindowLong_CBK delegate object and passes it to the SetWindowLong method, without storing it in a variable. The problem is: when the garbage collector fires, the delegate object is reclaimed and the application throws a CallbackOnCollectedDelegate exception as soon as it attempts to invoke the delegate.

This problem can appear with other API-related technique that uses callback methods, for example with the applications that set up keyboard handlers by means of the SetWindowsHook or SetWindowsHookEx API methods.

Fortunately, the solution is simple: instead of creating the delegate object and pass it to the API method on-the-fly, the VB.NET code should store it in a class-level field, so that the delegate is kept alive and protected from garbage collections. You can achieve this behavior in a number of ways, the simplest one being a set of InsertStatement pragma added to the original VB6 code:

Dim saveHWnd As Long        ' The handle of the subclassed window.
Dim oldProcAddr As Long     ' The address of the original window procedure
'## InsertStatement Dim newProcAddr As SetWindowLong_CBK

Sub StartSubclassing(ByVal hWnd As Long)
    saveHWnd = hWnd
    '## InsertStatement newProcAddr = AddressOf WndProc
    '## ReplaceStatement oldProcAddr = SetWindowLong(hWnd, GWL_WNDPROC, newProcAddr)
    oldProcAddr = SetWindowLong(hWnd, GWL_WNDPROC, AddressOf WndProc)
End Sub

If the Declare method is called by many places of the application, you might want to implement a different solution, based on wrappers for the Declare method that takes the callback value. The purpose of such wrappers is storing the delegate value in a collection, which indirectly protects it from garbage collections.

' this is the original Declare, renamed and made Private
Private Declare Function SetWindowLong_Private Lib "user32" Alias "SetWindowLongA"_
        (ByVal hWnd As Integer, ByVal ndx As Integer, _
        ByVal newValue As SetWindowLong_CBK) As Integer

' this is the wrapper method
Public Declare Function SetWindowLong(ByVal hWnd As Integer, _
        ByVal ndx As Integer, ByVal newValue As SetWindowLong_CBK) As Integer
    ' add to a collection that keeps the delegate alive
    ' (KeepAliveObjects6 is defined in CodeArchitects.VBLibrary.dll)
    KeepAliveObjects6.Add(newValue)
    ' call the original (renamed) Declare method
    Return SetWindowLong_Private(hWnd, ndx, newValue)
End Function

VB Migration Partner is able to automatically generate such wrapper methods for you. All you need is applying a WrapDeclareWithCallbacks pragma to the class or project level.

 

Previous | Index | Next