VB Migration Partner

KNOWLEDGE BASE - Language


Previous | Index | Next 

[PRB] VB6 applications that use window subclassing or other API callback methods throw a CallbackOnCollectedDelegate exception

VB6 applications typically implement window subclassing by using the SetWindowLong API function to replace the address of a window’s default procedure with the address of a method defined in the application. Such a method receives four arguments – the handle of the window, the message number, plus two 32-bit integers whose meaning depends on the specific message – and returns a 32-bit integer. This is the typical VB6 code that implements this technique:

Private Declare Function SetWindowLong Lib "user32" Alias "SetWindowLongA" ( _
	ByVal hWnd As Long, ByVal ndx As Long, ByVal newValue As Long) As Long
Private Declare Function CallWindowProc Lib "user32" Alias "CallWindowProcA" ( _
	ByVal lpPrevWndFunc As Long, ByVal hWnd As Long, ByVal Msg As Long, _
	ByVal wParam As Long, ByVal lParam As Long) As Long
' This is used with the SetWindowLong API function.
Const GWL_WNDPROC = -4

Dim saveHWnd As Long        ' The handle of the subclassed window.
Dim oldProcAddr As Long     ' The address of the original window procedure

Sub StartSubclassing(ByVal hWnd As Long)
    saveHWnd = hWnd
    ' replace address of window procedure
    oldProcAddr = SetWindowLong(hWnd, GWL_WNDPROC, AddressOf WndProc)
End Sub

Sub StopSubclassing()
    ' restore original window procedure
    SetWindowLong saveHWnd, GWL_WNDPROC, oldProcAddr
End Sub

Function WndProc(ByVal hWnd As Long, ByVal uMsg As Long, _
         ByVal wParam As Long, ByVal lParam As Long) As Long
    ' Send the message to the original window procedure, and prepare to 
    ' return Windows the return value from the original procedure.
    WndProc = CallWindowProc(oldProcAddr, hWnd, uMsg, wParam, lParam)
    
    ' process the message
    Select Case uMsg
        ' (omitted)
    End Select
End Function

When the AddressOf operator is used in a call to a Declare method (SetWindowLong, in this case), VB Migration Partner defines a delegate class that matches the target procedure’s signature (WndProc, in this case) and defines an overload of the Declare method with a delegate parameter instead of an integer parameter:

Public Delegate Function SetWindowLong_CBK(ByVal hWnd As Integer, ByVal uMsg As Integer,_
       ByVal wParam As Integer, ByVal lParam As Integer) As Integer

Friend Module SubclassingAPI
    Private Declare Function SetWindowLong Lib "user32" Alias "SetWindowLongA" _
        (ByVal hWnd As Integer, ByVal ndx As Integer, _
        ByVal newValue As Integer) As Integer
    Private Declare Function SetWindowLong Lib "user32" Alias "SetWindowLongA" _
            (ByVal hWnd As Integer, ByVal ndx As Integer, _
            ByVal newValue As SetWindowLong_CBK) 
    ' ...
    Public Sub StartSubclassing(ByVal hWnd As Integer)
        saveHWnd = hWnd
        oldProcAddr = SetWindowLong(hWnd, GWL_WNDPROC, AddressOf WndProc)
    End Sub
    ' ...
End Module

This converted code would work correctly, except for a subtle detail: the VB code creates a SetWindowLong_CBK delegate object and passes it to the SetWindowLong method, without storing it in a variable. The problem is: when the garbage collector fires, the delegate object is reclaimed and the application throws a CallbackOnCollectedDelegate exception as soon as it attempts to invoke the delegate.

This problem can appear with other API-related technique that uses callback methods, for example with the applications that set up keyboard handlers by means of the SetWindowsHook or SetWindowsHookEx API methods.

Fortunately, the solution is simple: instead of creating the delegate object and pass it to the API method on-the-fly, the VB.NET code should store it in a class-level field, so that the delegate is kept alive and protected from garbage collections. You can achieve this behavior in a number of ways, the simplest one being a set of InsertStatement pragma added to the original VB6 code:

Dim saveHWnd As Long        ' The handle of the subclassed window.
Dim oldProcAddr As Long     ' The address of the original window procedure
'## InsertStatement Dim newProcAddr As SetWindowLong_CBK

Sub StartSubclassing(ByVal hWnd As Long)
    saveHWnd = hWnd
    '## InsertStatement newProcAddr = AddressOf WndProc
    '## ReplaceStatement oldProcAddr = SetWindowLong(hWnd, GWL_WNDPROC, newProcAddr)
    oldProcAddr = SetWindowLong(hWnd, GWL_WNDPROC, AddressOf WndProc)
End Sub

If the Declare method is called by many places of the application, you might want to implement a different solution, based on wrappers for the Declare method that takes the callback value. The purpose of such wrappers is storing the delegate value in a collection, which indirectly protects it from garbage collections.

' this is the original Declare, renamed and made Private
Private Declare Function SetWindowLong_Private Lib "user32" Alias "SetWindowLongA"_
        (ByVal hWnd As Integer, ByVal ndx As Integer, _
        ByVal newValue As SetWindowLong_CBK) As Integer

' this is the wrapper method
Public Declare Function SetWindowLong(ByVal hWnd As Integer, _
        ByVal ndx As Integer, ByVal newValue As SetWindowLong_CBK) As Integer
    ' add to a collection that keeps the delegate alive
    ' (KeepAliveObjects6 is defined in CodeArchitects.VBLibrary.dll)
    KeepAliveObjects6.Add(newValue)
    ' call the original (renamed) Declare method
    Return SetWindowLong_Private(hWnd, ndx, newValue)
End Function

VB Migration Partner is able to automatically generate such wrapper methods for you. All you need is applying a WrapDeclareWithCallbacks pragma to the class or project level.

Previous | Index | Next 




Follow Francesco Balena on VB6 migration’s group on

LinkedIn





Read Microsoft Corp’s official case study of a VB6 conversion using VB Migration Partner.




Code Architects and its partners offers remote and onsite migration services.

More details




Subscribe to our free newsletter for useful VB6 migration tips and techniques.

newsletter



To learn more about your VB6 applications, run VB6 Analyzer on your source code and send us the generated text file. You will receive a detailed report about your VB6 applications and how VB Migration Partner can help you to quickly and effectively migrate it to .NET.

Get free advice



A fully-working, time-limited Trial Edition of VB Migration Partner allows you to test it against your actual code

Get the Trial




The price of VB Migration Partner depends on the size of the VB6 application, the type of license, and other factors

Request a quote




Migrating a VB6 application in 10 easy steps

Comparing VB Migration Partner with Upgrade Wizard

Migration tools: Feature Comparison Table

All whitepapers