VB6 applications typically implement window subclassing by using the SetWindowLong API function to replace the address of a window’s default procedure with the address of a method defined in the application. Such a method receives four arguments – the handle of the window, the message number, plus two 32-bit integers whose meaning depends on the specific message – and returns a 32-bit integer. This is the typical VB6 code that implements this technique:
Private Declare Function SetWindowLong Lib "user32" Alias "SetWindowLongA" ( _
ByVal hWnd As Long, ByVal ndx As Long, ByVal newValue As Long) As Long
Private Declare Function CallWindowProc Lib "user32" Alias "CallWindowProcA" ( _
ByVal lpPrevWndFunc As Long, ByVal hWnd As Long, ByVal Msg As Long, _
ByVal wParam As Long, ByVal lParam As Long) As Long
Const GWL_WNDPROC = -4
Dim saveHWnd As Long ' The handle of the subclassed window.
Dim oldProcAddr As Long ' The address of the original window procedure
Sub StartSubclassing(ByVal hWnd As Long)
saveHWnd = hWnd
oldProcAddr = SetWindowLong(hWnd, GWL_WNDPROC, AddressOf WndProc)
End Sub
Sub StopSubclassing()
SetWindowLong saveHWnd, GWL_WNDPROC, oldProcAddr
End Sub
Function WndProc(ByVal hWnd As Long, ByVal uMsg As Long, _
ByVal wParam As Long, ByVal lParam As Long) As Long
WndProc = CallWindowProc(oldProcAddr, hWnd, uMsg, wParam, lParam)
Select Case uMsg
End Select
End Function
When the AddressOf operator is used in a call to a Declare method (SetWindowLong, in this case), VB Migration Partner defines a delegate class that matches the target procedure’s signature (WndProc, in this case) and defines an overload of the Declare method with a delegate parameter instead of an integer parameter:
Public Delegate Function SetWindowLong_CBK(ByVal hWnd As Integer, ByVal uMsg As Integer,_
ByVal wParam As Integer, ByVal lParam As Integer) As Integer
Friend Module SubclassingAPI
Private Declare Function SetWindowLong Lib "user32" Alias "SetWindowLongA" _
(ByVal hWnd As Integer, ByVal ndx As Integer, _
ByVal newValue As Integer) As Integer
Private Declare Function SetWindowLong Lib "user32" Alias "SetWindowLongA" _
(ByVal hWnd As Integer, ByVal ndx As Integer, _
ByVal newValue As SetWindowLong_CBK)
Public Sub StartSubclassing(ByVal hWnd As Integer)
saveHWnd = hWnd
oldProcAddr = SetWindowLong(hWnd, GWL_WNDPROC, AddressOf WndProc)
End Sub
End Module
This converted code would work correctly, except for a subtle detail: the VB code creates a SetWindowLong_CBK delegate object and passes it to the SetWindowLong method, without storing it in a variable. The problem is: when the garbage collector fires, the delegate object is reclaimed and the application throws a CallbackOnCollectedDelegate exception as soon as it attempts to invoke the delegate.
This problem can appear with other API-related technique that uses callback methods, for example with the applications that set up keyboard handlers by means of the SetWindowsHook or SetWindowsHookEx API methods.
Fortunately, the solution is simple: instead of creating the delegate object and pass it to the API method on-the-fly, the VB.NET code should store it in a class-level field, so that the delegate is kept alive and protected from garbage collections. You can achieve this behavior in a number of ways, the simplest one being a set of InsertStatement pragma added to the original VB6 code:
Dim saveHWnd As Long ' The handle of the subclassed window.
Dim oldProcAddr As Long ' The address of the original window procedure
Sub StartSubclassing(ByVal hWnd As Long)
saveHWnd = hWnd
oldProcAddr = SetWindowLong(hWnd, GWL_WNDPROC, AddressOf WndProc)
End Sub
If the Declare method is called by many places of the application, you might want to implement a different solution, based on wrappers for the Declare method that takes the callback value. The purpose of such wrappers is storing the delegate value in a collection, which indirectly protects it from garbage collections.
' this is the original Declare, renamed and made Private
Private Declare Function SetWindowLong_Private Lib "user32" Alias "SetWindowLongA"_
(ByVal hWnd As Integer, ByVal ndx As Integer, _
ByVal newValue As SetWindowLong_CBK) As Integer
Public Declare Function SetWindowLong(ByVal hWnd As Integer, _
ByVal ndx As Integer, ByVal newValue As SetWindowLong_CBK) As Integer
KeepAliveObjects6.Add(newValue)
Return SetWindowLong_Private(hWnd, ndx, newValue)
End Function
VB Migration Partner is able to automatically generate such wrapper methods for you. All you need is applying a WrapDeclareWithCallbacks pragma to the class or project level.