When you specify the OFN_ALLOWMULTISELECT flag and the user selects two or more files, the GetOpenFilename API method returns a string that contains these file names, each terminated by a null char, with an extra null char after the last file name. Unfortunately, the PInvoke portion of the .NET Framework incorrectly interprets the embedded null chars as the termination of the entire string and doesn’t return any character that follows the first file name.

To explain this problem in more details, let’s start with the typical code that is produced by migrating the original VB6 application:

       Declare Function GetOpenFileName Lib "comdlg32.dll" Alias "GetOpenFileNameA" _
           (ByRef pOpenfilename As OPENFILENAME) As Integer
        Const MAXFILE As Integer = 10000

        Structure OPENFILENAME
            Public lStructSize As Integer
            Public hwndOwner As Integer
            Public hInstance As Integer
            Public lpstrFilter As String
            Public lpstrCustomFilter As String
            Public nMaxCustFilter As Integer
            Public nFilterIndex As Integer
            Public lpstrFile As String
            Public nMaxFile As Integer
            Public lpstrFileTitle As String
            Public nMaxFileTitle As Integer
            Public lpstrInitialDir As String
            Public lpstrTitle As String
            Public flags As Integer
            Public nFileOffset As Short
            Public nFileExtension As Short
            Public lpstrDefExt As String
            Public lCustData As Integer
            Public lpfnHook As Integer
            Public lpTemplateName As String
        End Structure

        ' a wrapper function

        Public Function OpenDlg(ByVal hWnd As Integer, ByVal Filter As String , _
            ByRef Title As String, ByVal InitialDir As String, _
            Optional ByVal MultiSelect As Boolean = False) As String

        Dim OFName As OPENFILENAME
	
        ' Replace all '|' chars in the filter with spaces
        Filter = Replace(Filter, "|", Chr(0))
        ' set initial directory
        If InitialDir = "" Then InitialDir = InDir
		
        OFName.lStructSize = Len6(OFName)
        OFName.hwndOwner = hWnd
        OFName.hInstance = App6.hInstance
        OFName.lpstrFilter = Filter
        OFName.lpstrFile = Space(MAXFILE)
        OFName.nMaxFile = MAXFILE
        OFName.lpstrFileTitle = Space(MAXFILE)
        OFName.nMaxFileTitle = MAXFILE
        OFName.lpstrInitialDir = InitialDir
        OFName.lpstrTitle = Title 
        OFName.flags = OFN_HIDEREADONLY
        ' Allow multi file selection, if necessary
        If MultiSelect Then 
            OFName.flags = OFName.flags Or OFN_EXPLORER Or OFN_ALLOWMULTISELECT
        End If
		
        ' Show the open dialog, return the selected file or a null string
        If GetOpenFileName(OFName) = 1 Then
            Return Trim(OFName.lpstrFile)
        Else
            Return ""
        End If
End Function

The problematic element is lpstrFile: if the user selects two or more files, on return from the GetOpenFileName API method this element should contain the following values:

        directoryname  NULL  file1  NULL  file2  NULL … fileN  NULL  NULL

However, the first NULL char cheats PInvoke into believing that the string ends there, thus the returned string contains only the directory name. You can spot this problem quite easily, because the application typically throws an exception as soon as it attempts to use the directory name as if it were a file name. To work around this issue you must tweak the VB.NET code.

First, add the following Imports statements at the top of the source file:

        Imports System.Runtime.InteropServices
        Imports System.Text

Next, ensure that the Unicode version of the Windows API method is invoked. You do this by using the “W” version of the API method in the Alias portion of the Declare statement:

        Declare Function GetOpenFileName Lib "comdlg32.dll" Alias "GetOpenFileNameW" _
            (ByRef pOpenfilename As OPENFILENAME) As Integer

Changing the Alias portion isn’t enough, though: you must also ensure that all string elements in the structure are passed as Unicode strings rather than ANSI strings. You do this with a correct StructLayout attribute:

        <StructLayout(LayoutKind.Sequential, CharSet:=CharSet.Unicode)> _
        Structure OPENFILENAME
            ...
        End Structure

You are now ready to face the actual problem, which as you know is caused by the lpstrFile element. When this element is passed to the GetOpenFileName method, this element must be a pointer to a buffer whose length is specified in the nMaxFile element. (In the original code, the buffer is correctly created and passed to the API method but the return value is incorrectly truncated on returning from the unmanaged method.) We can work around the problem by manually allocating the buffer and extracting the characters when the method returns.

To minimize the impact on surrounding code, we make the element private and change its name in something else, and then we add a public string property named lpstrFile. The getter and setter blocks of such property hide the allocation and deallocation details:

        <StructLayout(LayoutKind.Sequential, CharSet:=CharSet.Unicode)>
        Friend Structure OPENFILENAME
            Public lStructSize As Integer
            Public hwndOwner As Integer
            Public hInstance As Integer
            Public lpstrFilter As String
            Public lpstrCustomFilter As String
            Public nMaxCustFilter As Integer
            Public nFilterIndex As Integer

            Private m_lpstrFile As IntPtr

            Public nMaxFile As Integer
            Public lpstrFileTitle As String
            Public nMaxFileTitle As Integer
            Public lpstrInitialDir As String
            Public lpstrTitle As String
            Public flags As Integer
            Public nFileOffset As Short
            Public nFileExtension As Short
            Public lpstrDefExt As String
            Public lCustData As Integer
            Public lpfnHook As Integer
            Public lpTemplateName As String

            Property lpstrFile() As String
                Get
                    ' copy all chars from the buffer into an array
                    Dim chars(MAXFILE - 1) As Char
                    Marshal.Copy(m_lpstrFile, chars, 0, MAXFILE)
                    ' release the unmanaged buffer
                    Marshal.FreeHGlobal(m_lpstrFile)
                    ' convert the char to a string and return
                    Return New String(chars)
                End Get
                Set(ByVal value As String)
                    m_lpstrFile = Marshal.StringToHGlobalUni(value)
                End Set
            End Property
        End Structure

With these changes the call works correctly. However, you must not overlook an important detail: you must read the lpstrFile element once (and only once) after calling the GetOpenFileName method. If you read it more than once, you get an exception at the second attempt; if you fail to read it at all after the call, the unmanaged buffer won’t be reclaimed and your application will leak memory. Here’s how you must modify the calling method to account for this requirement:

        ' store the return value, then deallocate the buffer in all cases
        Dim ret As Integer = GetOpenFileName(OFName)
        Dim filename As String = Trim(OFName.lpstrFile)

        ' you can now process the result as you see fit
        If ret = 1 Then
            Return filename
        Else
            Return ""
        End If