The problem is that when you declare a property of type Form, it allows you to select a form instance, not a type/class that inherits from Form. What you need to do instead is declare a property of type Type and prepare the editor which will allow you to select the desired type (limiting the options to the types that inherit from Form).
Disclaimer: The idea of creating a custom editor was inspired by this
  answer and the code was adapted for this particular situation.
So, here we go. The custom label class would look something like this:
Public Class BMLabel
    Inherits Label
    ' Don't forget to change the namespace.
    '        ↓↓↓↓↓↓↓↓↓↓↓
    <Editor("WindowsApp1.TypeSelector, System.Design", GetType(UITypeEditor)), Localizable(True)>
    Public Property FormType As Type
    Private Sub BMLabel_Click(sender As Object, e As EventArgs) Handles Me.Click
        Using frm As Form = DirectCast(Activator.CreateInstance(FormType), Form)
            frm.ShowDialog(Me)
        End Using
    End Sub
End Class
Now, we need to create the TypeSelector class:
Public Class TypeSelector
    Inherits UITypeEditor
    Public Overrides Function GetEditStyle(context As ITypeDescriptorContext) As UITypeEditorEditStyle
        If context Is Nothing OrElse context.Instance Is Nothing Then
            Return MyBase.GetEditStyle(context)
        End If
        Return UITypeEditorEditStyle.Modal
    End Function
    Public Overrides Function EditValue(context As ITypeDescriptorContext, provider As IServiceProvider, value As Object) As Object
        Dim editorService As IWindowsFormsEditorService
        If context Is Nothing OrElse context.Instance Is Nothing OrElse provider Is Nothing Then
            Return value
        End If
        editorService = DirectCast(provider.GetService(GetType(IWindowsFormsEditorService)),
                                   IWindowsFormsEditorService)
        Dim dlg As New FormTypeSelector()
        dlg.Value = DirectCast(value, Type)
        dlg.StartPosition = System.Windows.Forms.FormStartPosition.CenterScreen
        If editorService.ShowDialog(dlg) = System.Windows.Forms.DialogResult.OK Then
            Return dlg.Value
        End If
        Return value
    End Function
End Class
Then, we create a form called FormTypeSelector with a ComboBox or a ListBox to list the available options:

Public Class FormTypeSelector
    Friend Property Value As Type
    Private Sub FormTypeSelector_Load(sender As Object, e As EventArgs) Handles MyBase.Load
        Dim availableFormTypes =
            System.Reflection.Assembly.GetExecutingAssembly().
                    GetTypes().
                    Where(Function(t) t.BaseType = GetType(Form) AndAlso t <> Me.GetType()).ToList()
        cboFormTypes.DisplayMember = "Name"
        cboFormTypes.DataSource = availableFormTypes
        cboFormTypes.SelectedItem = Value
    End Sub
    Private Sub BtnOK_Click(sender As Object, e As EventArgs) Handles btnOK.Click
        Value = DirectCast(cboFormTypes.SelectedItem, Type)
        DialogResult = DialogResult.OK
        Close()
    End Sub
    Private Sub btnCancel_Click(sender As Object, e As EventArgs) Handles btnCancel.Click
        DialogResult = DialogResult.Cancel
        Close()
    End Sub
End Class
And that's it; it should be ready to go:

Note: You'll probably need to add an option in FormTypeSelector to allow clearing the selected value of the FormType property which should be easy enough to do.