Your problem is due to the fact that all IE versions prior to 10 have no support for the HTML5 File API. So, considering that this is your HTMLInputElement, the following won't work:
this.files[0].size;
The reason is that HTMLInputElement.FileList does not exist since the lack of File API support, but you could get the file's name with HTMLInputElement.value. But that's not what you want. You want to get the file's size. Once again, because of the lack of the File API support, IE < 10 doesn't supply the file size information. So, the only way to check the file size for these lesser versions, is to use ActiveX, even if nobody likes doing so. 
First solution (client side - jQuery)
You proposed:
I could use browser detection to say "hey, if IE do activeX else do the jQuery"
If you want to do so, you could have something like that:
$(function(){
    $('#File1').bind('change', function() {
        var maxFileSize = 1024000; // 1MB -> 1000 * 1024
        var fileSize;
        // If current browser is IE < 10, use ActiveX
        if (isIE() && isIE() < 10) {
            var filePath = this.value;
            if (filePath != '') {
                var AxFSObj = new ActiveXObject("Scripting.FileSystemObject");
                var AxFSObjFile = AxFSObj.getFile(filePath);
                fileSize = AxFSObjFile.size;
            }
        } else {
            // IE >= 10 or not IE
            if (this.value != '') {
                fileSize = this.files[0].size;
            }
        }
        if (fileSize < maxFileSize) {
            // Enable submit button and remove any error message
            $('#button_fileUpload').prop('disabled', false);
            $('#lbl_uploadMessage').text('');
        } else {
            // Disable submit button and show error message
            $('#button_fileUpload').prop('disabled', true);
            $('#lbl_uploadMessage').text('File too big !');
        }
    });
});
// Check if the browser is Internet Explorer
function isIE() {
    var myNav = navigator.userAgent.toLowerCase();
    return (myNav.indexOf('msie') != -1) ? parseInt(myNav.split('msie')[1]) : false;
} 
Warning!
Before blindly copying this code, note that the ActiveX solution is really a poor one. To make it work, the user must change its Internet Options. Also, this solution is no good for public sites, only for intranet apps. 
But I am wondering if there is a more reliable method than ActiveX?
No, there is not for IE < 10
Second solution (jQuery plugin)
You could the jQuery File Upload. You can easily get the size with it.
Third solution (server side - VB.NET)
Considering you have these asp controls:
<asp:FileUpload ID="fileUpload" runat="server" />
<asp:Button ID="button_fileUpload" runat="server" Text="Upload File" />
<asp:Label ID="lbl_uploadMessage" runat="server" Text="" ForeColor="Red" />
You can check the chose file size once there is an interaction with the server side. For instance, here I am checking the file's size once the user clicks on the Upload Button.
Protected Sub btnFileUpload_click(ByVal sender As Object, ByVal e As System.EventArgs) Handles button_fileUpload.Click
    ' A temporary folder
    Dim savePath As String = "c:\temp\uploads\"
    If (fileUpload.HasFile) Then
        Dim fileSize As Integer = fileUpload.PostedFile.ContentLength
        ' 1MB -> 1000 * 1024
        If (fileSize < 1024000) Then
            savePath += Server.HtmlEncode(fileUpload.FileName)
            fileUpload.SaveAs(savePath)
            lbl_uploadMessage.Text = "Your file was uploaded successfully."
        Else
            lbl_uploadMessage.Text = "Your file was not uploaded because " +
                                     "it exceeds the 1 MB size limit."
        End If
    Else
        lbl_uploadMessage.Text = "You did not specify a file to upload."
    End If
End Sub
Edit
As you said, this last solution doesn't work with files which are too big. To allow this solution to work, you must increase the max upload file size in your web.config file:
<configuration>
    <system.web>
        <httpRuntime maxRequestLength="52428800" /> <!--50MB-->
    </system.web>
</configuration>