1

NB! to those who associate this post with:https://stackoverflow.com/questions/31524404/classic-asp-and-utf-8 it does NOT solve the issue, and everything in it has been tested.

I have an ASP Classic form that sends through CDO using an HTML template to send an e-mail.. passing the special language characters (In this case danish characters æ,ø,å) from the form to the ASP page holding CDO code is working OK and displays the characters correctly when making a Response.Write on that page. But, when the e-mail sends the passed characters are "mess-up". However, if I write the characters directly into the HTML template they display correct in the e-mail. All three documents have their charset set to UTF-8 and the ASP pages have CODEPAGE 65001 set. Can anyone see what is going on to cause this behaviour?

This is what happens in the e-mail:

This is what happens in the e-mail

E-mail template (only definition) :

<!DOCTYPE html>
<html>

<head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <title>Email</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />

    <style type="text/css">
        a[x-apple-data-detectors] {
            color: inherit !important;
        }
    </style>

</head>

<body style="margin: 0; padding: 0;">

 

    
</body>

</html>

sendmail.asp :

<%@Codepage = 65001%>
<%
Option explicit
Response.CodePage = 65001
Response.CharSet = "UTF-8"


    Dim SettingsConn
    Set SettingsConn = Server.CreateObject("ADODB.Connection")
    SettingsConn.ConnectionString="Provider=SQLOLEDB; DATA SOURCE=<SERVER>;UID=<USERNAME>;PWD=<PASSWORD>;DATABASE=<DB>"
    SettingsConn.Open

    Dim SettingsSQL, objSettings

    SettingsSQL = "SELECT SMTPServer,SMTPPort,MailFromName,MailFromEmail,MailCC,MailBCC,EFPVersion FROM EFP_Settings WHERE ID = 1;"

    Set objSettings = SettingsConn.Execute(SettingsSQL)


    dim pde : set pde = createobject("scripting.dictionary")

    function getTextFromFile(path)
        dim fso, f, txt
        set fso = createobject("Scripting.FileSystemObject")
        if not fso.fileexists(path) then
            getTextFromFile = ""
            exit function
        end if
        set f = fso.opentextfile(path,1)
        if f.atendofstream then txt = "" else txt = f.readall
        f.close
        set f = nothing
        set fso = nothing
        getTextFromFile = txt
    end function


    dim redir, mailto, mailfrom, subject, item, body, cc, bcc, message, html, template, usetemplate, testmode
    redir = request.form("redirect")
    mailto = request.form("mailto")
    if pde.exists(mailto) then mailto = pde(mailto)
    cc = objSettings("MailCC")
    bcc = objSettings("MailBCC")
    subject = request.form("subject")
    message = request.form("message")
    template = request.form("template")

    if len(template) > 0 then template = getTextFromFile(server.mappath(template))
    if len(template) > 0 then usetemplate = true else usetemplate = false
    dim msg : set msg = server.createobject("CDO.Message")
    dim smtpServer, yourEmail, yourPassword
    msg.Configuration.Fields.Item ("http://schemas.microsoft.com/cdo/configuration/sendusing") = 2
    msg.Configuration.Fields.Item ("http://schemas.microsoft.com/cdo/configuration/smtpserver") = "" & objSettings("SMTPServer") & ""
    msg.Configuration.Fields.Item ("http://schemas.microsoft.com/cdo/configuration/smtpserverport") = objSettings("SMTPPort")
    msg.Configuration.Fields.Update
    msg.subject = subject
    msg.to = mailto
    msg.from = """" & objSettings("MailFromName") & """ <" & objSettings("MailFromEmail") & ">"
    msg.Sender = """" & objSettings("MailFromName") & """ <" & objSettings("MailFromEmail") & ">"

    if len(cc) > 0 then msg.cc = cc
    if len(bcc) > 0 then msg.bcc = bcc

    if not usetemplate then
        body = body & message & vbcrlf & vbcrlf
    else
        body = template
    end if
    for each item in request.form
        select case item
            case "redirect", "mailto", "cc", "bcc", "subject", "message", "template", "html", "testmode"
            case else
                if not usetemplate then
                    if item <> "mailfrom" then body = body & item & ": " & request.form(item) & vbcrlf & vbcrlf
                else
                    body = replace(body, "[$" & item & "$]", replace(request.form(item),vbcrlf,"<br>"))
                end if
        end select
    next

    if usetemplate then
        dim rx : set rx = new regexp
        rx.pattern = "\[\$.*\$\]"
        rx.global = true
        body = rx.replace(body, "")
    end if

        msg.htmlbody = body

        msg.send

        response.redirect redir

        set msg = nothing
    %>

UPDATE:

Based on the comments below;

I have tried to convert the FSO to ADODB so it looks like the below, but is an error:

Code :

function getTextFromFile(path)
    Dim adoStream, txt
    Set adoStream = CreateObject("Adodb.Stream")
    if not adoStream.FileSystemObject(path) then
        getTextFromFile = ""
        exit function
    end if
    adoStream.Open
    adoStream.Charset = "UTF-8"
    txt = adoStream.ReadText(-1)
    adoStream.LoadFromFile txt
    adoStream.Close
    Set adoStream = Nothing
    getTextFromFile = txt
end function

Latest Update (28-03-2021) - 11:36

Latest sendmail.asp :

<%@Codepage = 65001%>
  <%
    Option explicit
    Response.CodePage = 65001
    Response.CharSet = "UTF-8"

    Dim SettingsConn
    Set SettingsConn = Server.CreateObject("ADODB.Connection")
    SettingsConn.ConnectionString="Provider=SQLOLEDB; DATA SOURCE=<SERVER>;UID=<USERNAME>;PWD=<PASSWORD>;DATABASE=<DATABASE>"
    SettingsConn.Open

    Dim SettingsSQL, objSettings

    SettingsSQL = "SELECT SMTPServer,SMTPPort,MailFromName,MailFromEmail,MailCC,MailBCC,EFPVersion FROM EFP_Settings WHERE ID = 1;"

    Set objSettings = SettingsConn.Execute(SettingsSQL)

    dim redir, mailto, mailfrom, subject, item, body, cc, bcc, html, template, usetemplate, testmode
    redir = request.form("redirect")
    mailto = request.form("mailto")
    cc = objSettings("MailCC")
    bcc = objSettings("MailBCC")
    subject = request.form("subject")
    template = request.form("template")

    Response.Write request.form("template")
    ' Output of request.form("template") is templates/emailtemplate_report_issue.htm
    
    Dim adoStream, getTextFromFile
    Set adoStream = CreateObject("Adodb.Stream")
    adoStream.Type = 2
    adoStream.Open
    adoStream.Charset = "UTF-8"
    adoStream.LoadFromFile server.mappath("" & template & "")
    template = adoStream.ReadText(-1)
    adoStream.Close
    Set adoStream = Nothing
    getTextFromFile = template
    

    dim msg : set msg = server.createobject("CDO.Message")
    dim smtpServer, yourEmail, yourPassword
    msg.Configuration.Fields.Item ("http://schemas.microsoft.com/cdo/configuration/sendusing") = 2
    msg.Configuration.Fields.Item ("http://schemas.microsoft.com/cdo/configuration/smtpserver") = "" & objSettings("SMTPServer") & ""
    msg.Configuration.Fields.Item ("http://schemas.microsoft.com/cdo/configuration/smtpserverport") = objSettings("SMTPPort")
    msg.Configuration.Fields.Update
    msg.subject = subject
    msg.to = mailto
    msg.from = """" & objSettings("MailFromName") & """ <" & objSettings("MailFromEmail") & ">"
    msg.Sender = """" & objSettings("MailFromName") & """ <" & objSettings("MailFromEmail") & ">"

    if len(cc) > 0 then msg.cc = cc
    if len(bcc) > 0 then msg.bcc = bcc

    body = template
    for each item in request.form
            body = replace(body, "[$" & item & "$]", replace(request.form(item),vbcrlf,"<br>"))
    next

    dim rx : set rx = new regexp
    rx.pattern = "\[\$.*\$\]"
    rx.global = true
    body = rx.replace(body, "")

    msg.htmlbody = body

    msg.send

    response.redirect redir

    set msg = nothing

    %>

Output on e-mail where characters are not displayed as expected (Both variables passed from form and hardcoded)

enter image description here

user692942
  • 16,398
  • 7
  • 76
  • 175
  • Follow the duplicate, it will explain how you need to tell IIS how the process the ASP page using the `@CodePage` directive. Also, make sure your SQL Server string data is being stored as `NVARCHAR` opposed to `VARCHAR`. – user692942 Mar 26 '21 at 19:03
  • The `@CodePage` does not make any difference to the e-mail output, still not display the characters correct ... @user692942 : In the process of passing variables from form asp page to send asp mail to HTML page I do not invoke any connection to DB. –  Mar 26 '21 at 20:15
  • Is the ASP page saved as UTF-8?, these are all things to check whenever you encounter encoding mismatches. – user692942 Mar 26 '21 at 21:35
  • @user692942 yes both HTML Template and all ASP pages are saved using UTF-8 in Visual Studio Code. –  Mar 26 '21 at 21:48
  • Could it be something to do with code in sendmail.asp where it converts `request.form` items into `[$FormItems$]`? –  Mar 26 '21 at 21:52
  • Could you update the code in your question to show where you tried adding the `@CodePage` directive? See the duplicate for where it should be placed. Also can you show the page that sends the form? – user692942 Mar 27 '21 at 07:31
  • @user692942 as requested I have updated sendmail.asp code above with ´@CodePage´ directive. Regarding the page that sends the form, as I wrote the sendmail.asp file translates the characters as they should when making a ´response write request.form("FormItem")´ so the issue lies between sendmail.asp and template.htm –  Mar 27 '21 at 09:34
  • 1
    Thanks for clarifying that. Not sure the issue is anything to do with the encoding in Classic ASP *(as you mentioned you've gone through all the steps)* but as it’s the email that doesn’t work the problem is likely in CDO. Are you setting `msg.BodyPart.Charset = "utf-8"`? *(See duplicate link)*. – user692942 Mar 27 '21 at 10:49
  • Does this answer your question? [How to send email with Unicode characters using VBScript?](https://stackoverflow.com/a/7235192) – user692942 Mar 27 '21 at 11:08
  • @user692942 now we are closing in .. `msg.htmlbody = Server.HTMLEncode(body)` does not work, but `msg.BodyPart.Charset = "utf-8"` does for the values past from the form, they are now displayed correctly in the E-mail, however all the Danish charaters hardcoded into the Template HTML file is now messed up, they worked before. –  Mar 27 '21 at 11:08
  • Sounds like your template file isn't actually saved as UTF-8. – user692942 Mar 27 '21 at 11:09
  • @user692942 Visual Studio Code says it is saved as UTF-8, and following is set in the the HEAD area ` ` –  Mar 27 '21 at 11:12
  • It's this line `getTextFromFile(server.mappath(template))` as with the duplicate question `FileSystemObject` doesn't read UTF-8 by default and it's best to use the `ADODB.Stream` object for that as described in the duplicate answer. Couldn't see that properly until I formatted the code in the question. – user692942 Mar 27 '21 at 11:14
  • @user692942 that makes sense, I read that before your post with `msg.BodyPart.Charset = "utf-8"` and was trying to see how to change FSO to ADODB.Stream, but to be honest I am blank .. any tips on how I should change my FSO to use ADODB.Stream? –  Mar 27 '21 at 11:20
  • 1
    The code is pretty much laid out line for line in the [answer linked](https://stackoverflow.com/a/7235192/692942). All you need to do is replace all the lines (except `getTextFromFile = txt`) in your `getTextFromFile()` function with the `ADODB.Stream` example then set the `adoStream.loadFromFile path` to load your path you pass it and set `txt = adoStream.ReadText(-1)` as the `txt` variable is what you return from the function. Hope this helps. – user692942 Mar 27 '21 at 11:24
  • @user692942 I have just tried to convert it, it seems that it is on the right path but I get an error .. I have just made an update with the code and the error in the original post. –  Mar 27 '21 at 11:55
  • Yeah, the `ADODB.Stream` does not support `FileExists()` that is the `FileSystemObject` you have also removed the `getTextFromFile = txt` line which you still need at the end of the function to return the value of `txt`. – user692942 Mar 27 '21 at 12:17
  • @user692942 .. I have just updated the code, but still get an error .. code and error is updated in my post. –  Mar 27 '21 at 13:03
  • I'm sorry but it's clear from your edits you don't understand how to work with VBScript components like `ADODB .Stream` and the `FileSystemObject`. I assumed you had that base knowledge because you were very articulate when it came to discussing the encoding issue. There are plenty examples of using `ADODB.stream` to take a file and convert it to a UTF-8 string. – user692942 Mar 27 '21 at 16:01
  • Yes, sorry, I have never worked with ADODB.Stream before, thats why I am a bit off my comfort zone. –  Mar 27 '21 at 16:27
  • 1
    Best advice I can give is remove the `If` (that includes the between and the `End If`) that you are trying to use to check whether the file exists and try that, think you’ll find it will work. The `If` file exists check can be added at a later date, but that should get you over the line – user692942 Mar 27 '21 at 18:29
  • Thank you @user692942 that helped me a little further.. but I am stock on `Arguments are of the wrong type, are out of acceptable range, or are in conflict with one another` .. I will see if I can find out what is wrong. :-) –  Mar 27 '21 at 21:15
  • Edit the question and show what you've changed, we should be able to get you over the line. – user692942 Mar 27 '21 at 21:28
  • Thank you @user692942 .. I have just added latest code for sendmail under the headline "Latest sendmail.asp :" –  Mar 27 '21 at 21:37
  • You need to `adoStream.LoadFromFile txt` before you call `txt = adoStream.ReadText(-1)`. Somehow you've messed up the order, that should fix it. – user692942 Mar 27 '21 at 21:40
  • @user692942 .. Yes, I changed the order when I tried to find a solution .. I still get the same error even after correction and setting in that order .. `adoStream.Charset = "UTF-8"` >> `adoStream.LoadFromFile txt` >> `txt = adoStream.ReadText(-1)` >> `adoStream.Close` .. the error is in line 23 which is `adoStream.LoadFromFile txt` –  Mar 27 '21 at 21:48
  • @user692942 : I got it working now, had to add `adoStream.Type = 2` and `adoStream.LoadFromFile server.mappath("" & template & "")` but the email I get does not return expected characters neigther for hardcoded or variables passed, so now all special characters are messed up, and I am back to where I started before ADODB.Stream. See updated code and screendump under **Latest Update (28-03-2021) - 11:36**. –  Mar 28 '21 at 09:42
  • 1
    OK, I am officially tired of charset, but have found a workaround that does what it should .. The template HTML file I have saved in charsetv ISO-8859-1, sendmail.asp is in UTF-8 aswell as the ADODB Stream charset this display´s the characters from the form correct in the e-mail, but does not display the characters hardcoded in the template .. to workaround that I have replaced æ,ø and å with character codes `æ, ø and å` which now display both form sent characters and hardcoded characters correct. –  Mar 28 '21 at 11:11
  • Right, so the default `Type` is 2 or `adTypeText` so setting that does nothing. Changing the charset in the HTML template is just going to cause other issues and confuse the problem further. Not sure why you deconstructed the function in favour of inline but your just creating more issues like leaving the function return reference in without a function, what is `getTextFromFile = template` meant to do now? You've kind of messed it up beyond all recognition now. – user692942 Mar 28 '21 at 11:26
  • @Dijkgraaf spoken like someone who doesn’t understand character encoding. Using HTML character codes is absolutely a workaround and I wouldn’t recommend it. Correctly encoding content from start to finish is the only way to do something like this. – user692942 Mar 29 '21 at 05:55
  • 1
    @Stkol76 Decided this needed to be addressed so I've put together [a gist](https://gist.github.com/lankymart/efbdcd3e2e1adb92b57c6434b6bf176c) that shows it does work. Tested locally using local IIS and [Smtp4dev](https://github.com/rnwood/smtp4dev). – user692942 Mar 29 '21 at 08:27
  • @user692942 wow, thanks .. I will definently have a look at this later today and get back :-) –  Mar 29 '21 at 08:30
  • 1
    @Stkol76 One thing I have noticed if you look at [`result.eml` in the gist](https://gist.github.com/lankymart/efbdcd3e2e1adb92b57c6434b6bf176c#file-result-eml) is that because you don't set a text body, CDO creates a default one that isn't UTF-8. So you might want to also [encode a text body](https://stackoverflow.com/a/18308518/692942). – user692942 Mar 29 '21 at 08:51
  • 1
    @user692942 .. YES!!!!! .. adding msg.BodyPart.Charset = "utf-8" to CDO script did the trick, all characters both from form and hardcoded in e-mail now displays correct without using æ &oslash and å .. this makes it a lot easier to maintain, THANK YOU! :-) .. please post an answer so that I can credit you ;-) –  Mar 29 '21 at 14:32
  • @Stkol76 didn't realise you weren't doing that (was mentioned 2 days ago). Glad it's working for you now hope the gist was useful. Just not a fan of people assuming encoding is the issue and then resorting to HTML codes needed addressing, especially [when others](https://stackoverflow.com/questions/66821988/passing-language-specific-characters-from-asp-to-cdo-to-html-template-not-workin?noredirect=1#comment118158525_66821988) start suggesting HTML codes is the only "reliable way". I'll flag to re-open the question, if it gets re-opened I'll leave an answer. – user692942 Mar 29 '21 at 15:00
  • 1
    @user692942 I must have overseen BodyPart Chaset in your previously comment, my bad ;-( .. And yes, the Gist was a nice touch Thanks ;-) .. I will keep an eye out to see if it gets reopened and and for your answer :-) –  Mar 29 '21 at 16:01
  • 1
    @user692942 seems it is open now. –  Mar 29 '21 at 16:02
  • @Dijkgraaf Just so we are clear - Correctly encoding the content is the only reliable way. [Everything else](https://stackoverflow.com/questions/66821988/passing-language-specific-characters-from-asp-to-cdo-to-html-template-not-workin#comment118158525_66821988) is most certainly a workaround. – user692942 Jul 13 '21 at 16:05

1 Answers1

2

This is an encoding problem but not specifically with how your Classic ASP is setup. After some extended discussion in the comments, it became clear the issue lies in how the CDO.Message is constructed.

Although the data in the Classic ASP script is being processed as UTF-8 the message is never told it should be, which can be rectified with this line;

msg.BodyPart.Charset = "utf-8"

Below is a working example of your code (with some of the SQL configuration elements removed for testability). It simulates the sending of the email using Smtp4Dev which is a fake SMTP Email Server for development and testing (Gist also included).

HTML Form POST (Encoded as UTF-8)

<html>
  <head>
    <title>Test 33</title>
  </head>
  <body>
    <form action="test.asp" method="post">
      <input type="hidden" name="template" value="template.htm" />
      <textarea name="message" rows="10" cols="100"></textarea>
      <input type="submit" value="Submit" />
    </form>
  </body>
</html>

HTML Template (Encoded as UTF-8)

<!DOCTYPE html>
<html>

<head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <title>Email</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />

    <style type="text/css">
        a[x-apple-data-detectors] {
            color: inherit !important;
        }
    </style>

</head>

<body style="margin: 0; padding: 0;">
  <p>æ,ø,å</p>
  <p>[$message$]</p>
</body>

</html>

Classic ASP Script (Encoded as UTF-8)

<%@Language="VBScript" Codepage = 65001%>
<%
Option Explicit
Response.CodePage = 65001
Response.Charset = "UTF-8"

Dim pde : Set pde = Server.CreateObject("scripting.dictionary")

Function getTextFromFile(path)
    Dim adoStream, txt
    Set adoStream = Server.CreateObject("ADODB.Stream")
    Call adoStream.Open()
    adoStream.Charset = "UTF-8"
    Call adoStream.LoadFromFile(path)
    txt = adoStream.ReadText(-1)
    Call adoStream.Close()
    Set adoStream = Nothing
    getTextFromFile = txt
End Function

Dim redir, mailto, mailfrom, subject, item, body, cc, bcc, message, html, template, usetemplate, testmode
redir = Request.Form("redirect")
mailto = Request.Form("mailto")
If pde.exists(mailto) Then mailto = pde(mailto)

subject = Request.Form("subject")
message = Request.Form("message")
template = Request.Form("template")

If Len(template) > 0 Then template = getTextFromFile(Server.MapPath(template))
usetemplate = (Len(template) > 0)

Dim msg : Set msg = Server.CreateObject("CDO.Message")
Dim smtpServer, yourEmail, yourPassword

msg.Configuration.Fields("http://schemas.microsoft.com/cdo/configuration/sendusing") = 2
msg.Configuration.Fields("http://schemas.microsoft.com/cdo/configuration/smtpserver") = "localhost"
msg.Configuration.Fields("http://schemas.microsoft.com/cdo/configuration/smtpserverport") = 25 
Call msg.Configuration.Fields.Update()

msg.Subject = "Test Email"
msg.To = "someone@example.com"
msg.From = "theserver@example.com"

If Len(cc) > 0 Then msg.cc = cc
If Len(bcc) > 0 Then msg.bcc = bcc

If Not usetemplate Then
    body = body & message & vbCrLf & vbCrLf
Else
    body = template
End If

For Each item In Request.Form
    Select Case item
    Case "redirect", "mailto", "cc", "bcc", "subject", "message", "template", "html", "testmode"
    Case Else
        If Not usetemplate Then
            If item <> "mailfrom" Then body = body & item & ": " & Request.Form(item) & vbCrLf & vbCrLf
        Else
            body = replace(body, "[$" & item & "$]", Replace(Request.Form(item), vbCrLf, "<br>"))
        End If
    End Select
Next

If usetemplate Then
    Dim rx : Set rx = New RegExp
    rx.Pattern = "\[\$.*\$\]"
    rx.Global = True
    body = rx.Replace(body, "")
End If

msg.BodyPart.Charset = "utf-8"
msg.htmlbody = body

Call msg.Send()
%>

Resultant Email

Downloaded from Smtp4Dev as an EML file.

Thread-Topic: Test Email
thread-index: AdckzyUf6ZF/uRsTSqG8szy1Ii2tbw==
From: <theserver@example.com>
To: <someone@example.com>
Subject: Test Email
Date: Mon, 29 Mar 2021 20:10:13 +0100
Message-ID: <9DB5C085BE5C40D784838A04215C21B9@FIMDLT1337>
MIME-Version: 1.0
Content-Type: multipart/alternative;
    boundary="----=_NextPart_000_0000_01D724D7.86E45B00"
X-Mailer: Microsoft CDO for Windows 2000
Content-Class: urn:content-classes:message
Importance: normal
Priority: normal
X-MimeOLE: Produced By Microsoft MimeOLE

This is a multi-part message in MIME format.

------=_NextPart_000_0000_01D724D7.86E45B00
Content-Type: text/plain;
    charset="utf-8"
Content-Transfer-Encoding: base64

w6Ysw7gsw6UNCg0K

------=_NextPart_000_0000_01D724D7.86E45B00
Content-Type: text/html;
    charset="utf-8"
Content-Transfer-Encoding: 8bit

<!DOCTYPE html>
<html>

<head>
    <meta http-equiv="Content-Type" content="text/html; charset=UTF-8" />
    <title>Email</title>
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />

    <style type="text/css">
        a[x-apple-data-detectors] {
            color: inherit !important;
        }
    </style>

</head>

<body style="margin: 0; padding: 0;">
  <p>æ,ø,å</p>
  <p></p>
</body>

</html>
------=_NextPart_000_0000_01D724D7.86E45B00--

Useful Links

user692942
  • 16,398
  • 7
  • 76
  • 175