Does Splice support Unicode?

Question

This is my code :

 testSplice :: C.Splice Handler
 testSplice = return $ C.yieldRuntimeText $ do
    return "中文"

And I bind it to a tag:

  splices :: Splices (C.Splice Handler)
  splices =
      "testSplice" ## testSplice

And used it on layout.tpl:

   <meta charset="UTF-8"> 
   <testSplice/>

And the output on the browser is

    -�

What have I done wrong?

Sorry for the delay, I have been busy for a while, and now I come back and I think I may not make the question specific enough @mightybyte Here is the code where the problem occurs, I hope it would make the question more specifier:

test.hs:

{-# LANGUAGE OverloadedStrings #-}
import Snap
import Heist
import qualified Heist.Compiled as C
import Data.Monoid
import Control.Monad.Trans.Either
import Data.Maybe

main :: IO ()
main = quickHttpServe site

site :: Snap ()
site = 
    route [("/", testSnap)]

testSnap :: Snap ()
testSnap = do
    hs <- liftIO $ load "template" splices
    let runtime = fromJust $ C.renderTemplate hs "test"
    builder <-liftIO  (fst runtime)
    writeBuilder builder
  where
    splices :: Splices (C.Splice IO)
    splices = 
        "testSplice" ## testSplice

load :: MonadIO n
    => FilePath
    -> Splices (C.Splice n)
    -> IO (HeistState n)
load baseDir splices = do
    tmap <- runEitherT $ do
        let t = loadTemplates baseDir
            hc = HeistConfig
                    defaultInterpretedSplices
                    defaultLoadTimeSplices
                    splices
                    mempty
                    [t]
        initHeist hc
    either (error . concat) return tmap

testSplice :: C.Splice IO
testSplice = return $ C.yieldRuntimeText $ do return "中文"

template/test.tpl

<html>
  <head>
    <meta charset="UTF-8">
  </head>
 <body>
     <testSplice/>
 </body>
 </html>

Now I tried heist-0.13.0.2 and it works fine now, great work Daniel!

You should probably provide a link to that library's source. I haven't heard of it and can't find it via hoogle/google. — jtobin, Oct 12 '13 at 06:02
Heist does support unicode. See [my answer here](http://stackoverflow.com/questions/18931786/how-to-show-utf8-text-with-snap-and-heist/18965748#18965748) for more details. — mightybyte, Oct 12 '13 at 23:46
@mightybyte But doesn´t yieldRuntimeText use the fromText that comes from Blaze.ByteString.Builder.Char8? Sorry if I´m mistaken, I´ll retract my answer in that case. — danidiaz, Oct 13 '13 at 00:12
@DanielDíazCarrete Ahh, yes. You are exactly right. I didn't read your answer close enough. The post I linked to is correct, but it is rendering strings directly using snap-core instead of using Heist. — mightybyte, Oct 13 '13 at 16:54
@mightybyte Please see the updated source code, sorry for the delay. — Lynton, Oct 20 '13 at 06:38

danidiaz · Accepted Answer · 2013-10-13T19:26:44.843

UPDATE: The problem described in this answer has been corrected in heist version 0.13.0.2.

The source code for yieldRuntimeText is:

yieldRuntimeText :: Monad n => RuntimeSplice n Text -> DList (Chunk n)
yieldRuntimeText = yieldRuntime .  liftM fromText

From what module comes that fromText function? In the import section we find:

import           Blaze.ByteString.Builder
import           Blaze.ByteString.Builder.Char8

The documentation for the latter package says:

Note: This package is intended for low-level use like implementing protocols. If you need to serialize Unicode characters use one of the UTF encodings (e.g. 'Blaze.ByteString.Builder.Char.UTF-8').

and also:

fromText :: Text -> BuilderSource

O(n). Serialize the lower 8-bits of all characters in the strict text.

Mmmmm so maybe the problem is that the encoding from Text to Builder is not being done in UTF-8? Try defining your own version of yieldRuntimeText, with exactly the same code but using the following imports instead:

import           Blaze.ByteString.Builder
import           Blaze.ByteString.Builder.Char.Utf8

yieldRuntimeTextUtf8 :: Monad n => RuntimeSplice n Text -> DList (Chunk n)
yieldRuntimeTextUtf8 = yieldRuntime .  liftM fromText

Excellent detective work. I just uploaded a fixed version to hackage as heist-0.13.0.2. If you upgrade to that version, it should work without defining your own `yieldRuntimeTextUtf8`. — mightybyte, Oct 13 '13 at 16:56

Does Splice support Unicode?

1 Answers1