C# Rijndael decryption returns extra question mark character

Question

I am organizing some very basic symmetric encryption/decryption codes in a class for future use. I am able to encrypt and decrypt successfully... only with a small problem.

Here is my code that reads in from a stream and en/decrypt to another stream:

public void Encrypt(Stream input, Stream output) {
    byte[] key = Encoding.UTF8.GetBytes(_pw);
    byte[] iv = Encoding.UTF8.GetBytes(GenerateInitVector());
    RijndaelManaged rm = new RijndaelManaged();
    CryptoStream cs = new CryptoStream(
        output,
        rm.CreateEncryptor(key, iv),
        CryptoStreamMode.Write);
    int data;
    while ((data = input.ReadByte()) != -1)
        cs.WriteByte((byte) data);
    cs.Close();
}

public void Decrypt(Stream input, Stream output) {
    byte[] key = Encoding.UTF8.GetBytes(_pw);
    byte[] iv = Encoding.UTF8.GetBytes(GenerateInitVector());
    RijndaelManaged rm = new RijndaelManaged();
    CryptoStream cs = new CryptoStream(
        input,
        rm.CreateDecryptor(key, iv),
        CryptoStreamMode.Read);
    int data;
    while ((data = cs.ReadByte()) != -1)
        output.WriteByte((byte) data);
    cs.Close();
}

For your information, the method GenerateInitVector() always returns the same value.

This is my test file.

hello
world
this
is
a
word
list

When I try to en/decrypt from FileStream to FileStream, everything works fine, using these methods:

public void Encrypt(string inputPath, string outputPath) {
    FileStream input = new FileStream(inputPath, FileMode.Open);
    FileStream output = new FileStream(outputPath, FileMode.Create);
    Encrypt(input, output);
    output.Close();
    input.Close();
}

public void Decrypt(string inputPath, string outputPath) {
    FileStream input = new FileStream(inputPath, FileMode.Open);
    FileStream output = new FileStream(outputPath, FileMode.Create);
    Decrypt(input, output);
    output.Close();
    input.Close();
}

When I try to decrypt the file into a MemoryStream and then load the bytes-converted-to-chars array into a StringBuilder and finally print it out to console as a string, I get in the console:

?hello
world
this
is
a
word
list

There is an extra question mark in front of all my text. Here is my decryption method:

public StringBuilder Decrypt(string inputPath) {
    FileStream input = new FileStream(inputPath, FileMode.Open);
    byte[] buffer = new byte[4096];
    MemoryStream output = new MemoryStream(buffer);
    Decrypt(input, output);
    StringBuilder sb = new StringBuilder(4096);
    sb.Append(Encoding.UTF8.GetChars(buffer, 0, (int) output.Position));
    input.Close();
    output.Close();
    return sb;
}

I have read something similar here regarding BOM and C# quietly writing unknown stuff as the '?' character:

Result of RSA encryption/decryption has 3 question marks

Therefore I have made doubly sure that I am using UTF-8 encoding. Still, I am seeing this extra question mark. For the sake of completeness, I have also written this method to encrypt contents inside a StringBuilder to a file:

public void Encrypt(string outputPath, StringBuilder builder) {
    char[] buffer = new char[builder.Length];
    builder.CopyTo(0, buffer, 0, builder.Length);
    byte[] buf = Encoding.UTF8.GetBytes(buffer);
    MemoryStream input = new MemoryStream(buf);
    FileStream output = new FileStream(outputPath, FileMode.Create);
    Encrypt(input, output);
    output.Close();
    input.Close();
}

Then I cross-check by doing:

var builder = new StringBuilder();
builder.Append("Hello world.");
Encrypt("test.txt.enc", builder);
Decrypt("test.txt.enc", "test.txt");
builder = Decrypt("test.txt.enc");
Console.WriteLine(builder.ToString());

For the file test.txt, things are fine. Strangely though, for the text printed on console, I get NO extra question mark in front:

Hello world.

What is wrong with the whole process?

Answer 1

The Questionmark is the BOM (Byte Order Mark) of UTF8 is 0xef 0xbb 0xbf This bytes are written at the beginning of your files encoded in UTF8.

Because FileStream reads the files as bytes and doesnt intepret it as UTF8 Textfile the BOM is included in your encryption so if you decrypt it and save it to file it looks all fine in a TextEditor but if you dump it to console the BOM is also printed cause the Console doesnt know it is some kind of control sequence/marker

EDIT: And here is the solution to get the string without the BOM.

    public static string Decrypt(string inputPath)
    {
        FileStream input = new FileStream(inputPath, FileMode.Open);
        MemoryStream output = new MemoryStream();
        Decrypt(input, output);
        StreamReader reader = new StreamReader(output, new UTF8Encoding()); //Read with encoding
        output.Seek(0, SeekOrigin.Begin); //Set stream Position to beginning
        string result = reader.ReadToEnd(); //read to string
        reader.Close();
        input.Close();
        output.Close();
        return result;
    }

Answer 2

Some issues:

• Key and IVs are fixed length binary sequence and can contain arbitrary bytes, so UTF-8 can't be right.

  Copying the key from a variable called pw, presumably short for 'password', indicates a related confusion. A password isn't a key. You should use a key-derivation-function, preferably one that's specialized on password hashing, like PBKDF2 or scrypt.

• Using a fixed IV misses the whole point of an IV. You need to use a new random value for each message. It's not secret, so simply put it in front of the ciphertext.

• Without a MAC you'll be open to active attacks, including padding oracles.
• Use Stream.CopyTo to copy data between two streams.