Encode a FileStream to base64 with c#

Question

I know how to encode / decode a simple string to / from base64.

But how would I do that if the data is already been written to a FileStream object. Let's say I have only access to the FileStream object not to the previously stored original data in it. How would I encode a FileStream to base64 before I flush the FileStream to a file.

Ofc I could just open my file and encode / decode it after I have written the FileStream to the file, but I would like to do this all in one single step without doing two file operations one after another. The file could be larger and it would also take double time to load, encode and save it again after it was just saved a short time before.

Maybe someone of you knows a better solution? Can I convert the FileStream to a string, encode the string and then convert the string back to a FileStream for example or what would I do and how would such a code look like?

Answer 1

An easy one as an extension method

public static class Extensions
{
    public static Stream ConvertToBase64(this Stream stream)
    {
        byte[] bytes;
        using (var memoryStream = new MemoryStream())
        {
            stream.CopyTo(memoryStream);
            bytes = memoryStream.ToArray();
        }

        string base64 = Convert.ToBase64String(bytes);
        return new MemoryStream(Encoding.UTF8.GetBytes(base64));
    }
}

Answer 2

When dealing with large streams, like a file sized over 4GB - you don't want to load the file into memory (as a Byte[]) because not only is it very slow, but also may cause a crash as even in 64-bit processes a Byte[] cannot exceed 2GB (or 4GB with gcAllowVeryLargeObjects).

Fortunately there's a neat helper in .NET called ToBase64Transform which processes a stream in chunks. For some reason Microsoft put it in System.Security.Cryptography and it implements ICryptoTransform (for use with CryptoStream), but disregard that ("a rose by any other name...") just because you aren't performing any cryptographic tasks.

You use it with CryptoStream like so:

using System.Security.Cryptography;
using System.IO;

//

using( FileStream   inputFile    = new FileStream( @"C:\VeryLargeFile.bin", FileMode.Open, FileAccess.Read, FileShare.None, bufferSize: 1024 * 1024, useAsync: true ) ) // When using `useAsync: true` you get better performance with buffers much larger than the default 4096 bytes.
using( CryptoStream base64Stream = new CryptoStream( inputFile, new ToBase64Transform(), CryptoStreamMode.Read ) )
using( FileStream   outputFile   = new FileStream( @"C:\VeryLargeBase64File.txt", FileMode.CreateNew, FileAccess.Write, FileShare.None, bufferSize: 1024 * 1024, useAsync: true ) )
{
    await base64Stream.CopyToAsync( outputFile ).ConfigureAwait(false);
}

Answer 3

A simple Stream extension method would do the job:

public static class StreamExtensions
{
    public static string ConvertToBase64(this Stream stream)
    {
        if (stream is MemoryStream memoryStream)
        {
            return Convert.ToBase64String(memoryStream.ToArray());
        }

        var bytes = new Byte[(int)stream.Length];

        stream.Seek(0, SeekOrigin.Begin);
        stream.Read(bytes, 0, (int)stream.Length);

        return Convert.ToBase64String(bytes);
    }
}

The methods for Read (and also Write) and optimized for the respective class (whether is file stream, memory stream, etc.) and will do the work for you. For simple task like this, there is no need of readers, and etc.

The only drawback is that the stream is copied into byte array, but that is how the conversion to base64 via Convert.ToBase64String works unfortunately.

Answer 4

You may try something like that:

    public Stream ConvertToBase64(Stream stream)
    {
        Byte[] inArray = new Byte[(int)stream.Length];
        Char[] outArray = new Char[(int)(stream.Length * 1.34)];
        stream.Read(inArray, 0, (int)stream.Length);
        Convert.ToBase64CharArray(inArray, 0, inArray.Length, outArray, 0);
        return new MemoryStream(Encoding.UTF8.GetBytes(outArray));
    }

Answer 5

You can also encode bytes to Base64. How to get this from a stream see here: How to convert an Stream into a byte[] in C#?

Or I think it should be also possible to use the .ToString() method and encode this.

Answer 6

Since the file will be larger, you don't have very much choice in how to do this. You cannot process the file in place since that will destroy the information you need to use. You have two options that I can see:

1. Read in the entire file, base64 encode, re-write the encoded data.
2. Read the file in smaller pieces, encoding as you go along. Encode to a temporary file in the same directory. When you are finished, delete the original file, and rename the temporary file.

Of course, the whole point of streams is to avoid this sort of scenario. Instead of creating the content and stuffing it into a file stream, stuff it into a memory stream. Then encode that and only then save to disk.

Answer 7

The answers recommending the use of ToBase64Transform are valid, but there is a big catch. Not sure if this should be an answer, but had I known this it would have saved me a lot of time.

The problem I ran into with ToBase64Transform is that it is hard-coded to read 3 bytes at a time. If each write to the input stream specified in CryptoStream constructor is something like a websocket or anything that has non trivial overhead or latency, this can be a huge problem.

Bottom line - if you are doing something like this:

using var cryptoStream = new CryptoStream(httpRequestBodyStream, new ToBase64Transform(), CryptoStreamMode.Write);

It may be worthwhile to fork the class ToBase64Transform to modify the hard-coded 3/4 byte values to something substantially larger so that it incurs fewer writes. In my case, with the default 3/4 value, transmission rate was about 100 KB/s. Changing to 768/1024 (same ratio) worked and transmission rate was about 50-100 MB/s because of way fewer writes.

    public class BiggerBlockSizeToBase64Transform : ICryptoTransform
    {
        // converting to Base64 takes 3 bytes input and generates 4 bytes output
        public int InputBlockSize => 768;
        public int OutputBlockSize => 1024;
        public bool CanTransformMultipleBlocks => false;
        public virtual bool CanReuseTransform => true;

        public int TransformBlock(byte[] inputBuffer, int inputOffset, int inputCount, byte[] outputBuffer, int outputOffset)
        {
            ValidateTransformBlock(inputBuffer, inputOffset, inputCount);

            // For now, only convert 3 bytes to 4
            byte[] tempBytes = ConvertToBase64(inputBuffer, inputOffset, 768);

            Buffer.BlockCopy(tempBytes, 0, outputBuffer, outputOffset, tempBytes.Length);
            return tempBytes.Length;
        }

        public byte[] TransformFinalBlock(byte[] inputBuffer, int inputOffset, int inputCount)
        {
            ValidateTransformBlock(inputBuffer, inputOffset, inputCount);

            // Convert.ToBase64CharArray already does padding, so all we have to check is that
            // the inputCount wasn't 0
            if (inputCount == 0)
            {
                return Array.Empty<byte>();
            }

            // Again, for now only a block at a time
            return ConvertToBase64(inputBuffer, inputOffset, inputCount);
        }

        private byte[] ConvertToBase64(byte[] inputBuffer, int inputOffset, int inputCount)
        {
            char[] temp = new char[1024];
            Convert.ToBase64CharArray(inputBuffer, inputOffset, inputCount, temp, 0);
            byte[] tempBytes = Encoding.ASCII.GetBytes(temp);
            if (tempBytes.Length != 1024)
                throw new Exception();

            return tempBytes;
        }

        private static void ValidateTransformBlock(byte[] inputBuffer, int inputOffset, int inputCount)
        {
            if (inputBuffer == null) throw new ArgumentNullException(nameof(inputBuffer));
        }

        // Must implement IDisposable, but in this case there's nothing to do.

        public void Dispose()
        {
            Clear();
        }

        public void Clear()
        {
            Dispose(true);
            GC.SuppressFinalize(this);
        }

        protected virtual void Dispose(bool disposing) { }

        ~BiggerBlockSizeToBase64Transform()
        {
            // A finalizer is not necessary here, however since we shipped a finalizer that called
            // Dispose(false) in desktop v2.0, we need to keep it in case any existing code had subclassed
            // this transform and expects to have a base class finalizer call its dispose method.
            Dispose(false);
        }
    }