How can I get unicode characters from robocopy process standard ouput in c#

Question

Our application runs various actions and displays the output in a log window. One action uses robocopy to copy files between folders.

This works ok until the robocopy output contains unicode characters. I understand that I need to use the /unicode option but all I seem to get back is gibberish.

Here's my simplified code sample:

class Program
{
    static void Main(string[] args)
    {


        StreamReader outputReader = null;
        StreamReader errorReader = null;


        using (Process process = new Process())
        {

            Encoding encoding = Encoding.Default;

            if (encoding != null)
            {
                process.StartInfo.StandardOutputEncoding = encoding;
                process.StartInfo.StandardErrorEncoding = encoding;
            }

            process.StartInfo.FileName = @"C:\Windows\system32\robocopy.exe";
            process.StartInfo.Arguments = @"""D:\temp\некоторые случайные папки"" ""D:\temp\другой случайные папки"" /unicode";
            process.StartInfo.ErrorDialog = false;
            process.StartInfo.LoadUserProfile = false;
            process.StartInfo.RedirectStandardError = true;
            process.StartInfo.RedirectStandardOutput = true;
            process.StartInfo.UseShellExecute = false;
            process.StartInfo.CreateNoWindow = true;

            process.StartInfo.WorkingDirectory = @"D:\temp\некоторые случайные папки";



            bool processStarted = process.Start();
            if (processStarted)
            {
                //Get the output stream
                outputReader = process.StandardOutput;
                errorReader = process.StandardError;
                process.WaitForExit();

                string standardOutput = outputReader.ReadToEnd();
                string errorOutput = errorReader.ReadToEnd();
                if (!string.IsNullOrEmpty(standardOutput))
                {

                    byte[] bytes = encoding.GetBytes(standardOutput);
                    byte[] convertedBytes = Encoding.Convert(encoding, Encoding.UTF8, bytes);

                    string convertedStandardOutput = Encoding.UTF8.GetString(convertedBytes);

                    Console.Write("Standard output: ");
                    Console.WriteLine(convertedStandardOutput);
                }
                if (!string.IsNullOrEmpty(errorOutput))
                {
                    Console.Write("Error output: ");
                    Console.WriteLine(errorOutput);
                }
            }

        }

        Console.ReadKey();
    }
}

I've tried various encoding types and conversions to no avail. Here's the type of output I'm getting:

standardOutput: "ⴊⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭ †佒佂佃奐††㨠›††潒畢瑳䘠汩⁥潃祰映牯圠湩潤獷†††††††††††††††ⴊⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭਊ†瑓牡整⁤›潍摮祡‬㘱䴠牡档㈠㄰‵㐱ㄺ㨵㈵ †潓牵散㨠䐠尺整灭㽜㼿㼿㼿㼿㼠㼿㼿㼿㼿㼠㼿㼿ੜ††䐠獥⁴›㩄瑜浥屰㼿㼿㼿㼠㼿㼿㼿㼿㼠㼿㼿ੜ †䘠汩獥㨠⨠⨮ऊ†† 传瑰潩獮㨠⨠⸀⨀ ⼀唀一䤀䌀伀䐀䔀 ⼀䐀䌀伀倀夀㨀䐀䄀 ⼀䌀伀倀夀㨀䐀䄀吀 ⼀刀㨀㄀　　　　　　 ⼀圀㨀㌀　 ਀ⴊⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭਭऊ†††††††††〠䐉尺整灭㽜㼿㼿㼿㼿㼠㼿㼿㼿㼿㼠㼿㼿ੜⴊⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭⴭਭ †††††††潔慴⁬†䌠灯敩⁤†歓灩数⁤䴠獩慭捴⁨†䘠䥁䕌⁄†䔠瑸慲ੳ††楄獲㨠††††ㄠ††††〠††††〠††††〠††††〠††††〠 †楆敬⁳›††††‰††††‰††††‰††††‰††††‰††††ਰ†䈠瑹獥㨠††††〠††††〠††††〠††††〠††††〠††††〠 †楔敭⁳›†㨰〰〺‰†㨰〰〺‰†††††††††††㨰〰〺‰†㨰〰〺ਰ†䔠摮摥㨠䴠湯慤ⱹㄠ‶慍捲⁨〲㔱ㄠ㨴㔱㔺ਲ"


convertedStandardOutput: "?????????????????????????????????????????†????††?›††??????????????†††††††††††††††?????????????????????????????????????????†????›??????????`?????†??????????????????????††??4›?????????????????†???????††????????? ???????? ????????? ????????? ????       ????  ??????????????????????????????????????????†††††††††????????????????????????????????????????????????????????????†††††††???†????†?????????†???/†????††???††††?††††?††††?††††?††††?††††??†???›††††‰††††‰††††‰††††‰††††‰††††?†????††††?††††?††††?††††?††††?††††??†???›†???‰†???‰†††††††††††???‰†????†????????????????????"

The output displayed when running in a command window is:

 ■
-------------------------------------------------------------------------------
   ROBOCOPY     ::     Robust File Copy for Windows
-------------------------------------------------------------------------------

  Started : Monday, 16 March 2015 14:24:01
   Source : D:\temp\некоторые случайные папки\
     Dest : D:\temp\другой случайные папки\

    Files : *.*

  Options : * . *   / U N I C O D E   / D C O P Y : D A   / C O P Y : D A T   / R : 1 0 0 0 0 0 0   / W : 3 0

------------------------------------------------------------------------------

                           0    D:\temp\некоторые случайные папки\

------------------------------------------------------------------------------

               Total    Copied   Skipped  Mismatch    FAILED    Extras
    Dirs :         1         0         0         0         0         0
   Files :         0         0         0         0         0         0
   Bytes :         0         0         0         0         0         0
   Times :   0:00:00   0:00:00                       0:00:00   0:00:00
   Ended : Monday, 16 March 2015 14:24:01

Any ideas?

Michael Liu · Accepted Answer · 2015-03-25T03:44:50.707

It looks like the /UNICODE option is buggy: the only thing it affects in the console output is the Options : line. (You can tell that this part is Unicode from the spaces between the characters, which are caused by extra null bytes.) ROBOCOPY still seems to write everything else using the system code page. But the /UNICODE option does cause ROBOCOPY to write out a Unicode byte-order mark at the beginning of the output, so StreamReader switches to Unicode no matter what StandardOutputEncoding you set. The result: gibberish.

Instead of /UNICODE, use the /UNILOG option, which appears to work correctly (at least on Windows 8.1):

using (Process process = new Process())
{
    string logFileName = Path.GetTempFileName();
    process.StartInfo.FileName = @"C:\Windows\system32\robocopy.exe";
    process.StartInfo.Arguments = @"""D:\temp\некоторые случайные папки"" ""D:\temp\другой случайные папки"" /UNILOG:" + logFileName;
    process.StartInfo.ErrorDialog = false;
    process.StartInfo.LoadUserProfile = false;
    process.StartInfo.UseShellExecute = false;
    process.StartInfo.CreateNoWindow = true;
    process.StartInfo.WorkingDirectory = @"D:\temp\некоторые случайные папки";
    bool processStarted = process.Start();

    if (processStarted)
    {
        process.WaitForExit();
        string output = File.ReadAllText(logFileName);
        File.Delete(logFileName);
        // TODO: Do something with the output.
    }
}

But see this tale of woe on Windows 7: http://superuser.com/questions/295934/robocopy-unilog-output-is-gibberish/584346 — Michael Liu, Mar 25 '15 at 03:48
That seems to do the trick - thanks. It's a shame that I need to waste IO writing to a file in order to get the output. Hopefully the "all new Microsoft" will fix the /unicode option some time. Now to try running this on the earlier versions of Windows which this software supports... — sparkplug, Mar 25 '15 at 05:11

score 1 · Answer 2 · answered Jul 21 '15 at 03:07

1

You could just create a blank file that is ANSI, instead of the UTF-16 little-endian byte order that the robocopy creates in the header... Use Notepad++ to find out the file encoding.

answered Jul 21 '15 at 03:07

vr_driver

1,867
28
39

score 0 · Answer 3 · edited Jun 20 '20 at 09:12

I found a workaround in this Super User answer by Karan:

If your file or directory names contain Unicode characters then before issuing the Robocopy command with the /unilog parameter use the chcp 65001 command.

Once you have the mangled Unicode log, just open it up in MS Word as Unicode (UTF-8) and save it:

Obviously, you could write your own code to read the file as UTF-8 rather than using MS word.

score 0 · Answer 4 · answered Mar 15 '22 at 16:51

I had a problem using the /unilog+ key on win 8.1 and win10. When using just /unilog (without plus), files with Cyrillic were normally written to a log file, and using unilog+ is already gibberish. This is an annoying bug of robocopy itself: it turned out that /unilog+ simply creates a log file as ANSI. One of the solutions to the problem is to first create a file for the log with utf-8 or unicode encoding and specify the file name in the key. Then the log will be written to a unicode file, and the contents will be saved as needed.

score -1 · Answer 5 · edited May 23 '17 at 11:54

Let me preface this post.. I'm not a c# developer, but I understand file encoding.

Your file has an encoding, and your console has an encoding for stdin and stdout. C# has an encoding for its strings in its runtime.. An encoding is the mapping of symbols to bits. Unicode is an attempt to unify many different types of encoding, blah blah

Ignore robocop for a second, and just figure out getting the console to print correctly.

Get the encoding of the file.

Get the encoding of your console.

Get the identifier which maps to the encoding
Get the encoding behind that identifier.

Now all that remains, is to convert from the file's encoding to the encoding expected by your console before you write to stdout.

Tips: Your console log is displaying Unicode characters fine. My guess is that the file has UTF-16 and your console is expecting UTF-8.

How can I get unicode characters from robocopy process standard ouput in c#

5 Answers5

Linked