Convert byte in hex to actual byte

Question

I have a file that is written in bytes like this

\r\x00\x00\x00\xd0{"a": "test"}

which has the following bytes

[13, 0, 0, 0, -48, 123, 34, 97, 34, 58, 32, 34, 116, 101, 115, 116, 34, 125]

when this file is read into Java I'm getting everything escaped

\\r\\x00\\x00\\x00\\xd0{"a": "test"}

when I do a .getBytes() on this string I get

[92, 114, 92, 120, 48, 48, 92, 120, 48, 48, 92, 120, 48, 48, 92, 120, 100, 48, 123, 34, 97, 34, 58, 32, 34, 116, 101, 115, 116, 34, 125]

I have to convert the string to the valid bytes, I don't have the ability to change how a file is read unfortunately. I know in Python, you open a file with the 'rb' mode and you're good to go. If java has that ability, I can't use it.

So in short, how can I convert the string Java reads into the original byte array that was written to the file?

Sorry if this question is stupid simple, but I am so green when it comes to Java.

EDIT: So I believe my question is different than the proposed "duplicate question" link. It's not taking every literal value in the java string and converting that back to a byte. The string in java has been escaped by the reader. \x00 is now \\x00 which is not the same byte value. So I guess I need some way of unescaping the string?

The file as viewed in a hex editor

0000000: 5c72 5c78 3030 5c78 3030 5c78 3030 5c78  \r\x00\x00\x00\x
0000010: 6430 7b22 6122 3a20 2274 6573 7422 7d0a  d0{"a": "test"}.

The string that java gets viewed in a hex editor

0000000: 5c5c 725c 5c78 3030 5c5c 7830 305c 5c78  \\r\\x00\\x00\\x
0000010: 3030 5c5c 7864 307b 2261 223a 2022 7465  00\\xd0{"a": "te
0000020: 7374 227d 0a                             st"}.

Well explained and formatted questions are never stupid, at least for me. — Maroun, Apr 24 '15 at 17:04
The results you have from getBytes(), is that what you're expecting? — Shar1er80, Apr 24 '15 at 17:10
@toadzky sorry, I'm not exactly sure how the file is being read. I'm extending a plugin for `Hive` so I'm guessing it's being read in `Hive` somewhere. — Jeff, Apr 24 '15 at 17:14
And you're file literraly contains "\r\x00\x00\x00\xd0{"a": "test"}"? — Shar1er80, Apr 24 '15 at 17:14
@Shar1er80 yes, if you were to `cat` the file you would see that string but without the surrounding quotation marks. — Jeff, Apr 24 '15 at 17:16
@Jeff Please post the Java code that you are using to read the file. — bakoyaro, Apr 24 '15 at 17:18
Come on people, this question gets asked every single day, please VTC-duplicate — durron597, Apr 24 '15 at 17:18
If you need to convert escaped sequences (like both \r and \xd0) into single bytes, there is probably no simple solution in Java for that. Check this one http://stackoverflow.com/questions/3537706/howto-unescape-a-java-string-literal-in-java — Alex Nevidomsky, Apr 24 '15 at 17:53

score 2 · Accepted Answer · edited Aug 29 '18 at 18:31

In Java, you're going to have to interpret the input string to get the byte values you want.

I wrote a Java application that interprets the input string.

Here's the input string:

\r\x00\x00\x00\xd0{"a": "test"}

Here's the result:

[13, 0, 0, 0, -48, 34, 97, 34, 58, 32, 34, 116, 101, 115, 116, 34, 125]

And here's the code. You'll probably have to modify the code somewhat to handle cases you didn't put in your question.

package com.ggl.testing;

import java.io.BufferedReader;
import java.io.FileNotFoundException;
import java.io.IOException;
import java.io.InputStreamReader;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.List;

public class ConvertBytes implements Runnable {

    private String fileName;

    public static void main(String[] args) {
        new ConvertBytes("bytes.txt").run();
    }

    public ConvertBytes(String fileName) {
        this.fileName = fileName;
    }

    @Override
    public void run() {
        BufferedReader br = null;

        try {
            br = new BufferedReader(new InputStreamReader(getClass()
                    .getResourceAsStream(fileName)));
            String line = "";
            while ((line = br.readLine()) != null) {
                processLine(line);
            }
        } catch (FileNotFoundException e) {
            e.printStackTrace();
        } catch (IOException e) {
            e.printStackTrace();
        } finally {
            try {
                if (br != null) {
                    br.close();
                }
            } catch (IOException e) {
                e.printStackTrace();
            }
        }
    }

    private void processLine(String line) {
        String[] parts = line.split("(?=\\\\)");
        List<Byte> byteList = new ArrayList<Byte>();

        for (int i = 0; i < parts.length; i++) {
            if (parts[i].equals("")) {
                continue;
            } else {
                byteList.addAll(getValue(parts[i]));
            }
        }

        Byte[] bytes = byteList.toArray(new Byte[byteList.size()]);
        System.out.println(Arrays.toString(bytes));
    }

    private List<Byte> getValue(String s) {
        List<Byte> byteList = new ArrayList<Byte>();

        if (s.startsWith("\\x")) {
            int value = Integer.valueOf(s.substring(2, 4), 16);
            if (value > 127) {
                value = value - 256;
            }
            byteList.add(Byte.valueOf((byte) value));
            if (s.length() > 4) {
                byteList.addAll(getAsciiValue(s.substring(4)));
            }
        } else if (s.equals("\\r")) {
            byteList.add(Byte.valueOf((byte) 13));
        } else if (s.equals("\\t")) {
            byteList.add(Byte.valueOf((byte) 9));
        } else {
            byteList.addAll(getAsciiValue(s));
        }

        return byteList;
    }

    private List<Byte> getAsciiValue(String s) {
        List<Byte> byteList = new ArrayList<Byte>();

        for (int i = 0; i < s.length(); i++) {
            int value = (int) s.charAt(i);
            byteList.add(Byte.valueOf((byte) value));
        }

        return byteList;
    }

}

The bytes.txt file has to be in the same directory as the Java application.

Shar1er80 · Answer 2 · 2015-04-24T19:06:29.520

Looks like your have to parse the "String" line yourself.

I would have a Map of escaped characters ('\r', '\n', '\b', etc...)

private static Map<String, Byte> escapedCharacters; 
static {
    escapedCharacters = new HashMap<>();
    escapedCharacters.put("\\b", (byte)'\b');
    escapedCharacters.put("\\f", (byte)'\f');
    escapedCharacters.put("\\n", (byte)'\n');
    escapedCharacters.put("\\r", (byte)'\r');
    escapedCharacters.put("\\t", (byte)'\t');
    // Add more if needed
};

Then the following to process your file:

public static void main(String[] args) throws Exception {
    String myFile = "PathToYourFile";

    // Read your file in
    List<String> myFileLines = Files.readAllLines(Paths.get(myFile));

    // List to hold all the lines as translated bytes
    List<byte[]> myFileLinesAsBytes = new ArrayList<>();
    for (String line : myFileLines) {
        myFileLinesAsBytes.add(translateEscapedBytes(line));
    }

    // Displays all translated lines
    for (byte[] byteLine : myFileLinesAsBytes) {
        System.out.println(Arrays.toString(byteLine));
    }
    System.out.println();
}

private static byte[] translateEscapedBytes(String line) throws UnsupportedEncodingException {
    List<Byte> translatedBytes = new ArrayList<>();
    for (int i = 0; i < line.length();) {
        if (line.charAt(i) == '\\') { // Escaped byte
            String escapedByte = line.substring(i, i + 2);
            if (escapedByte.endsWith("x")) { // Hexidecimal number
                escapedByte = line.substring(i + 2, i + 4); // + 4 to get the two numbers after \x
                translatedBytes.add(hexStringToByte(escapedByte));
                i += 4;
            } else { // Escaped character
                translatedBytes.add(escapedCharacters.get(escapedByte));
                i += 2;
            }
        } else { // Non Escapted Character
            translatedBytes.add((byte)(line.charAt(i)));
            i++;
        }
    }

    // Copy List to actual byte[] to return
    byte[] result = new byte[translatedBytes.size()];
    for (int i = 0; i < translatedBytes.size(); i++) {
        result[i] = translatedBytes.get(i);
    }
    return result;
}

private static byte hexStringToByte(String s) {
    return (byte) ((Character.digit(s.charAt(0), 16) << 4) + Character.digit(s.charAt(1), 16));
}

translatedEscapedBytes() looks for the "\" character in the string and identifies that combined with the next character, you'll have an escaped character. If the escaped character is an \x, then you know the next two numbers is a hexadecimal number that needs to be converted into a byte (hexStringToByte(String s)), otherwise covert the escaped character to a byte using the map of escaped characters. All other characters are treated as non-escaped characters and just get converted to their byte value.

Results (Using the data you provided):

enter image description here

score 0 · Answer 3 · answered Apr 24 '15 at 17:51

You aren't getting "everything escaped" when the file is read in Java. Why do you think that? The conversion to bytes shows that the String contains exactly what the hex editor shows in the file. In other words,

92, 114, 92, 120, 48, 48, 92, 120, 48, 48, 92, 120, 48, 48, 92, 120 (decimal)

is the same as

5c72 5c78 3030 5c78 3030 5c78 3030 5c78 (hex)

If you want to decode the escape sequences that are in the file, you'll need to write some code to handle them; it's not a character encoding issue.

Convert byte in hex to actual byte

3 Answers3