How to deal with huge amount of byte arrays

Question

So I'm receiving byte arrays from USB device (some log messages) and I'm facing an issue, I don't know what is the best way to parse or read them...

So this is the receiver:

static class ReceiveLogsThread implements Runnable {
    private static final String TAG = "IoTReceiveLogsThread";
    Message msgRead;

    ReceiveLogsThread() {
    }

    public void run() {
        byte[] rbuf = new byte[4096];
        while (!Thread.currentThread().isInterrupted()) {
            try {
                int len = mSerial.readLog(rbuf, mSerialPortLog);
                if (len > 0) {
                //    Crashlytics.log(Log.DEBUG, TAG, "ReceiveLogsThread: " + printHex(rbuf));
                //    this.msgRead = receiveLogsHandler.obtainMessage(HANDLE_READ, printHex(rbuf));
                    this.msgRead = receiveLogsHandler.obtainMessage(HANDLE_READ, rbuf);
                    receiveLogsHandler.sendMessage(this.msgRead);
                }
            } catch (NullPointerException e1) {
                e1.printStackTrace();
            } catch (Exception e) {
                e.printStackTrace();
            }
        }
        Thread.currentThread().interrupt();
        if (!mReadFlag) {
            Crashlytics.log(Log.WARN, TAG, "Receive thread finished");
        }
    }
}

As you can see, printHex() method is commented because I think it caused me an issue with losing some of those messages due to real time parsing, as you can see from it's implementation

   private static String printHex(byte[] bytes) {
    Formatter formatter = new Formatter();

    for (byte b : bytes) {
        formatter.format("%02x", b);
    }
    String hex = formatter.toString();
    return hex;
}

I don't thinks it's a good idea to do a printHex method as soon as I receive byte array, because bytes are comming so fast, so I want to try another way..

I want to send them as byte arrays and then parse them after I'm done with everything, so I'm not sure how to implement it right...

Here is a receive handler from my activity where I'm storing those arrays into List of byte arrays that could contain like 30000 byte arrays:

    private List<byte[]> logs = new ArrayList<>();

Handler receiveLogsHandler = new Handler(Looper.getMainLooper()) {
    public void handleMessage(Message msgRW) {
        super.handleMessage(msgRW);
        //   logMessagesList.add(msgRW.obj.toString().toUpperCase());
        //      String message = msgRW.obj.toString().toUpperCase();

        if(shouldCollectLogs) {
            byte[] message = (byte[]) msgRW.obj;
            logs.add(message);
        }

....

So the problem I'm facing here is, how to combine all those byte arrays into one! And then do printHex on that one big array..

Answer 1

Okay, so here are two alternatives to process the final list of byte arrays at the end, depending on what you want.

Alternative 1: Collect to List of String

List<String> hexStrings = listOfBytes.parallelStream() //Process in paralell each byte array
            .map(bytes -> printHex(bytes)) // Map each byte array from byte[] to String
            .collect(Collectors.toList());// Collect the streams into one final list

Alternative 2: Collect to one String

String oneBigStringOfBytesInHex = listOfBytes.parallelStream() //Process in paralell each byte array
                .map(bytes -> printHex(bytes)) // Map each byte array from byte[] to String
                .collect(Collectors.joining()); // Concat the streams into one final String

You should be careful with the second alternative, you have to consider that the maximum size of a String is 2147483647 (2^31 - 1). Anyway, if your final amount of data is a List of 30.000 bytes arrays of 4096 bytes each, you shouldn't have any problem.

Answer 2

Your printToHex function should be as follows, passing the bytes read count. I copied some of the code from a different post.

private final static char[] hexArray = "0123456789ABCDEF".toCharArray(); 
public static String printToHex(byte[] bytes, int len) { 
    char[] hexChars = new char[len * 2]; 
    for ( int j = 0; j < len; j++ ) { 
        int v = bytes[j] & 0xFF; 
        hexChars[j * 2] = hexArray[v >>> 4];
        hexChars[j * 2 + 1] = hexArray[v & 0x0F]; 
    }
    return new String(hexChars); 
}