0

Env: Windows 7, java 1.8, default OS encodings

I'm trying to read a byte stream of currency market data from a socket to a file, and then play that file back to simulate the market over a fixed period; however, the file has a few malformed bytes, seemingly at random.

Below, I outline the problem with metacode, where the notation "..." indicates skipped irrelevant or boilerplate code.

Bytes are coming over the socket, and I'm reading them with a non-blockingNIO selector, then writing to disk via BufferedOutputStream:

class SocketReadDiskWrite implements Runnable{
    ...
    blobWriter = new BufferedOutputStream(new FileOutputStream(blobFileName));  
    sc = SocketChannel.open(addr)
    sc.configureBlocking(false);

And then in the run() method, when the selector deems the socket readable, 

public void run(){
...
while(keyIterator.hasNext())
    {
    SelectionKey key = keyIterator.next();
    if (key.isReadable()) {
        if(bytesRead == -1)
        {
            connected = false;
            logger.warn("no bytes to read");
            break;
        }

        readBuffer.flip();

        // Write bytes from socket to file, then rewind and process data
        while (readBuffer.hasRemaining()){ 
            byte[] b = new byte[readBuffer.remaining()];
            readBuffer.get(b);
            blobWriter.write(b);
        }
        readBuffer.rewind();
        processData(readBuffer);  //<-- Further processing
        ...
    }

The processData method works fine when reading from a live stream of the market. For example, maybe processData reads a list of currencies and prints them, and the output is,

`EUR.USD.SPOT, EUR.AUD.SPOT, ..<thousands more>.. AUD.CAD.SPOT`

However, if I instead try to play back the captured bytestream (ie. Read in the contents of the file that was just previously created), on occasion, a corrupt symbol appears,

`EUR.USD.SPOT, EUR.AUD.SPOT, ..<thousands more>.. AUD.C@#$@##X`

Looking at the file in notepad++, indeed I find incorrect bytes (blue = correct symbols, red = malformed).

enter image description here

Subsequently, when the application points to the bytefile reader (instead of live market), the app fails at exactly these lines, throwing errors like Invalid symbol: EUR.-XD@#O@#$.

For what it's worth, this is how I playback the file by reading it from disk and streaming to socket:

class FilePlayer implements runnable (Socket clientSocket) {
    clientWriter= clientSocket.getOutputStream();
    blobReader = new FileInputStream(blobFileName);
    byte[] dataArray = new byte[1024]; //<-- Store 1024 bytes data at a time
    ...
    }

    public void run() {
       while(true){
        blobReader.read(dataArray);    //<-- Read 1024 bytes of data from disk
        clientWriter.write(dataArray); //<-- Write 1024 bytes of data to socket
        }
       }

Note, I recently opened a related thread similar thread, but that was in regard to FileChannels, which were actually not the culprit. Figured that discussion had deviated enough to warrant a fresh post.

Community
  • 1
  • 1
Adam Hughes
  • 14,601
  • 12
  • 83
  • 122
  • There is no code here that reads from the socket channel. Your final loop to copy the file to the socket channel ignores the result returned by `read()`. – user207421 Nov 21 '16 at 20:30
  • EJP, sorry there was a typo in my final codeblock. `blobReader.read(dataArray)` copies 1024 bytes from disk to a byteArray, then clientWriter writes those to a socket. When you say there is "no code here that reads from the socket channel", do you mean in the final block? – Adam Hughes Nov 21 '16 at 21:01

0 Answers0