51

My file reader api code has been working good so far until one day I got a 280MB txt file from one of my client. Page just crashes straight up in Chrome and in Firefox nothing happens. 

// create new reader object 
var fileReader = new FileReader(); 

// read the file as text 
fileReader.readAsText( $files[i] );  
fileReader.onload = function(e) 
{   // read all the information about the file 
    // do sanity checks here etc... 
    $timeout( function() 
    {    
        // var fileContent = e.target.result;
        // get the first line 
        var firstLine = e.target.result.slice(0, e.target.result.indexOf("\n") ); }}

What I am trying to do above is that get the first line break so that I can get the column length of the file. Should I not read it as text ? How can I get the column length of the file without breaking the page on big files?

gsamaras
  • 71,951
  • 46
  • 188
  • 305
ODelibalta
  • 2,194
  • 1
  • 18
  • 28
  • Example of how I managed to handle reading big files in browser line by line: https://stackoverflow.com/a/55377748/2358659 – Maz T Mar 27 '19 at 13:10

2 Answers2

93

Your application is failing for big files because you're reading the full file into memory before processing it. This inefficiency can be solved by streaming the file (reading chunks of a small size), so you only need to hold a part of the file in memory.

A File objects is also an instance of a Blob, which offers the .slice method to create a smaller view of the file.

Here is an example that assumes that the input is ASCII (demo: http://jsfiddle.net/mw99v8d4/).

function findColumnLength(file, callback) {
    // 1 KB at a time, because we expect that the column will probably small.
    var CHUNK_SIZE = 1024;
    var offset = 0;
    var fr = new FileReader();
    fr.onload = function() {
        var view = new Uint8Array(fr.result);
        for (var i = 0; i < view.length; ++i) {
            if (view[i] === 10 || view[i] === 13) {
                // \n = 10 and \r = 13
                // column length = offset + position of \r or \n
                callback(offset + i);
                return;
            }
        }
        // \r or \n not found, continue seeking.
        offset += CHUNK_SIZE;
        seek();
    };
    fr.onerror = function() {
        // Cannot read file... Do something, e.g. assume column size = 0.
        callback(0);
    };
    seek();

    function seek() {
        if (offset >= file.size) {
            // No \r or \n found. The column size is equal to the full
            // file size
            callback(file.size);
            return;
        }
        var slice = file.slice(offset, offset + CHUNK_SIZE);
        fr.readAsArrayBuffer(slice);
    }
}

The previous snippet counts the number of bytes before a line break. Counting the number of characters in a text consisting of multibyte characters is slightly more difficult, because you have to account for the possibility that the last byte in the chunk could be a part of a multibyte character.

Rob W
  • 341,306
  • 83
  • 791
  • 678
  • 6
    You are officially my hero. At first, I had the same idea of reading it as a blob chunks instead of reading the whole thing but I did not know enough to make it work. You have no idea how much I appreciate this. Thank you! – ODelibalta Sep 12 '14 at 19:20
  • What would you recommend if I need to determine the last chunk in the file? The system I'm working with has a different REST api for the last chunk which then commits the entire file. But I can't figure out how to determine that last piece. If you wouldn't mind looking at my question that would be helpful http://stackoverflow.com/questions/39312451/determining-the-last-file-chunk/39312577#39312577 – Batman Sep 04 '16 at 01:30
  • 1
    I am wondering how I can use this to [read `n` lines from a big file](http://stackoverflow.com/questions/39479090/read-n-lines-of-a-big-text-file)..... :/ – gsamaras Sep 13 '16 at 21:39
  • 1
    As for the multibyte character dilemma, [TextDecoder#decode](https://developer.mozilla.org/en-US/docs/Web/API/TextDecoder/decode) now/soon has a `stream` option flag for this very purpose. – Blake Regalia Sep 20 '17 at 21:42
  • @RobW how can we preview large image, chunk by chunk? if image size is >25mb each, total multiple image size is around 900mb to display? How can we set chunk by chunk view on img src? – Developer Jul 16 '20 at 14:31
4

There is a awesome library called Papa Parse that do that in a graceful way! It can really handle big files and also you can use web worker.

Just try out the demos that they provide: https://www.papaparse.com/demo

Edy Segura
  • 517
  • 6
  • 10