Reading file line by line in dart

Question

I'm trying to process large text files in a language Dart. The files have a size over 100 MB.

I tried readAsLines and readAsLinesSync methods of dart:io library. Every time I run out of memory: Exhausted heap space.

Is there a way to read a file line by line or byte by byte as in other languages?

score 26 · Answer 1 · edited Dec 20 '19 at 10:36

26

This should read the file in chunks:

import 'dart:async';
import 'dart:io';
import 'dart:convert';

main() {
  var path = ...;
  new File(path)
    .openRead()
    .map(utf8.decode)
    .transform(new LineSplitter())
    .forEach((l) => print('line: $l'));
}

There isn't much documentation about this yet. Perhaps file a bug asking for more docs.

edited Dec 20 '19 at 10:36

Tom Yeh

1,987
2
15
23

answered Feb 16 '14 at 20:56

Greg Lowe

15,430
2
30
33

score 13 · Answer 2 · edited Feb 05 '23 at 21:57

13

In the latest verion of dart the UTF8.decoder is now utf8.decoder:

import 'dart:async';
import 'dart:io';
import 'dart:convert';

main() {
  var path = ...;
  File(path)
    .openRead()
    .transform(utf8.decoder)
    .transform(LineSplitter())
    .forEach((l) => print('line: $l'));
}

edited Feb 05 '23 at 21:57

Hassan

380
6
20

answered Oct 10 '19 at 00:58

Brett Sutton

3,900
2
28
53

1

Downvoted because you should have edited the previous answer. – Michael Pfaff Oct 28 '19 at 17:14
1

I didn't even know you could edit someone else's answer. Is there some etiquette to just updating someelse's answer? – Brett Sutton Oct 28 '19 at 22:09
4

Upvoted because the etiquette is generally to not modify code in other people's answers, though modifying other content is usually fine. However, this could have just been a comment below the answer. – Suragch Jan 17 '20 at 01:51
Thanks for the comment and up vote. As a user of stack overflow I would actually prefer a complete answer (however we arrive at that) as it's too easy to miss notes in the comments section. – Brett Sutton Jan 19 '20 at 05:23
I just posted my own answer, but then I realized that it was the same as yours so I deleted it. In addition to changing `UTF8` to `utf8`, you also replaced the map with another transform. I didn't notice that at first. That map was causing an exception in my code. Could you add an explanation for why you removed map? What does transform mean? (By the way, if you add @Suragch in your comment, I'll get notified when you reply.) – Suragch Jan 24 '20 at 02:27
@suragch The map is used to do translation on the input from one object to another. So you a reading a text file, you do the utf8 transform to convert from bytes (which is what openRead returns, then the second transforms breaks the characters (well in dart there are actually single character strings) to into lines. You could then do a map to transform the lines into say an integer if you knew that the file your reading had a single number on each line. – Brett Sutton Jan 24 '20 at 02:52
Why did you remove the map from your answer that Greg Lowe's answer uses? – Suragch Jan 24 '20 at 05:16
Doesn't work "Error: Unsupported operation: _Namespace" – Eight Rice Oct 26 '21 at 21:40
1

@EightRice look at this article. https://stackoverflow.com/questions/54861467/unsupported-operation-namespace-while-using-dart-io-on-web. Essentially you are using dart:io in a web app which isn't allowed. – Brett Sutton Oct 27 '21 at 22:02

Reading file line by line in dart

2 Answers2

Linked