Reading huge txt files from C++?

Question

I'm trying to read a huge txt through c++. It has 70mb. My objective is to substring line by line and generate another smaller txt containing only the information that I need.

I got to the code below to read the file. It works perfectly with smaller files, but not with the 70mb monster.

#include "stdafx.h"
#include <iostream>
#include <fstream>
#include <string>

using namespace std;

int main()
{
  ifstream myReadFile;
  myReadFile.open("C:/Users/Lucas/Documents/apps/COTAHIST_A2010.txt");
  char output[100];
  if (myReadFile.is_open()) {
    while (myReadFile.eof()!=1) {
         myReadFile >> output;
         cout<<output;
         cout<<"\n";
     }


    }
  system("PAUSE");
  return 0;
}

This is the error I get: Unhandled exception at 0x50c819bc (msvcp100d.dll) in SeparadorDeAcoes.exe: 0xC0000005: Access violation reading location 0x3a70fcbc.

If someone can point a solution in C or even in C#, that would be acceptable too!

Thanks =)

does it die immediately? Midway through processing? At the end of processing the file? — Mike McMahon, May 03 '12 at 17:13
The way your input loop tests for EOF is a [bad practice](http://stackoverflow.com/questions/5605125/why-is-iostreameof-inside-a-loop-condition-considered-wrong). — Blastfurnace, May 03 '12 at 18:23

Anya Shenanigans · Accepted Answer · 2012-05-04T08:20:43.783

6

your char output[100] buffer is not able to take the content of one of the lines.

Ideally you should use a string destination, and not a char[] buffer.

Edit As has been pointed out, this is bad practice, and leads to reading the last line twice or a stray empty last line. A more correct writing of the loop would be:

string output;
while (getline(myReadFile, output)) {
  cout<<output<<"\n";
}

**Edit - Leaving the bad, evil code here:

A quick rewrite of your inner while loop might be:

string output;
while (myReadFile.good()) {
  getline(myReadFile, output);
  cout<<output<<"\n";
}

edited May 04 '12 at 08:20

answered May 03 '12 at 17:12

Anya Shenanigans

91,618
3
107
122

The way you test for EOF in this code is a [bad practice](http://stackoverflow.com/questions/4324441/testing-stream-good-or-stream-eof-reads-last-line-twice). – Blastfurnace May 03 '12 at 18:21

score 2 · Answer 2 · answered May 03 '12 at 17:12

2

I think that your problem is that one of your lines is over 100 characters long. Need to increase the size of the character array.

answered May 03 '12 at 17:12

Ed Heal

59,252
17
87
127

score 0 · Answer 3 · answered May 03 '12 at 17:18

You are not using std::string, but you include the header file. Decide. Use either std::string or character array.

Also, use std::istream::read and provide the size of the array to the function. You will need to repeat many times as 100 characters is far smaller than 70mb.

Try allocating a much larger array using dynamic memory:

const unsigned int array_size = 1024 * 1024 * 1024;

int main(void)
{
  char * output;
//...
  output = new char [array_size];
// read into output
// ...
// clean up
  delete [] output;
  return EXIT_SUCCESS;
}

If you use std::string, use the constructor that takes a size parameter so you can specify the initial size of the string.

Reading huge txt files from C++?

3 Answers3