How to write large data into file c#

Question

I have to write a large data (12L)records in a csv file.I am using Streamwriter, to write in file but after around 6L records it gives me outofmemory error.

also I am writing this file on server directly, is there any way I can than download this on browser later?

This is my code: searchResult.Entities=is a list of EntityObject

using (StreamWriter fs = new StreamWriter
(new FileStream("Temp.csv", FileMode.Create)))
                    {
                        AddText(fs, headerString.ToString());
                        foreach (var org in searchResult.Entities.ToList())
                        {
                            StringBuilder sb = new StringBuilder();

                            appendText(sb, org.Id);
                            appendText(sb, org.LeadNo.HasValue ? org.LeadNo.Value.ToString() : "");


                            fs.WriteLine(sb.ToString());

                           foreach (var activity in org.Activities)
                           {
                              var activityString = new StringBuilder();
                             activityString.Append(" , , , , , , , , , , , , , , , , , ,");

                               appendText(activityString, org.Id);
                                appendText(activityString, org.LeadNo.HasValue ? org.LeadNo.Value.ToString() : "");

                            fs.WriteLine(activityString.ToString());

                           }

                        }

Getting all results from database int the object only take 100ms, but writing in file is around 1minute for 6L records.

Maybe this will help you: http://stackoverflow.com/a/9643111/4610605 — Felix D., Oct 17 '16 at 11:18
I may assume that problem is not in the StreamWriter but in the following line that forces loading ALL data into memory: "searchResult.Entities.ToList()" — Vitaliy Fedorchenko, Oct 17 '16 at 11:22
Have you tried using a library like Linq2CSV? It handles lots of edge-cases for you (what do you do if one of your fields contains a comma, for example), and is likely to be more performant. — RB., Oct 17 '16 at 11:22

score 0 · Answer 1 · answered Oct 17 '16 at 11:32

0

You may try not to access all the properties

foreach (var org in searchResult.Entities.ToList()) //Change this

but just the ones which you needed:

var toItrate = searchResult.Entities.Select(s => new { Id = s.Id, LeadNo = (s.LeadNo.HasValue ? s.LeadNo.Value : String.Empty) }).ToList();

answered Oct 17 '16 at 11:32

kashi_rock

539
1
5
19

2

Why using `ToList` at all? wouldn't it be better to just enumerate the entities while writing the file? – grek40 Oct 17 '16 at 11:36

How to write large data into file c#

1 Answers1