Fastest way to read a file line by line?

Question

I currently have a .TXT file and I'm trying to read all the lines, and sort them, and then display them in order.

Here is what I have

                string inFile = "Z:/Daniel/Accounts.txt";
                string outFile = "Z:/Daniel/SortedAccounts.txt";
                var contents = File.ReadAllLines(inFile);
                Array.Sort(contents);
                File.WriteAllLines(outFile, contents);

                int i = 0;
                int lineCount = File.ReadLines("Z:/Daniel/SortedAccounts.txt").Count();
                do
                {
                    string accounts = File.ReadLines("Z:/Daniel/SortedAccounts.txt").Skip(i).Take(1).First();
                    //First Name
                    int pFrom1 = accounts.IndexOf("#1#") + "#1#".Length;
                    int pTo1 = accounts.LastIndexOf("#2#");
                    String accountFirstName = accounts.Substring(pFrom1, pTo1 - pFrom1);
                    //Last Name
                    int pFrom2 = accounts.IndexOf("#2#") + "#2#".Length;
                    int pTo2 = accounts.LastIndexOf("#3#");
                    String accountLastName = accounts.Substring(pFrom2, pTo2 - pFrom2);
                    //Email
                    int pFrom3 = accounts.IndexOf("#3#") + "#3#".Length;
                    int pTo3 = accounts.LastIndexOf("#4#");
                    String accountEmail = accounts.Substring(pFrom3, pTo3 - pFrom3);
                    //Phone Number
                    int pFrom4 = accounts.IndexOf("#4#") + "#4#".Length;
                    int pTo4 = accounts.LastIndexOf("#5#");
                    String accountNumber = accounts.Substring(pFrom4, pTo4 - pFrom4);
                    //Preferred Contact
                    int pFrom5 = accounts.IndexOf("#5#") + "#5#".Length;
                    int pTo5 = accounts.LastIndexOf("#6#");
                    String accountPreferredContact = accounts.Substring(pFrom5, pTo5 - pFrom5);
                    //Populate Combobox
                    accountComboBox.Items.Add(accountLastName + "," + accountFirstName);
                    i = i + 1;
                } while (i < lineCount);

And an example of what's inside Accounts.txt is

#1#Daniel#2#Mos#3#dasdnmasdda@gmail.com#4#31012304#5#EMAIL#6# #1#Daniael#2#Mosa#3#dddasdsa@gmail.com#4#310512304#5#EMAIL#6# #1#Dansdael#2#Mossdsa#3#dasdsdssa@gmail.com#4#31121234#5#TEXT#6# #1#Danasdl#2#Mosasaa#3#daasda@gmail.com#4#310123304#5#EMAIL#6# #1#Dandasel#2#Moasddand#3#daasdsda@gmail.com#4#3123551234#5#TEXT#6# #1#Danasdl#2#Mossdsadd#3#daasddsa@gmail.com#4#310213304#5#TEXT#6#

The issue is that sometimes, Accounts.txt will have over 10,000 lines and it then takes a while for the program to load.

Is there a faster implementation of the code I have written?

in general, it might be faster to read each line and process it instead of reading the entire file into one giant string in memory and then parsing it. there are lots of similar questions, though. — John Gardner, Jan 28 '16 at 01:44
Hi, you already loaded all text in the string[] contents why just use it to loop it all instead of reading again from the output file? — Dr. Stitch, Jan 28 '16 at 01:44
@JohnGardner , well it looks to me as if that thread is saying what I'm doing is already the fastest method. That's unfortunate — IdkHowToCodeAtAll, Jan 28 '16 at 01:44
and instead of all the string.indexof/lastindexof, that looks like a pretty good candiate for a compiled regex that gets you all the values in one pass. — John Gardner, Jan 28 '16 at 01:45
except you aren't just "reading a file". you're: reading a file, SORTING IT, writing it back out, READING IT AGAIN, applying a ton of string manipulations, and then *adding items to a combobox*. that last step might be the worst part of all this, depending on what ui technology this is. — John Gardner, Jan 28 '16 at 01:47
@JohnGardner This is what I came up with for regex, but I get an error now about OutOfRange http://pastebin.com/HtY12yk5 — IdkHowToCodeAtAll, Jan 28 '16 at 01:56
Fastest to read while displaying? [Async](https://msdn.microsoft.com/en-us/library/hh191443.aspx) is what you need. — choz, Jan 28 '16 at 02:38
This is not about reading a file (as it is done with `File.RealAllLines()`) but about parsing a text file. — John Alexiou, Jan 28 '16 at 03:05

score 0 · Answer 1 · answered Jan 28 '16 at 01:53

My suggestions:

read the file line by line, like file.readlines, streaming as you go instead of reading the whole file (especially not reading the file twice the way you are!)
for each line, apply a compiled regex that gets the values you need out of that string
create an account class (or just a single string value i guess) that holds all the values from 2 as necessary. looks like your loop only cared about 2 of the strings (accountLastName and accountFirstName)
add those to a list that isn't the combobox's items.
sort those using linq/sort if you need them sorted (sort as little as necessary and as late as possible), something like items.OrderBy( x => x.LastName ).ThenBy( y => y.FirstName) or whatever
add the entire block of items to your combobox at the very end, instead of one at a time. ideally something like combobox.Items.AddRange(items) (many of the combobox/etc collections might fire a collection change event every time one item is added, that can be a lot of overhead if you're adding 1000's of items)

Everything else makes a lot of sense to me but I already got stuck lol. I can't get the regex to work? http://pastebin.com/HtY12yk5 — IdkHowToCodeAtAll, Jan 28 '16 at 02:09

Bassam Gamal · Answer 2 · 2016-04-19T06:36:44.983

All code should be refactored in below way. You need to measure performance for either both approaches.

const string inFile = "Z:/Daniel/Accounts.txt";
                const string outFile = "Z:/Daniel/SortedAccounts.txt";
                string[] contents = File.ReadAllLines(inFile);
                Array.Sort(contents);
                File.WriteAllLines(outFile, contents);

                IEnumerable<string> lines = File.ReadLines("Z:/Daniel/SortedAccounts.txt");
                foreach (string line in lines)
                {

                    //First Name
                    string[] data = Regex.Split(line, "[#\\d#]");
                    string accountFirstName = data[0];

                    string accountLastName = data[1];

                    string accountEmail = data[2];

                    //Phone Number
                    string accountNumber = data[3];

                    //Preferred Contact
                    string accountPreferredContact = data[4];

                    //Populate Combobox
                    //accountComboBox.Items.Add(accountLastName + "," + accountFirstName);
                }

Edit "Use AddRange"

class Account
    {
        public string FirstName { get; set; }

        public string LastName { get; set; }

        public string Email { get; set; }

        public string Number { get; set; }

        public string PreferredContact { get; set; }

    }
            accountComboBox.Items.AddRange(
                lines.Select(line => Regex.Split(line, "[#\\d#]")).Select(data => new Account
                {
                    FirstName = data[0],
                    LastName = data[1],
                    Email = data[2],
                    Number = data[3],
                    PreferredContact = data[4]
                }).Select(item => string.Format("{0},{1}", item.LastName, item.FirstName)).ToArray()
                );

John Gardner says "add the entire block of items to your combobox at the very end, instead of one at a time. ideally something like combobox.Items.AddRange(items)". How could I do that? — IdkHowToCodeAtAll, Jan 28 '16 at 07:21

Fastest way to read a file line by line?

2 Answers2