Replacing Multiple String Values in Files

Question

I am trying to cleanup some files I get on a quarterly basis. They have a bunch of repeating headers and I would like to replace multiple string values at a single time. I can remove one string at a time, but I am not understanding how I can stream the file and look at each line and remove if it is String 1 or String 2.

Each file has at least 100-300 thousand lines and I get between 10 and 50 files each time the data is dumped to me about once a quarter... Would be easier if they didn't add these lines, but that is not an option.

Sorry for the newbie question, but I don't get to code very often. Any help is appreciated...

static void Main(string[] args)
{
    string tempFile = Path.GetTempFileName();
    string t1 = "-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------";

    string fName = "C:\\PoC\\test\\test.txt";
    using (var sr = new StreamReader(fName))
    using (var sw = new StreamWriter(tempFile))
    {
        string line;

        while ((line = sr.ReadLine()) != null)
        {

            if (line.Contains(t1) == false)
            {
                sw.WriteLine(line);
            }
        }
        sr.Close();
        sw.Close();
    }

    File.Delete(fName);
    File.Move(tempFile, fName);
}

Answer 1

Calling string.Contains() is almost as expensive as calling string.Replace() because in either case the entire line must be scanned for your substring. In the case of Replace() finding a match it creates and returns a new string representing the result of the replacement, otherwise it returns the original string. Change

if (line.Contains(t1) == false)
    sw.WriteLine(line);

to

sw.WriteLine(line.Replace(t1, whatYouWantToReplaceWith));

If you are replacing multiple values in a single line, you can write

sw.WriteLine(
    line
    .Replace(t1, whatYouWantToReplaceWith1)
    .Replace(t2, whatYouWantToReplaceWith2)
    .Replace(t3, whatYouWantToReplaceWith3)
);

Note that using multiple .Replace() will cause the line to be scanned for matches multiple times. Though this reduces performance slightly, most of your processing time will probably still be file IO.

If you know that the replacement will only ever happen e.g. in the first line, you can add a counter to track what line number you are on and only apply the Replace() code to appropriate line(s).

Note that you might get some additional improvement on a large file by using a BufferedStream.

UPDATE

Based on the statement that you just want to remove the line, I suggest you go with @Eser's answer.

Answer 2

I would like to replace multiple string values at a single time.

Using Linq can make your code simpler

string[] stringsToRemove = new[] { "str1", "str2", "str2" };

var query = File.ReadLines(fName)
                .Where(line => !stringsToRemove.Any(s => line.Contains(s)));

File.WriteAllLines(tempFile, query);

Answer 3

I know you are working on a c# program, if the purpose is simply to remove the lines that matches the patter then I'd use something like Unix Stream editor sed command, see sed for Windows stand alone command, or cygwin and you could simply use the command to remove all lines that matches the pattern and lines following it as well, you can write a .bat script to copy, rename, remove lines that matching more than one patterns. It is really fast as well.

sed -i '/^---------/d' filepath