tail -F without interruptions

Question

I am tailing a file. I want to remove lines from the file while I am tailing it. I'd like to avoid overwriting the file (truncating) and I would like to avoid replacing the file with a new file, because this most likely messes up / corrupts the tail command results.

Currently I have tried two different ways of doing this:

1. Read the whole contents of the file, remove the unwanted lines of data, then write back to the file with less data than before. This results in some stderr spewed from the tail command => "file was truncated"...tail -F is stilling working, but it does log this stderr.

2. Use sed -i '/pattern/d' my-file.txt to delete lines from the file that I no longer want. This results in some stderr spewed from the tail command => "file was replaced" (note different than above)...tail -F is still working, but it does log this stderr.

I am wondering if there is a way to delete lines from a file without truncating the file or replacing the file, as this seems to make life a little bit harder for tail than otherwise.

Should I just ignore this stderr? If I just ignore the stderr, I think the tail results will just be inaccurate. I need the tail results to be as accurate as possible because they are feeding into a new program, not being read by a human.

Answer 1

One workaround I'm seeing would be:

• open the file in read/write
• identify the line to be removed
• instead of removing it, replace the previous linefeed+the contents of the line by space characters.

before replacement:

aaaaaaa\n
bbbbbbb\n
ccccccc\n

after replacement:

aaaaaaa        \n
ccccccc\n

Visually, the log has the line removed.

If you don't mind the extra spaces/can perform an off-line cleanup using sed 's/ *$//g', you're good, since opening in read-write does not change the location of unchanged data or the file node.

As a bonus, this is very fast, because even if the file is huge, you're just changing a few bytes, not rewriting the whole file.

I had a tough time writing this python implementation which works:

import re,os,sys
logfile = sys.argv[1]
regex = sys.argv[2]
replacement_char = " "  # default: space
if len(sys.argv)>3:
   replacement_char = sys.argv[3][0]  # first char of 3rd arg

pattern = re.compile(regex)

with open(logfile,"r+") as f:
    while True:
        old_offset = f.tell()
        l = f.readline()
        if not l:
            break
        if pattern.search(l):
            # match: blank the line
            new_offset = f.tell()
            if old_offset > len(os.linesep):
                old_offset-=len(os.linesep)
            f.seek(old_offset)
            f.write(replacement_char*(new_offset-old_offset-len(os.linesep)))

How to use:

blank.py logfile regex <optional replacement char>

How it works:

• opens the file in read/write mode
• loops on the lines
• stores the current file offset
• read a line
• if matches the regex, get current offset, rewind to previous file offset and writes the appropriate number of blanks/replacement chars specified, removing the previous linefeed so the blanks are after a valid line, so visually it's the same as if the line were removed.
• since the file is open in read/write mode, an external program reading it (on Linux) won't notice the changes since size & inode doesn't change: no more warnings from tee

Since it overwrites the previous linefeed, it just adds blanks/replacement chars to the previous line.

The only problem as you already noted is that if the first line matches, then it puts replacement chars in it. It's the only time it is visible. As a workaround, you could start your logfiles with a special, non-matchable header.