How to search for lines in a file between two timestamps using Bash

Question

In bash I am trying to read a log file and will print only the lines that have a timestamp between two specific times. The time format is hh:mm:ss. For example, I would be searching for lines that would fall between 12:52:33 to 12:59:33.

I want to use regular expression because I can use it in grep function.

Each log line begins with some_nr 2014-05-15 21:58:00,000000 rest_of_line.

My solution gives me lines with 1 min margin. I cut out ss and take all lines with hh:mm:[0-9]{2}. $2 has format filename_hh:mm:; for example: "24249_16:05:;24249_16:05:;24249_16:07:;24249_16:07:;24249_16:08:"

My code:

B=$2  

for line in ${B//;/ } ;
do  
    TENT=`echo $line | awk '{split($0,numbers,"_"); print numbers[1]}'`"_logs.txt"
    TIME=`echo $line | awk '{split($0,numbers,"_"); print numbers[2]}'`"[0-9]{2}"

    grep -iE ${TIME} ${TENT} >> ${FILE1}
done

I need a solution with 15 sec margin for any time not 60. I want to have input in format filename_hh:mm:ss and take lines for hh:mm:ss +/- 15s or filename_hh:mm:ss(1)_hh:mm:ss(2) and take lines between hh:mm:ss(1) and hh:mm:ss(2). For sometime there is no lines so the solution should 'recognize' if sometimes match inputted interval or not.

Log files look like this:

1002143 1002143 2014/15/05 22:09:52.937004 bla 
1002130         2014/15/05 22:09:44.786002 bla bla
1001667         2014/15/05 22:09:44.592009 bl a bla
1001667 1001667 2014/15/05 22:09:44.592009 bl a bla

Answer 1

I believe sed is the best option:

sed -rne '/<timestamp>/,/<timestamp>/ p' <file>

ex:

tiago@dell:~$ sed -rne '/08:17:38/,/08:24:36/ p' /var/log/syslog 
May 16 08:17:38 dell AptDaemon.Worker: INFO: Processing transaction /org/debian/apt/transaction/08a244f7b8ce4fad9f6b304aca9eae7a
May 16 08:17:50 dell AptDaemon.Worker: INFO: Finished transaction /org/debian/apt/transaction/08a244f7b8ce4fad9f6b304aca9eae7a
May 16 08:18:50 dell AptDaemon.PackageKit: INFO: Initializing PackageKit transaction
May 16 08:18:50 dell AptDaemon.Worker: INFO: Simulating trans: /org/debian/apt/transaction/37c3ef54a6ba4933a561c49b3fac5f6e
May 16 08:18:50 dell AptDaemon.Worker: INFO: Processing transaction /org/debian/apt/transaction/37c3ef54a6ba4933a561c49b3fac5f6e
May 16 08:18:51 dell AptDaemon.PackageKit: INFO: Get updates()
May 16 08:18:52 dell AptDaemon.Worker: INFO: Finished transaction /org/debian/apt/transaction/37c3ef54a6ba4933a561c49b3fac5f6e
May 16 08:24:36 dell AptDaemon: INFO: Quitting due to inactivity

Answer 2

log file is usually sorted by timestamp, assume the timestamp is on the first column, you could:

awk -v from="12:52:33" -v to="12:59:33" '$1>=from && $1<=to' foo.log

in this way, you can change the from and to to get different set of log entries. regex is not a good tool to do number calculation/comparison.

Answer 3

You can use this regex in egrep:

egrep '12:5[2-9]:33' file.log

Answer 4

You are using the wrong tool for this task. Once you have a regular expression like the one given by @anubhava, you can easily find a time interval that is not matched by it. grep and regexps might work for a few special cases, but they do not scale to the general case.

Can you use some tool that can actually "understand" the timestamps? Probably every scripting language out there (perl, python, ruby, lua) has builtin or library support for parsing time and date.

However, you might be able to employ the powers of GNU date:

% date --date="2014-05-15 21:58:00 15 sec ago" +'%Y-%m-%d %H:%M:%S'
2014-05-15 21:57:45
% date --date="2014-05-15 21:58:00 15 sec" +'%Y-%m-%d %H:%M:%S' 
2014-05-15 21:58:15

and plug that into Tiago's sed filter idea.

Answer 5

You can try the following perl script:

#! /usr/bin/perl

use warnings;
use strict;
use Time::Piece;
use autodie;

my $arg=shift;
my @a =split("_",$arg);
my $fn=shift @a;

my $dfmt='%Y/%d/%m';
my $fmt=$dfmt.' %H:%M:%S';
my $t = localtime;
my $date=$t->strftime($dfmt);
my $t1; my $t2;
if (@a == 1) {
   my $d=$date.' '.$a[0];
   my $tt=Time::Piece->strptime($d, $fmt);
   $t1=$tt-15;
   $t2=$tt+15;
} elsif (@a == 2) {
   $t1=Time::Piece->strptime($date.' '.$a[0], $fmt);
   $t2=Time::Piece->strptime($date.' '.$a[1], $fmt);
} else {
   die "Unexpected input argument!";
}

$fn=$fn.'_logs.txt';
doGrep($fn,$t1,$t2,$fmt);

sub doGrep { 
   my ($fn,$t1,$t2,$fmt) = @_;

   open (my $fh, "<", $fn);
   while (my $line=<$fh>) {
      my ($d1,$d2) = $line=~/\S+\s+(\S+)\s+(\d\d:\d\d:\d\d)/;
      my $d=$d1.' '.$d2;
      my $t=Time::Piece->strptime($d, $fmt);
      print $line if ($t>$t1 && $t<$t2);
   }
   close ($fh);
}

Run it from command line using syntax : ./p.pl A_22:09:14.