4

Suppose I have a 10MB text file foo.txt, and it has 100,000 lines. Now, I want to process foo.txt window by window, with a window size 10.

My current script is like this:

for ((i=0;i<$lines;i=i+$step))
do    
    head -$((i+step)) $1 | tail -$step > tmp1
    head -$((i+step)) $2 | tail -$step > tmp2
    setstr=$setstr' '`./accuracy.sh tmp1 tmp2`
done
echo $setstr | awk '{for (i=1;i<=NF;i++) sum+=$i; }END{print sum/NF}'

But it runs slowly. Is there any simple and more efficient way to do this?

JackWM
  • 333

4 Answers4

5

You can do that with split:

Here's an example of how to use it:

split -l 10 input_file output_file_prefix_

The -l option stands for --lines=

And this will split input_file into chunks with 10 lines each, into these files:

output_file_prefix_aa
output_file_prefix_ab
output_file_prefix_ac
...

and so on.

For other ways you can use split, see man split or here

sampson-chen
  • 369
  • 1
  • 3
1

It would be useful to have a bit more context as to your ultimate goal rather than a code snippet. In particular, do you have any control over accuracy.sh?

Anyway, if you want to keep using bash, then you could do

for ((i=0;i<$lines;i+=$step))
do
  let end=i+10
  sed -n $i,${end}p $1 >tmp1
  sed -n $i,${end}p $2 >tmp2
  ...
done
Peter Sundstrom
  • 11
0

Not sure why this got migrated from StackOverflow. While split is a superuser-style answer, the question was about programming. For example, here's an answer that implements what you're looking for in awk.

One of the really handy aspects of awk is the way it handles pipes.

#!/usr/bin/awk -f

BEGIN {
  cmd="/path/to/handler"
}

{
  print | cmd
}

NR % 10 == 0 {
  close(cmd)
}

Your cmd will be re-opened if it's closed ... and gets closed every 10th line, to be reopened the next line of output.

The effect will be to run handler every 10 lines of input. At the end of the file, handler will be run with whatever lines are left, as cmd is automatically closed as awk exits.

Strictly speaking, you don't need to use a variable like cmd to store the command ... but it does make it simpler to adjust the command, since you'd otherwise need to watch VERY carefully for typos in your close().

ghoti
  • 871
0

This solution does not use any temporary files. What is does is store every line in a buffer array that can hold ten lines. Every time the line number is divisible by ten, it prints all the lines in the buffer.

The obvious pitfall is when the input file (# lines) is not divisible by ten. The solution is to make checks in an END{} clause. Something like: 

$ echo {1..33} | tr \  \\n |\
    awk '{lines=NR} END{ if (lines%10!=0) { print "leftover lines"} }'
leftover lines

# STEP1 use modulo to do something every tenth
$ echo {1..200} |tr \  \\n |\
    awk '{a[NR%10]=$0; if (NR%10==0) {print "ten"} }' | cat -n
     1  ten
     2  ten
     3  ten
     4  ten
     5  ten
     6  ten
     7  ten
     8  ten
     9  ten
    10  ten
    11  ten
    12  ten
    13  ten
    14  ten
    15  ten
    16  ten
    17  ten
    18  ten
    19  ten
    20  ten

# STEP 2 do something with every line
$ echo {1..10} | tr \  \\n | awk '{ b+=$0} END {print b}'
55

# putting it together
$ cat every10.awk
{
        a[NR%10]=$0;
        if (NR%10==0) {
                for (i in a) {
                        printf "%s+", a[i]
                        b+=a[i];
                }
                print "0=" b;
                b=0
        }
}
$ echo {1..200} | tr \  \\n | awk -f every10.awk  | column -s= -t
4+5+6+7+8+9+10+1+2+3+0                     55
14+15+16+17+18+19+20+11+12+13+0            155
24+25+26+27+28+29+30+21+22+23+0            255
34+35+36+37+38+39+40+31+32+33+0            355
44+45+46+47+48+49+50+41+42+43+0            455
54+55+56+57+58+59+60+51+52+53+0            555
64+65+66+67+68+69+70+61+62+63+0            655
74+75+76+77+78+79+80+71+72+73+0            755
84+85+86+87+88+89+90+81+82+83+0            855
94+95+96+97+98+99+100+91+92+93+0           955
104+105+106+107+108+109+110+101+102+103+0  1055
114+115+116+117+118+119+120+111+112+113+0  1155
124+125+126+127+128+129+130+121+122+123+0  1255
134+135+136+137+138+139+140+131+132+133+0  1355
144+145+146+147+148+149+150+141+142+143+0  1455
154+155+156+157+158+159+160+151+152+153+0  1555
164+165+166+167+168+169+170+161+162+163+0  1655
174+175+176+177+178+179+180+171+172+173+0  1755
184+185+186+187+188+189+190+181+182+183+0  1855
194+195+196+197+198+199+200+191+192+193+0  1955

The idea here is to use awk print blocks of ten lines and process that, or process with awk directly if the operation is simple arithmetic or string operations.

Ярослав Рахматуллин
  • 11,633