Iterate over a text file line by line within a for loop in a shell script

Question

I have a txt file that looks like this called locations.txt:

location-1a
location-2c
location-3d
location-4
location-5a
...

I wish to iterate over the locations.txt file within my for loop, each location should be used for 5 nodes, then the next location should be used. so for example, I want the following commands to run,

server create --location location-1a node-1
server create --location location-1a node-2
server create --location location-1a node-3
server create --location location-1a node-4
server create --location location-1a node-5
server create --location location-2c node-6

.....

So far, I only have the following script called script.sh with takes an argument from the cli. I can't change the location like I want to.

mapfile -t locationArr < ~/Documents/files/locations.txt
serverCounter=1
locationCounter=1
counter=1

counter=1
for (( i = 1; i <= $1; i++ ))
do
 server create --location $locationArr[$locationCounter] node-$i
 counter=$((counter+1))
 if [ $counter == 5 ]
 then
  counter=1
  locationCounter=$((locationCounter+1))
done

I want something like

if counter == 5:
 counter=0
 nextline

So for example if I run script.sh 11, I want the following result,

server create --location location-1a node-1
server create --location location-1a node-2
server create --location location-1a node-3
server create --location location-1a node-4
server create --location location-1a node-5
server create --location location-2c node-6
server create --location location-2c node-7
server create --location location-2c node-8
server create --location location-2c node-9
server create --location location-2c node-10
server create --location location-3d node-11

How do I iterate over the location.txt file within my for loop? Should I convert the text file into some sort of list variable?

Does this answer your question? [Looping through the content of a file in Bash](https://stackoverflow.com/questions/1521462/looping-through-the-content-of-a-file-in-bash) — Léa Gris, Sep 03 '20 at 19:51
Your `for` loop isn't reading the file at all. See also https://mywiki.wooledge.org/DontReadLinesWithFor — tripleee, Sep 03 '20 at 19:51
I know it isn't, I want it to read it on top of my for loop, I want the for loop to be the main loop while having a separate counter for the file — anarchy, Sep 03 '20 at 19:52

hesham_EE · Accepted Answer · 2020-09-03T20:04:54.173

2

This is my attempt:

counter=1
while IFS= read -r line
do
    for (( i=1; i <= 5 && $counter <= $1; i++ ))
    do
     server create --location $line node-$counter
     counter=$((counter+1))
    done 
done < ~/Documents/files/locations.txt

The output with 11:

server create --location location-1a node-1
server create --location location-1a node-2
server create --location location-1a node-3
server create --location location-1a node-4
server create --location location-1a node-5
server create --location location-2c node-6
server create --location location-2c node-7
server create --location location-2c node-8
server create --location location-2c node-9
server create --location location-2c node-10
server create --location location-3d node-11

edited Sep 03 '20 at 20:04

answered Sep 03 '20 at 19:55

hesham_EE

1,125
13
24

1

I guess that should be `node-"$counter"` but the question is vaguely confusing on this point. – tripleee Sep 03 '20 at 19:58
Yeah, I realized this now. My output doesn't match to the OP question. – hesham_EE Sep 03 '20 at 19:59
I think I got now what he wants. – hesham_EE Sep 03 '20 at 20:06
hey how does the `i <= 5 && $counter <= $1` part work? how does it go to the next line within the while loop? – anarchy Sep 03 '20 at 20:12
For every line in the file. It'll loop on all valid i, from 1 to 5. In order to do this iteration, it'll have also to satisfy the condition that `counter` is still less than the supplied command line value. I see one issue now, it'll keep reading the input file even when the `count` is beyond `$1` but won't spit any output. If this is an issue, you can use `break` after the inner for loop once this condition is met to stop reading the input file. Let me know if this isn't clear. – hesham_EE Sep 03 '20 at 21:10

markp-fuso · Answer 2 · 2020-09-04T06:57:47.720

General outline:

outer loop for reading locations from locations.txt
inner loop for generating a max of 5 nodes for a given location
each time a node is generated increment a count
if the counter > input 'max' then break out of loops

One implementation of the above:

max=${1}
curr=1

while read -r location
do
    for i in {1..5}
    do
        echo "server create --location ${location} node-${curr}"
        curr=$((curr+1))
        [ ${curr} -gt ${max} ] && break 2
    done
done < locations.txt

Test run with max=${1}=11:

server create --location location-1a node-1
server create --location location-1a node-2
server create --location location-1a node-3
server create --location location-1a node-4
server create --location location-1a node-5
server create --location location-2c node-6
server create --location location-2c node-7
server create --location location-2c node-8
server create --location location-2c node-9
server create --location location-2c node-10
server create --location location-3d node-11

OP can add additional checks for max being an integer greater than 0.

Nice! I like `break 2` .. i completely forgot about this construct. — hesham_EE, Sep 04 '20 at 03:07

Léa Gris · Answer 3 · 2020-09-04T22:42:52.173

1

Done with single loop, no Bashism POSIX syntax:

#!/usr/bin/env sh

i=0
while [ $i -lt "$1" ] && {
  [ $((i % 5)) -ne 0 ] || IFS= read -r line || [ "$line" ]
}; do
  server create --location "$line" "node-$((i = i + 1))"
done <locations.txt

edited Sep 04 '20 at 22:42

answered Sep 03 '20 at 20:35

Léa Gris

17,497
4
32
41

Nice, but you may need to limit the output to the command line value `$1` not the end of locations.txt – hesham_EE Sep 04 '20 at 03:05
Your solution continues generating output for all lines in locations.txt. From the OP sample output, the output needs to stop when it reaches the given limit, `$1`. – hesham_EE Sep 04 '20 at 16:57
Great ... i have a Q, what does `|| [ "$line" ]` do exactly? It seems to work fine without it as well. Is it to avoid empty lines in locations.txt ? – hesham_EE Sep 04 '20 at 17:19
1

@hesham_EE It returns `true` if the last line does not end with a `newline` character. Otherwise the last `read` would cause the condition to fail and the last line would not be processed. – Léa Gris Sep 04 '20 at 18:17

Iterate over a text file line by line within a for loop in a shell script

3 Answers3