bash - how to properly read an input file which values are in an array

Question

I'm trying to write a script that reads a file (more than one line) which is "#" separated and putting them in an array so I can match some values.

Example input file:

us-east#1-1#abcdefg1234Ad
us-west#1-3#654kjhytgr
us-east#1-4#lkjhg765

What I'm trying to do is read through each line and give me the match based on my input param ($1). I'm stuck as its just evaluating the first line only.

Here's my code: (to execute: ./myscript.sh us-east-1-3)

#!/usr/local/bin/bash
set -x
cluster=$1

KEY=./.keyfile

while IFS=#
declare -a arr=($(< $KEY)); do
if [[ ${arr[0]}-${arr[1]} == $1 ]]; then
    echo "We have a match"
 else
    echo "No match"
    exit 1
fi
done

set +x

You can read the data in keyfile into the array arr with "readarray arr < keyfile" — Raman Sailopal, Aug 30 '17 at 14:03

Akshay Hegde · Accepted Answer · 2017-08-30T14:29:13.133

I'm stuck as its just evaluating the first line only.

Because in your else block you have exit statement, suppose if line does not match, loop will be terminated due to exit 1, so further iteration will not take place.

After reading first line, us-east-1-1 is not equal to us-east-1-3, Boolean false, so in your else block you have exit statement, so termination

+ cluster=us-east-1-3
+ KEY=./file
+ IFS='#'
+ arr=($(< $KEY))
+ declare -a arr
+ [[ us-east-1-1 == us-east-1-3 ]]
+ echo 'No match'                  
No match
+ exit 1

You can modify like below so that you will use less resource, read line by line instead of reading entire file into array

[akshay@localhost tmp]$ cat t.sh
#!/usr/bin/env bash

set -x
cluster="$1"

while IFS=# read -r  field1 field2 restother; do
if [[ "$field1-$field2" == $1 ]]; then
    echo "We have a match"
 else
    echo "No match"
fi
done < "file"

set +x

Output when cluster=us-east-1-3

[akshay@localhost tmp]$ bash t.sh us-east-1-3
+ cluster=us-east-1-3
+ IFS='#'
+ read -r field1 field2 restother
+ [[ us-east-1-1 == us-east-1-3 ]]
+ echo 'No match'
No match
+ IFS='#'
+ read -r field1 field2 restother
+ [[ us-west-1-3 == us-east-1-3 ]]
+ echo 'No match'
No match
+ IFS='#'
+ read -r field1 field2 restother
+ [[ us-east-1-4 == us-east-1-3 ]]
+ echo 'No match'
No match
+ IFS='#'
+ read -r field1 field2 restother
+ set +x

Output when cluster=us-west-1-3

[akshay@localhost tmp]$ bash t.sh us-west-1-3
+ cluster=us-west-1-3
+ IFS='#'
+ read -r field1 field2 restother
+ [[ us-east-1-1 == us-west-1-3 ]]
+ echo 'No match'
No match
+ IFS='#'
+ read -r field1 field2 restother
+ [[ us-west-1-3 == us-west-1-3 ]]
+ echo 'We have a match'
We have a match
+ IFS='#'
+ read -r field1 field2 restother
+ [[ us-east-1-4 == us-west-1-3 ]]
+ echo 'No match'
No match
+ IFS='#'
+ read -r field1 field2 restother
+ set +x

You can use awk for this type of purpose, reasonably faster too

Here is some example

$ cat file 
us-east#1-1#abcdefg1234Ad
us-west#1-3#654kjhytgr
us-east#1-4#lkjhg765

Output (when cluster="us-east-1-3")

$ awk -F'#' -v cluster="us-east-1-3"  '{ print (cluster==$1"-"$2)? "We have a match": "No match"}' file
No match
No match
No match

Output (when cluster="us-west-1-3")

$ awk -F'#' -v cluster="us-west-1-3"  '{ print (cluster==$1"-"$2)? "We have a match": "No match"}' file
No match
We have a match
No match

great input. I appreciate it and thanks for showing another solution using `awk`. I could use that. — noober, Aug 30 '17 at 14:33

score 1 · Answer 2 · answered Aug 30 '17 at 14:27

I agree with Akshay that this is a good problem to solve with awk. However, if you really want to do it with shell alone, it's not difficult:

#!/usr/bin/env bash

cluster="$1"
keyfile=keyfile.txt

ret=0
while IFS='#' read -r one two three; do
  if [[ "${one}-${two}" = "$cluster" ]]; then
    echo "match"
  else
    echo "no match"
    ret=1
  fi
done < "$keyfile"

exit $ret

The key difference is that this is using read to process a stream of input that is handed to the while loop, rather than re-evaluating the $(< $keyfile) expression for each run of the loop (which I would expect would give you the first line for each run).

this fixes my current script and I will take into account using `awk` to solve this. Thank you. — noober, Aug 30 '17 at 14:34

bash - how to properly read an input file which values are in an array

2 Answers2