replace and count bash unable through awk and sed

Question

I need a bash script which should do the following: - read from an input file line by line which having format: [Environment]=[file name]=[property key]=[property value] - it will modify all files by replacing all properties in a $SOURCE directory according to the input file. Now I'm stuck at the point "replace and count".

The code until the "replace and count" phase:

PROPERTIES_LIST=/home/user/test_scripts/envPropList.txt

SOURCE=/home/user/test_scripts/source-directory

PROPERTIES_LOCATION=/WEB-INF/classes/

PROPERTIES_SOURCE=$SOURCE$PROPERTIES_LOCATION

ENV=$1
echo "> Update for ENVIRONMENT: $ENV... Please wait!"

NR=1
while read line; do
    IFS== read env prop <<< "$line"
    if [[ $env == $ENV]]
    then
        IFS== read file key value <<< "$prop"
        if [[ -z $file ]] || [[ -z $key ]] || [[ -z $value ]]
        then
            echo "> [ERROR] [LINE $NR] - WRONG/MISSING PROPERTY: $line"
        else
            //MODIFY $file BY  REPLACE AND COUNT
            //IF $key IS NOT FOUND AT ALL AN ERROR MESSAGE SHOULD BE DISPLAYED. ITERATION SHOULD CONTINUE
            //IF $key IS FOUND MORE THEN ONCE, A WARNING MESSAGE SHOULD BE DISPLAYED. ITERATION SHOULD CONTINUE

            echo "done"
        fi
    fi
    NR=$(( $NR + 1 ))
done <$PROPERTIES_LIST

I have tried the following without success without success because value in properties can be any character (like: &,/,....):

COUNT=$(grep -c "$key" $PROPERTIES_SOURCE$file)
sed -i "s/${OLD}/${NEW}/g" $PROPERTIES_SOURCE$file

Also awk didn't worked as expected:

DEST=/home/user/test_scripts/test.txt
OLD='asd.asd'
NEW='test/test?test.test&test=test'
COUNT=$(grep -c "$OLD" $DEST)
#sed -i "s/#${OLD}#/#${NEW}#p/g" $DEST
#echo "$OLD=$NEW"
echo "nr de rezultate: "$COUNT
awk -v OLD=$OLD -v NEW=$NEW '
    ($0 ~ OLD) {gsub(OLD, NEW); count++}1
    END{print count " substitutions occured."}
' "$DEST"

And for input file:

asd.asd
ewrqfg
qweasd.asdqwreqe
asd asd.asd
egd
test

I have the following output:

test/test?test.testasd.asdtest=test
ewrqfg
qwetest/test?test.testasd.asdtest=testqwreqe
test/test?test.testasd asdtest=test.asd
egd
test

If I remove "&" from $NEW, everything goes fine.

score 0 · Answer 1 · edited May 23 '17 at 12:16

You didn't post your expected output so it's a guess but this is probably what you want:

$ cat tst.sh
dest='file'
old='asd.asd'
new='test/test?test.test&test=test'
count=$(grep -c "$old" "$dest")
#sed -i "s/#${old}#/#${new}#p/g" "$dest"
#echo "$old=$new"
echo "nr de rezultate: $count"
awk -v old="$old" -v new="$new" '
    {
        head = ""
        tail = $0
        lgth = length(old)
        while ( start = index(tail,old) ) {
            head = head substr(tail,1,start-1) new
            tail = substr(tail,start+lgth)
            count++
        }
        print head tail
    }
    END { print count+0, "substitutions occured." }
' "$dest"

$ ./tst.sh file
nr de rezultate: 3
test/test?test.test&test=test
ewrqfg
qwetest/test?test.test&test=testqwreqe
asd test/test?test.test&test=test
egd
test
3 substitutions occured.

Note you cannot use gsub() as then you'll be in escaping hell just like if you used sed, instead you have to use index() and substr() as they operate on literal strings rather than regexps and backreferences in the replacements.

Aside: always quote your shell variables and in shell use of all upper case is reserved for exported variables by convention while in both awk and shell you should avoid all upper case variable names anyway to avoid clashing with builtin variables.

score 0 · Answer 2 · answered May 01 '16 at 22:50

This is my solving through "sed" command which is doing exactly what I need. I am not sure how fast it will run cause there will be a lot of properties to be changed when this script will be running.

Guys please have a review over my solution, constructive feedback is welcomed:

PROPERTIES_SOURCE=$SOURCE$PROPERTIES_LOCATION

MEDIUM=$1
echo "> Update release for ENVIRONMENT: "$MEDIUM"... Please wait!"

if [[ -z $1 ]]
then
    echo "> [ERROR] - Illegal script use. You have to send one of the following parameters representing environment name which will be updated:"
    echo "> [ERROR] - DEV ST UAT PROD"
    echo "> [ERROR] - Example: ./updateEnv.sh PROD"
    exit
fi

nr=1
while read -r line; do
    IFS== read -r env prop <<< "$line"
    if [[ $env == $MEDIUM ]]
    then
        IFS== read -r file key value <<< "$prop"
        if [[ -z $file ]] || [[ -z $key ]] || [[ -z $value ]]
        then
            echo "> [ERROR] [LINE $NR] - wrong or missing property: $line"
        else
            esc_key=$(echo $key | sed -e 's/\./\\./g')
            esc_val=$(echo $value | sed -e 's/\&\|\.\|\\\|\/\|\%/\\&/g')
            occurances=$(grep -c "$key=" "$PROPERTIES_SOURCE$file")
            if [[ $occurances > 1 ]]
            then
                echo "> [WARNING] [LINE $NR] - key found $occurances times: $key"
            elif [[ $occurances < 1 ]]
            then
                echo "> [ERROR] [LINE $NR] - key was not found: $key"
            fi
            sed -i "s/${esc_key}=.*/${esc_key}=${esc_val}/g" $PROPERTIES_SOURCE$file
        fi
    fi
    nr=$(( $nr + 1 ))
done <$PROPERTIES_LIST

Also I succeeded with awk too with some sed for escaping Strings in my variables, which is different from sed:

STRING_OLD='asd.asd'
STRING_NEW='!@#$%^&*()_+-=[]\{}|;"<>~`'
OLD=$(echo $STRING_OLD | sed -e 's/\./\\\\./g')
NEW=$(echo $STRING_NEW | sed -e 's/\&\|\\/\\\\&/g')

And the replacement awk code would be something like this:

awk -v OLD="$OLD" -v NEW="$NEW" '
    BEGIN {print "OLD:"OLD" and NEW:"NEW}
        ($0 ~ OLD) {gsub(/OLD=.*/,OLD=NEW);}1
' "$DEST" > "DEST.tmp" mv "DEST.tmp" "$DEST"

replace and count bash unable through awk and sed

2 Answers2