Is there a way to optimize this code and make it faster or anyother better solution?

Question

I am looking to collect yang models from my project .jar files.Though i came with an approach but it takes time and my colleagues are not happy.

#!/bin/sh

set -e

# FIXME: make this tuneable
OUTPUT="yang models"
INPUT="."

JARS=`find $INPUT/system/org/linters -type f -name '*.jar' | sort -u`

# FIXME: also wipe output?
[ -d "$OUTPUT" ] || mkdir "$OUTPUT"
for jar in $JARS; do
    artifact=`basename $jar | sed 's/.jar$//'`
    echo "Extracting modules from $artifact"
    # FIXME: better control over unzip errors
    unzip -q "$jar" 'META-INF/yang/*' -d "$artifact" \
        2>/dev/null || true

    dir="$artifact/META-INF/yang"
    if [ -d "$dir" ]; then
        for file in `find $dir -type f -name '*.yang'`; do
            module=`basename "$file"`
            echo -e "\t$module"
            # FIXME: better duplicate detection
            mv -n "$file" "$OUTPUT"
        done
    fi

    rm -rf "$artifact"
done

Use globstars instead of find and `$()` instead of ``. You should only extract `*.yang` files from the zip instead of the whole `yang/` directory. Extract directly to the final output directory to eliminate the inner loop (which also could be replaced with a single `mv` command). You might also want to take a look at https://www.shellcheck.net/. — Socowi, Jun 06 '19 at 10:13
Performance issues not relating to comms or database are often the number of child processes created inside a loop. `find`, `basename`, `sed`, are all external programs, try to use shell commands instead. — cdarke, Jun 06 '19 at 10:20
For example, `basename` can be replaced by `jar=${jar##*/}` and the `sed` can be replaced by `artifact=${jar%.*}` — cdarke, Jun 06 '19 at 10:21
@Socowi Can you provide the code changes at *.yang code line. — Engineer Graduate, Jun 06 '19 at 11:40
With `shopt -s globstar` that would be `for file in "$dir"/**/*.yang; do` — Socowi, Jun 06 '19 at 12:02
What errors do you get exactly? In my tests I had no problems. If it is possible that there are no yang files add `shopt -s nullglob`. And what does this have to do with the `if`? The line `for ...` is inside the `if` and therefore shouldn't affect the `if`. — Socowi, Jun 06 '19 at 12:07
@EngineerGraduate again you're just telling us "there are errors" in contexts where we can't imagine what the errors might be. Please [edit] your question to show us the code you're executing (or running shellcheck against) that's producing the errors and what the error messages are if you'd like help debugging them. — Ed Morton, Jun 06 '19 at 12:14

Ed Morton · Answer 1 · 2019-06-06T11:46:18.957

If the .jar files don't all change between invocations of your script then you could make the script significantly faster by caching the .jar files and only operating on the ones that changed, e.g.:

#!/bin/env bash
set -e

# FIXME: make this tuneable
output='yang models'
input='.'
cache='/some/where'
mkdir -p "$cache" || exit 1

readarray -d '' jars < <(find "$input/system/org/linters" -type f -name '*.jar' -print0 | sort -zu)

# FIXME: also wipe output?
mkdir -p "$output" || exit 1
for jarpath in "${jars[@]}"; do

    diff -q "$jarpath" "$cache" || continue
    cp "$jarpath" "$cache"

    jarfile="${jarpath##*/}"
    artifact="${jarfile%.*}"
    printf 'Extracting modules from %s\n' "$artifact"
    # FIXME: better control over unzip errors
    unzip -q "$jarpath" 'META-INF/yang/*' -d "$artifact" 2>/dev/null

    dir="$artifact/META-INF/yang"
    if [ -d "$dir" ]; then
        readarray -d '' yangs < <(find "$dir" -type f -name '*.yang' -print0)
        for yangpath in "${yangs[@]}"; do
            yangfile="${yangpath##*/}"
            printf '\t%s\n' "$yangfile"
            # FIXME: better duplicate detection
            mv -n "$yangpath" "$output"
        done
    fi

    rm -rf "$artifact"
done

See Correct Bash and shell script variable capitalization, http://mywiki.wooledge.org/BashFAQ/082, https://mywiki.wooledge.org/Quotes, How can I store the "find" command results as an array in Bash for some of the other changes I made above.

I assume you have some reason for looping on the .yang files and not moving them if a file by the same name already exists rather than unzipping the .jar file into the final output directory.

Then you've probably copy/pasted wrong so check that but if not then I guess fix whatever shellcheck tells you to fix? I don't have your environment so I can't test the code myself and when I run shellcheck on it copy/pasted into https://www.shellcheck.net/ I get `$ shellcheck myscript No issues detected!` and with only having `shell check is yelling at me` from you to go on I can't provide any feedback on what the issues might be in the code you're running shellcheck on. — Ed Morton, Jun 06 '19 at 11:49

Is there a way to optimize this code and make it faster or anyother better solution?

1 Answers1