Expand bash array with globbing and extra words

Question

In a CI/CD job, I have a shell variable X defined. It contains one or more words, each of which might have glob operators:

X="foo bar* 'a b c'"

Suppose bar* matches 3 files, bard, bare, and barf. And the file a b c exists and has 2 spaces in it.

I want to create a Bash array Y with these contents:

Y=(--arg foo --arg bard --arg bare --arg barf --arg 'a b c')

In other words: do glob/quote expansion, then map each word w to --arg $w.

A clean solution would allow spaces in the expanded names (to be honest, I'm never going to have that situation in this CI/CD code - but I or someone else might easily copy/paste this technique somewhere else where it does matter).

It would also be awesome if I could do glob/quote expansion without invoking all other possible shell expansions (e.g. process substitution & subshells) - I'm having trouble thinking of how to do that, though.

The only solution I've come up with so far is to use unprotected expansion:

Y=()
for a in $X; do
    Y+=(--arg "$a")
done

Is that the best I can do? Is it safe? It works well on the foo and bar* cases, but not well on the a b c case.

I think this is difficult. I don't think there's a way to do wildcard expansion that doesn't also do word splitting. — Barmar, May 11 '22 at 21:46
Just realized my example is wrong - it assumes `X` is an array, which it's not. I'll fix that. — Ken Williams, May 11 '22 at 21:48
The only other expansion that's done is word splitting. But it doesn't split within the words of a filename that came from glob expansion. So your loop should work. — Barmar, May 11 '22 at 21:53
The only problem is if you want to match a file named `foo bar1`. You can't have it both ways, either the space in `X` separates wildcards, or it should be ignored and treated as part of the wildcard. — Barmar, May 11 '22 at 21:54
You could change `IFS` to just a single character (e.g. `IFS=$'\n'` for newline), and then use that as a delimiter in the pattern string -- that'd work as long as you never need to include that character in a pattern. (But be sure to set `IFS` back to normal afterward, to avoid weird effects later in the script.) — Gordon Davisson, May 11 '22 at 21:58
Thanks, I have this working for all the *reasonable* cases now, but I'm adding one more case with spaces in filenames to the example. — Ken Williams, May 11 '22 at 22:04

Charles Duffy · Accepted Answer · 2022-05-11T23:21:59.670

To expand globs while honoring quotes (for grouping but not glob-suppression), but not expand variables or handle process substitutions or other dangerous syntax...

X="foo bar* 'a b c'"

IFS=
args=( )
while read -r -d '' word; do
  for item in $word; do
    args+=( --arg "$item" )
  done
done < <(xargs printf '%s\0' <<<"$X")

See this running in an online sandbox (where bard, bare and barf exist) at https://replit.com/@CharlesDuffy2/UnlawfulDecentApplicationserver#main.sh

The use of xargs printf '%s\0' has xargs do the work of word-splitting in an almost POSIX-compliant way (if you want something completely POSIX-compliant, you need to use the Python shlex module instead -- other Q&A on the site demonstrates how), and the unquoted expansion with an empty IFS performs globbing only.

Fravadona · Answer 2 · 2022-05-12T06:48:05.397

1

You could use declare which has eval capabilities:

#!/bin/bash

x="foo bar* 'a b c'"
declare -a x="($x)"

y=()
for a in "${x[@]}"; do y+=(--arg "$a"); done

Now it comes down to what you would like, for example, x='$HOME \ \$USER $( ls )' to really represent...

edited May 12 '22 at 06:48

answered May 11 '22 at 22:37

Fravadona

13,917
1
23
35

score 1 · Answer 3 · answered May 12 '22 at 02:35

Take @Fravadona's answer over this one.

You have to trust that nothing malicious is going to be in that string.

This is a case where you need eval: given

$ touch bar{d,e,f}
$ X="foo bar* 'a b c'"

then this won't get you what you want:

$ y=($X)
$ declare -p y
declare -a y=([0]="foo" [1]="bard" [2]="bare" [3]="barf" [4]="'a" [5]="b" [6]="c'")

however

$ eval "y=($X)"
$ declare -p y
declare -a y=([0]="foo" [1]="bard" [2]="bare" [3]="barf" [4]="a b c")

then

$ Y=(); for elem in "${y[@]}"; do Y+=(--arg "$elem"); done
$ declare -p Y
declare -a Y=([0]="--arg" [1]="foo" [2]="--arg" [3]="bard" [4]="--arg" [5]="bare" [6]="--arg" [7]="barf" [8]="--arg" [9]="a b c")

If you have a bad actor, and you get X='nasty $(bad command here)', then you cannot use eval.

Expand bash array with globbing and extra words

3 Answers3