How to pass associative array to Gnu Parallel

Question

Like in a title, I've declared bunch of variables and a function. And when I pass them through a function I've got what I expected. But running same code through parallel didn't... How to fix it?

#!/bin/bash                                                

declare -xA MAP # export associative array                 
declare -x str="ing" # export variable                     
MAP[bar]="baz"                                             
MAP[bar2]="baz2"                                           


foo() {                                                    
  echo "$@"                                                
  echo "variable: ${str}"                                  
  echo "map: ${MAP[@]}"                                    
}                                                          

export -f foo                                              

foo "call function directly:"

call function directly:
variable: ing
map: baz2 baz

parallel foo ::: "call function through parallel" ::: 1 2 3

call function through parallel 1
variable: ing
map:
call function through parallel 2
variable: ing
map:
call function through parallel 3
variable: ing
map:

edit after comments

It looks that accepted answer for this question is: There isn't really a good way to encode an array variable into the environment.

Which is a bit sad... ;)

You cannot export (associative) arrays. However, you should be able to pass the array's values to `parallel` which passes them on to the script. If you specify what exactly you are trying to do with the array we might be able to help you with that. — Socowi, Mar 23 '19 at 17:44
I wish I could add this as an answer: The answer is `env_parallel`. You cannot export associative arrays _without_ _cheating_ - which is what `env_parallel` does. I cannot add this as an answer on the other question, because it only applies to GNU Parallel - not in general. — Ole Tange, Mar 24 '19 at 19:47

LookAheadAtYourTypes · Accepted Answer · 2019-03-24T08:53:07.023

From comments under the question it looks that accepted answer is: There isn't really a good way to encode an array variable into the environment.

And that's all... ;)

It looks that there is another one: Accessing Associative Arrays in GNU Parallel with quite sophisticated approach. But...

after some tinkering (as Socovi suggested that specific problem can have solutions) it looks that for my case good enough workaround is to "serialize" array in temporary file and deserialize it in function.

#!/bin/bash                                                                   

declare -A MAP # export associative array                                    
declare -x str="ing" # export variable                                        
MAP[bar]="baz"                                                                
MAP[bar2]="baz2"                                                              
declare -x serialized_array=$(mktemp)   

# declare -p can be used to dump the definition 
# of a variable as shell code ready to be interpreted                                      
declare -p MAP > "${serialized_array}" 


# perform cleanup after finishing script                                      
cleanup() {                                                                   
  rm "${serialized_array}"                                                    
}                                                                             
trap cleanup EXIT                                                             


foo() {                                                                       
  echo "$@"                                                                   
  echo "variable: ${str}"                                                     
  source "${serialized_array}" # deserialize an array                         
  echo "map: ${MAP[@]}"                                                       
}                                                                             

export -f foo                                                                 

foo "call function directly:"

call function directly:
variable: ing
map: baz2 baz

parallel foo ::: "call function through parallel" ::: 1 2 3

call function through parallel 1
variable: ing
map: baz2 baz
call function through parallel 2
variable: ing
map: baz2 baz
call function through parallel 3
variable: ing
map: baz2 baz

How to pass associative array to Gnu Parallel

1 Answers1