Sharing variables across modules

Question

I have a program- main.py that generates values for a few parameters depending on the user provided inputs. I need to share these variables with other modules- aux.py. To do that I did the following (simplified code below) -

This the main.py file:

# these three variables are dynamically calculated depending on the user provided input
a = 12
b = 13
c = 14

def declare_global():
    global a, b, c

declare_global()
import aux
aux.print_all()

This is the aux.py file

def print_all():
    a = globals()['a']
    b = globals()['b']
    c = globals()['c']
    print(a)
    print(b)
    print(b)

Running the main.py file results in the following error

Traceback (most recent call last): File "/Users/new/Library/Preferences/PyCharmCE2018.3/scratches/global_experiment/main.py", line 13, in aux.print_all() File "/Users/new/Library/Preferences/PyCharmCE2018.3/scratches/global_experiment/aux.py", line 2, in print_all a = globals()['a'] KeyError: 'a'

There is a very similar post addressing the same issue here but the posts there suggest adding all global variables to a new file and then importing the created file. But I cannot do that as I'll have no prior information as to what values these variables will take.

So, how do I share variables across modules?

EDIT

I lost some complexity of the actual program in order to make a minimalistic example. The reason I cannot pass the variables a,b,c to the print_all function is because in the actual program, the variables a,b,c are not serializable. They are spark dataframe objects. And one of the modules I use there (which here I am representing by print_all) serializes all its inputs I end up with an error (spark dataframes aren't serializable). Using a few workarounds I have arrived here where I use global variables and no inputs to the functions.

Why don't you pass the values to the other function? Or have aux its own variables that you modify in `main`? (e.g. `aux.some_variable = a`) — h4z3, Aug 07 '19 at 11:06
why don't you create a python file holds the variables with None as default value? — Ramy M. Mousa, Aug 07 '19 at 11:24
I advise you to write a function that serializes the Spark inputs instead. — IMCoins, Aug 07 '19 at 11:29
@IMCoins,Thanks for the suggestion but the spark inputs are not serializable. They are generally saved as parquet or csv files and secondly, even if they were serializable or I tweaked the function that was serializing them, those spark inputs can be anywhere between a few GBs to a hundred GB or more. The disk space and time required are prohibitively expensive. — Clock Slave, Aug 08 '19 at 04:20
You don't have to print it entirely, that's not my point. The function "print_all" does show something right ? Would it be on stdout or in a file. Well, make a special case to print your Spark Data Frame. You could create a class that would act as a really thin wrapper around Spark DataFrames, and using the __repr__ dunder method, you could have a way to print it. Do you know how __repr__ works ? — IMCoins, Aug 08 '19 at 07:16

score 0 · Answer 1 · answered Aug 07 '19 at 12:04

0

a = 12
b = 13
c = 14

import aux
aux.print_all(a, b, c)

Then your aux file will become:

def print_all(a, b, c):
    a = globals()['a']
    b = globals()['b']
    c = globals()['c']
    print(a)
    print(b)
    print(b)

answered Aug 07 '19 at 12:04

Izalion

708
4
11

Hi, thanks for the answer. But I cannot use the method you propose. Can you take a look at the edit section once? – Clock Slave Aug 08 '19 at 04:21

Sharing variables across modules

1 Answers1