Pass keyword argument to another function where the keyword has a default value

Question

The intent of the following code is for keyword arguments passed to main to be passed on to the other called functions. However, if no value is given to main for the key1 argument, then the default value of key1 in f2 is overridden by the value of None passed by main.

def main(x, key1=None, key2=None):
    f1(x, key1=key1)
    f2(x, key1=key1, key2=key2)

def f1(x, key1=None): # note key2 is absent
    print(x, key1)

def f2(x, key1=42, key2=None):
    print(x, key1, key2)

main(1)

Output:

(1, None)
(1, None, None)

Is there a simple way to change just the definition of main (possibly including the arguments) so that the output of the called functions would be as below?

(1, None)
(1, 42, None)

Obviously, there are ridiculous ways to alter just main to create that output, but it would be silly to insert logic into main to pass the default value for a missing argument for a specific function, or to alter the keywords that are passed by main depending on which of them is None. So I'm not looking for a solution like that.

The usual way to get around this problem would be to use **kwargs in all of the functions. Alternatively, you could change the definition of f2:

def f2(x, key1=None, key2=None):
    if key1 is None:
        key1 = 42
    print(x, key1, key2)

But I'm wondering if there's another option.

Answer 1

There isn't, because at the time you call f2, Python doesn't know whether what you pass as key1 was passed in or not. For instance, you could have done this:

def main(x, key1=None, key2=None):
    if some.otherCondition():
      key1 = "Blah"
    f2(x, key1=key1, key2=key2)

To do what you describe, Python would have to be checking every single use of key1 to see if it happened to equal the default value. This could easily become very confusing if you were doing anything with key1 except passing it straight along to another function.

Moreover, there's no particular reason why it should be assumed that you want to "ignore" the value of key1 just because it happens to be the default in main. It's equally possible that a function similar to your main might be written specifically to override f2's default with its own default, something like:

def read_csv(filename, separator=","):
    read_delimited_file(separator=separator)

def read_delimited_file(filename, separator="\t"):
    # .,..

Here the whole point of read_csv would be to override the default delimiter in the function it calls, while still allowing the user to over-override it when calling read_csv.

In other words, there is no escaping the fact that deciding which arguments to pass on involves decisions, and you have to implement those decisions explicitly in your code. You seem to be aware of several possible solutions already, and you'll have to use one of them. I think one of the clearest would be just creating an argument dictionary:

def main(x, key1=None, key2=None):
    kwargs = {}
    if key1 is not None:
        args['key1'] = key1
    if key2 is not None:
        args['key2'] = key2
    f2(x, **kwargs)

(If you have lots of arguments with similar logic you could start getting clever with locals() to automate this somewhat, but for just a couple arguments doing it explicitly is more clear.)

It might seem "silly" to have to do this, but it's not really silly, because it's not obvious that this is what the default behavior should be.

Answer 2

Problem in the original code

The difficulty arises because main collects keyword arguments for two functions that it calls, but not all of the keyword arguments are accepted by one of the functions. The main function cannot pass a single dict to both functions using **kwargs without raising an error in f1 if key2 is specified when calling main.

To work around this, the original definition of main manually handles the keyword arguments in order to pass only the necessary arguments to the other functions. This in turn causes the problem of "overriding" the default values provided by the called functions.

Possible solution

One option is to use introspection to define a function that will subset the kwargs in main to an appropriate dict for each function that main calls. This avoids writing logic specific to each function to subset the dict.

This change allows the functions called by main to explicitly list the accepted keyword arguments in their definition, and allows main to collect all of the keyword arguments without setting default values before handing them to the other functions.

def kwargs_for_func(func, kwargs, omit_none=False):
    arg_count = func.func_code.co_argcount
    func_kwargs = func.func_code.co_varnames[:arg_count]
    new_kwargs = {}
    for k in kwargs:
        if k not in func_kwargs:
            continue
        val = kwargs[k]
        if not (omit_none and val is None):
            new_kwargs[k] = val
    return new_kwargs

def apply_with_valid_kwargs(func, *args, **kwargs):
    return func(*args, **kwargs_for_func(func, kwargs))

def main(*args, **kwargs):
    apply_with_valid_kwargs(f1, *args, **kwargs)
    apply_with_valid_kwargs(f2, *args, **kwargs)  

def f1(x, key1=None):
    pass # different def to emphasize changed output from f2

def f2(x, key1=42, key2=None):
    print(x, key1, key2)

main(1)            # f2 prints (1, 42, None)
main(1, key1=None) # f2 prints (1, None, None)

Output:

(1, 42, None)
(1, None, None)

Since main is collecting a kwarg dict, this sidesteps the issue of main providing default values for keywords that you really want to be set by the functions called in main. If you do want a None keyword value passed to main to mean "use the default value when you pass this keyword to other functions", the omit_none option can be set in kwargs_for_func.

def main(x, **kwargs):
    f1(x, **kwargs_for_func(f1, kwargs, True))
    f2(x, **kwargs_for_func(f2, kwargs, True))    

main(1, key1=None)

Output:

(1, 42, None)

Notes

This approach was inspired by this other question on SO. @BrenBarn's questions and comments helped identify what the problem was, so thanks! I'll leave this question open for a while to see what different approaches others might have.