14

The way to deal with mutable default arguments in Python is to set them to None.

For example:

def foo(bar=None):
    bar = [] if bar is None else bar
    return sorted(bar)

If I type in the function definition, then the only type for bar says that bar is Optional when, clearly, it is not Optional by the time I expect to run that sorted function on it:

def foo(bar: Optional[List[int]]=None):
    bar = [] if bar is None else bar
    return sorted(bar) # bar cannot be `None` here

So then should I cast?

def foo(bar: Optional[List[int]]=None):
    bar = [] if bar is None else bar
    bar = cast(List[int], bar) # make it explicit that `bar` cannot be `None`
    return sorted(bar)

Should I just hope that whoever reads through the function sees the standard pattern of dealing with default mutable arguments and understands that for the rest of the function, the argument should not be Optional?

What's the best way to handle this?

EDIT: To clarify, the user of this function should be able to call foo as foo() and foo(None) and foo(bar=None). (I don't think it makes sense to have it any other way.)

EDIT #2: Mypy will run with no errors if you never type bar as Optional and instead only type it as List[int], despite the default value being None. However, this is highly not recommended because this behavior may change in the future, and it also implicitly types the parameter as Optional. (See this for details.)

Pro Q
  • 4,391
  • 4
  • 43
  • 92

5 Answers5

4

None is not the only sentinel available. You can choose your own list value to use as a sentinel, replacing it (rather than None) with a new empty list at run time.

_sentinel = []

def foo(bar: List[int]=_sentinel):
    bar = [] if bar is _sentinel else bar
    return sorted(bar)

As long as no one calls foo using _sentinel as an explicit argument, bar will always get a fresh empty list. In a call like foo([]), bar is _sentinel will be false: the two empty lists are not the same object, as the mutability of lists means that you cannot have a single empty list that always gets referenced by [].

chepner
  • 497,756
  • 71
  • 530
  • 681
  • So for each mutable type I do this for, I would need to add a new sentinel corresponding to that type? – Pro Q Jun 04 '21 at 17:52
  • 1
    Yes. The use of a single `None` type to represent the "absence" of a value of any arbitrary type is antithetical to static typing. For example, even in Haskell, which has a `Maybe` type constructor for representing optional values, the value `Nothing :: Maybe a` is polymorphic: it's not a single value, but rather a whole family of monomorphic values like `Nothing :: Maybe Int`, `Nothing :: Maybe String`, etc. – chepner Jun 04 '21 at 17:59
  • You should make `_sentinel` an `object()` instead of an empty list. Having a mutable default argument is a common pitfall: https://docs.python-guide.org/writing/gotchas/#mutable-default-arguments – Zecong Hu Jun 21 '21 at 21:23
  • A mutable default is only a problem if you actually *mutate* the default. We aren't doing that here; the only thing we do with `_sentinel` is compare it to `bar` and pass it to `sorted`. `sorted(_sentinel)` will always return a *new* empty list without modifying `_sentinel` at all. – chepner Jun 21 '21 at 21:28
  • @chepner But it's still a bad practice. You can end up accidentally mutating it without meaning to do so, and it'll introduce obscure bugs. For simple functions like this it's probably fine, but it'll be harder to make sure you're not mutating the object in more complex code. – Zecong Hu Jun 24 '21 at 14:32
  • This seems like a bad idea. It'll be confusing for anyone reading the code, and in generated documentation, the default will show up as `[]`, which looks like a bug to anyone who knows about Python's mutable default problem. Just sticking with `=None` seems like a much better option. – user2357112 Oct 28 '21 at 05:11
  • Plus, for anyone who *doesn't* know about Python's mutable default problem, seeing `[]` as a default in generated docs makes it look like mutable default values are totally fine. – user2357112 Oct 28 '21 at 05:14
1

Why not just cut out the cast when you shadow bar:

def foo(bar: Optional[List[int]]=None):
    bar : List[int] = [] if bar is None else bar
    return sorted(bar)
Kraigolas
  • 5,121
  • 3
  • 12
  • 37
  • 2
    mypy [reports an error](https://mypy-play.net/?mypy=latest&python=3.9&gist=b86a0d4030c952533e29822a34974a36) for this kind of variable type redefinition. – user2357112 Jun 04 '21 at 02:10
  • @user2357112supportsMonica That's interesting. It works fine in my python editor. – Kraigolas Jun 04 '21 at 02:14
  • 2
    @Kraigolas mypy is a tool that is used to verify type hints and their usage to help enforce typing in python. The editors may be ok with how it looks and python will run just fine but mypy is more strict on type hints and says there is errors. – TeddyBearSuicide Jun 04 '21 at 02:19
  • 1
    @Mythalorian Thanks! To me, as types aren't actually checked in python, that restriction that you *must* cast instead of shadow seems arbitrary and clunky (even in Rust I can shadow a variable as I've done in this answer), so I'll leave the answer up in case OP is fine with a disagreement with mypy, but thanks for pointing that out to me! – Kraigolas Jun 04 '21 at 02:27
  • 1
    I personally am looking for something that doesn't throw a Mypy error, but I think that this answer is still helpful for other people who are okay with Mypy errors (and maybe Mypy will change its ways eventually) – Pro Q Jun 04 '21 at 02:30
  • @ProQ Are you wanting people to call `foo()` with no arguments? Or do you require they pass something into `foo`? – TeddyBearSuicide Jun 04 '21 at 02:33
  • I want both to be available. – Pro Q Jun 04 '21 at 02:34
  • Ah gotcha, yea sorry once you give a default value typing will only let it be Optional since if you don't pass anything the function runs just fine and returns. Is there a reason for the empty call? What use is the function if it just returns a sorted empty list? – TeddyBearSuicide Jun 04 '21 at 02:38
  • It's just a prototype example. I agree that the function would be useless in that case, but I'm not asking about this particular case, I'm wondering about the general case. This particular function was just an easy-to-understand example. – Pro Q Apr 16 '23 at 04:47
1

I'm not sure what's the issue here, since using Optional[List[int]] as the type is perfectly fine in mypy: https://mypy-play.net/?mypy=latest&python=3.9&gist=2ee728ee903cbd0adea144ce66efe3ab

In your case, when mypy sees bar = [] if bar is None else bar, it is smart enough to realize that bar cannot be None beyond this point, and thus narrow the type to List[int]. Read more about type narrowing in mypy here: https://mypy.readthedocs.io/en/stable/kinds_of_types.html?highlight=narrow#union-types

Here's some other examples of type narrowing:

from typing import *

a: Optional[int]
assert a is not None
reveal_type(a)  # builtins.int

b: Union[int, float, str]
if isinstance(b, int):
    reveal_type(b)  # builtins.int
else:
    reveal_type(b)  # Union[builtins.float, builtins.str]
Zecong Hu
  • 2,584
  • 18
  • 33
  • I'm aware that MyPy has this functionality, but I'm worried about the clarity for human programmers. I would like it to be obvious that `None` is not really an acceptable value for that parameter, but instead is a default and will be replaced with something that is not `None` as soon as possible. If I were reading the function and jumped straight from the function definition to the return statement, I would be very confused, because the function definition says the parameter can be `None`, but the function in the return statement cannot take a `None` parameter. – Pro Q Jun 23 '21 at 00:28
  • 2
    @ProQ I understand your concern, but I would argue that `Optional[List[int]]` is still the better choice here. IMO function signatures are for users, users read the signature to understand what arguments they need to provide, and what are their types. Here it's possible to provide `None` as an argument, so the type should include `Optional`. It's a different story for those who are reading the function implementation, since they'll need to read through the entire function body before they can understand what the function does, and they'll see the part where you substitute `None` with default. – Zecong Hu Jun 24 '21 at 14:26
  • If you're still worried about clarity, I would say that a comment or a docstring might work better. Also, I believe mypy by default [allows omitting](https://mypy.readthedocs.io/en/stable/command_line.html#cmdoption-mypy-no-implicit-optional) the `Optional` in the type hints, if the default is None. – Zecong Hu Jun 24 '21 at 14:30
0

If you add a default value to a parameter in a function than yes it is optional. Also in your code you're allowing the caller to supply nothing to the function and it still works cause it just creates an empty list.

Also in python even with typing it doesn't enforce the type. That's why its called "type hinting" in the documentation.

So if you want to allow callers to call the function with no arguments and it just sort an empty list then your code here is proper.

def foo(bar=None):
    bar = [] if bar is None else bar
    return sorted(bar)

But if you never want a caller to NOT supply something even None, then you should change you function signature.

def foo(bar: List[int]):
    bar = [] if bar is None else bar
    return sorted(bar)

Now type hinting detects that bar is a List[int]. You have to pass something to foo. So you can do foo(None) which is why you need the None check but now foo() is invalid and throws an error.

If you do want them to not pass in anything then just do this and type hinting still works.

def foo(bar: List[int]=None):
   bar = [] if bar is None else bar
   return sorted(bar)
TeddyBearSuicide
  • 1,377
  • 1
  • 6
  • 11
  • 1
    This is absolutely a valid point that you make, but `bar: List[int]` does not make it clear that `None` will be handled in any way, and it looks like OP wants the type hints to be explicit for someone reading the code to be able to understand. – Kraigolas Jun 04 '21 at 02:15
  • 2
    I see what you're saying but the OP says that the list isnt optional which sounds like they are expecting something to be passed in and not an empty function call like `foo()`. If that's the case than they should remove the default value and let python throw an exception so the person calling it knows it's a required parameter. – TeddyBearSuicide Jun 04 '21 at 02:21
  • `Optional` is assumed only because `None`, rather than an actually list, is used as the default. The `Optional[a]` type hint does not mean an argument at runtime is optional; it's just short for `Union[a,None]`. – chepner Jun 04 '21 at 17:41
0

How about using Sequence[int]? It will be type-checked that the argument (including its default) is almost readonly.

from collections.abc import Sequence

def foo(bar: Sequence[int] = []) -> list[int]:
    return sorted(bar)

Still bar can be mutated as below, but the risk would not be high. (_sentinel: list[int] = [] can be mutated, too.)

def foo(bar: Sequence[int] = []) -> list[int]:
    if isinstance(bar, list):
        # reveal_type(bar)  # => Revealed type is "builtins.list[Any]"
        bar.append(0)
    return sorted(bar)
tos
  • 281
  • 1
  • 3