How to pass several list of arguments to @click.option

Question

I want to call a python script through the command line with this kind of parameter (list could be any size, eg with 3):

python test.py --option1 ["o11", "o12", "o13"] --option2 ["o21", "o22", "o23"]

using click. From the docs, it is not stated anywhere that we can use a list as parameter to @click.option

And when I try to do this:

#!/usr/bin/env python
import click

@click.command(context_settings=dict(help_option_names=['-h', '--help']))
@click.option('--option', default=[])
def do_stuff(option):

    return

# do stuff
if __name__ == '__main__':
    do_stuff()

in my test.py, by calling it from the command line:

python test.py --option ["some option", "some option 2"]

I get an error:

Error: Got unexpected extra argument (some option 2])

I can't really use variadic arguments as only 1 variadic arguments per command is allowed (http://click.pocoo.org/5/arguments/#variadic-arguments)

So if anyone can point me to the right direction (using click preferably) it would be very much appreciated.

Answer 1

If you don't insist on passing something that looks like a list, but simply want to pass multiple variadic arguments, you can use the multiple option.

From the click documentation

@click.command()
@click.option('--message', '-m', multiple=True)
def commit(message):
    click.echo('\n'.join(message))

$ commit -m foo -m bar
foo
bar

Answer 2

You can coerce click into taking multiple list arguments, if the lists are formatted as a string literals of python lists by using a custom option class like:

Custom Class:

import click
import ast

class PythonLiteralOption(click.Option):

    def type_cast_value(self, ctx, value):
        try:
            return ast.literal_eval(value)
        except:
            raise click.BadParameter(value)

This class will use Python's Abstract Syntax Tree module to parse the parameter as a python literal.

Custom Class Usage:

To use the custom class, pass the cls parameter to @click.option() decorator like:

@click.option('--option1', cls=PythonLiteralOption, default=[])

How does this work?

This works because click is a well designed OO framework. The @click.option() decorator usually instantiates a click.Option object but allows this behavior to be over ridden with the cls parameter. So it is a relatively easy matter to inherit from click.Option in our own class and over ride the desired methods.

In this case we over ride click.Option.type_cast_value() and then call ast.literal_eval() to parse the list.

Test Code:

@click.command(context_settings=dict(help_option_names=['-h', '--help']))
@click.option('--option1', cls=PythonLiteralOption, default=[])
@click.option('--option2', cls=PythonLiteralOption, default=[])
def cli(option1, option2):
    click.echo("Option 1, type: {}  value: {}".format(
        type(option1), option1))
    click.echo("Option 2, type: {}  value: {}".format(
        type(option2), option2))

# do stuff
if __name__ == '__main__':
    import shlex
    cli(shlex.split(
        '''--option1 '["o11", "o12", "o13"]' 
        --option2 '["o21", "o22", "o23"]' '''))

Test Results:

Option 1, type: <type 'list'>  value: ['o11', 'o12', 'o13']
Option 2, type: <type 'list'>  value: ['o21', 'o22', 'o23']

Answer 3

The following can be an easier hack fix:

#!/usr/bin/env python
import click
import json

@click.command(context_settings=dict(help_option_names=['-h', '--help']))
@click.option('--option', help='Whatever')
def do_stuff(option):
    try:
        option = json.loads(option)    
    except ValueError:
        pass

# do stuff
if __name__ == '__main__':
    do_stuff()

This can help you to use 'option' as a list or a str.

Answer 4

@Murphy's "easy hack" almost worked for me, but the thing is that option will be a string unless you single quote the options, so I had to do this in order to recompose the list:

#!/usr/bin/env python
import click
import json

@click.command(context_settings=dict(help_option_names=['-h', '--help']))
@click.option('--option', help='Whatever')
def do_stuff(option):
    try:
        option = json.loads(option)
        # option = str(option)  # this also works
    except ValueError:
        pass

    option = option[1:-1]  # trim '[' and ']'

    options = option.split(',')

    for i, value in enumerate(options):
        # catch integers
        try:
            int(value)
        except ValueError:
            options[i] = value
        else:
            options[i] = int(value)

    # Here use options as you need

# do stuff
if __name__ == '__main__':
    do_stuff()

You could catch some other types

To use it, enclose the list into quotes:

python test.py --option "[o11, o12, o13]"

Or you can avoid quotes by not leaving spaces:

python test.py --option [o11,o12,o13]

Answer 5

The answer of Stephen Rauch gave me some issues when working with Kubernetes arguments. This because the string formatting becomes a hassle and often new lines, single quotation marks or spaces were added before and after the array. Hence I made my own parser to improve this behaviour. This code should be self explanatory.
Note that this code does not support single or double quotation marks ' or " in the variables itself.

Custom class:

class ConvertStrToList(click.Option):
    def type_cast_value(self, ctx, value) -> List:
        try:
            value = str(value)
            assert value.count('[') == 1 and value.count(']') == 1
            list_as_str = value.replace('"', "'").split('[')[1].split(']')[0]
            list_of_items = [item.strip().strip("'") for item in list_as_str.split(',')]
            return list_of_items
        except Exception:
            raise click.BadParameter(value)

Custom class usage:

@click.option('--option1', cls=ConvertStrToList, default=[])

Answer 6

Click supports an option taking multiple arguments, as long as the number of arguments is predetermined. The arguments are separated by whitespace with no list-like syntax: one would write --my-2-arg-option arg1 arg2 rather than --my-2-arg-option ["arg1", "arg2"].

This response does answer your question about "how to pass several lists of arguments to @click.option," just that the lists need to be given without brackets or commas on the command line. A Python list could be formatted this way as follows, using shlex.join to add quotation marks when necessary:

>>> import shlex
>>> args = ["o11", "o12", "o13"]
>>> shlex.join(args)
'o11 o12 o13'
>>> args_needing_quotes = ["I have whitespace",
                             "$I_LOOK_LIKE_A_VARIABLE",
                             "$(i_have_special_characters)#!\n > *"]
>>> shlex.join(args_needing_quotes)
"'I have whitespace' '$I_LOOK_LIKE_A_VARIABLE' '$(i_have_special_characters)#!\n > *'"

Option that takes multiple arguments, all of the same type

This kind of option seems to be what you are looking for, and can be implemented with the following code (call it script1.py):

import click
@click.command()
@click.option("--option1", nargs=3)
def do_stuff(option1):
    print("Option 1 is", option1)
do_stuff()

The parameter nargs tells Click how many arguments the option must accept; by default, nargs is 1.

Running this code prints the following output:

$ python script1.py --option1 four 4 IV
Option 1 is ('four', '4', 'IV')

We can see that the variable option1 is a tuple of three str values.

If fewer or more than 3 arguments are given for option1, then the script exits immediately:

$ python script1.py --option1 four 4
Error: Option '--option1' requires 3 arguments.

If option1 is omitted, then it defaults to None:

$ python script1.py
Option 1 is None

Default values of multi-argument options

A default value can be specified when the option is created. Note that the default value should also be a tuple of three str values to avoid any unexpected behavior. Particularly, it should not be a list (or any other mutable type), as mutable default arguments can cause unexpected behavior. Call the code with this change script2.py:

@click.option("--option1", nargs=3, default=('', '', ''))

Now the script prints the default when run with no arguments:

$ python script2.py          
Option 1 is ('', '', '')

Types other than strings

Click will also automatically cast values to a different data type if given. Let script3.py be

import click
@click.command()
@click.option("--option1", nargs=3, type=float)
def do_stuff(option1):
    print("Option 1 is", option1)
do_stuff()

Now option1 is a tuple of three float values:

$ python script3.py --option1 1 2.718 3.142
Option 1 is (1.0, 2.718, 3.142)

Option that takes multiple arguments of different data types

The previous examples showed how to create an option that takes multiple values of the same data type. What about an option that takes arguments of multiple data types?

Click provides a way to do this as well. Instead of setting nargs, set type to be a tuple of the desired data types. For example, let script4.py be

import click
@click.command()
@click.option("--comp", type=(float, str, float))
def calculate(comp):
    num1, op, num2 = comp
    if op == "+":
        result = num1 + num2
    elif op == "-":
        result = num1 - num2
    else:
        raise ValueError(f"Unsupported operation: {op}")
    print(f"{num1} {op} {num2} = {result}")
calculate()

Then we can use our rudimentary calculator:

$ python script4.py --comp 3 + 6
3.0 + 6.0 = 9.0
$ python script4.py --comp -1.2 - 3.7
-1.2 - 3.7 = -4.9

If any of the values are invalid, then click will raise an error automatically:

$ python script4.py --comp 1 + two
Usage: script4.py [OPTIONS]
Try 'script4.py --help' for help.

Error: Invalid value for '--comp': 'two' is not a valid float.

Option that can be given multiple times and takes multiple values

Click can also create options that can be given multiple times using the multiple keyword (script5.py):

import click
@click.command()
@click.option("--option1", nargs=2, multiple=True)
def do_stuff(option1):
    print(option1)
do_stuff()

Now we see that option1 becomes a tuple of tuples of two strs each:

$ python script5.py --option1 3 a --option1 b 7
(('3', 'a'), ('b', '7'))

We can also mix multiple=True with the type keyword in script6.py:

import click
@click.command()
@click.option("--comps", type=(float, str, float), multiple=True)
def calculate(comps):
    for comp in comps:
        num1, op, num2 = comp
        if op == "+":
            result = num1 + num2
        elif op == "-":
            result = num1 - num2
        else:
            raise ValueError(f"Unsupported operation: {op}")
        print(f"{num1} {op} {num2} = {result}")
calculate()

This functionality can allow us to, for example, code a simple calculator that performs multiple operations in one call to script6.py:

python script6.py --comps 4 - -7 --comps -8 + 4.2 --comps 16 - 34.1
4.0 - -7.0 = 11.0
-8.0 + 4.2 = -3.8
16.0 - 34.1 = -18.1

The full documentation for multi-value options in Click can be found here.

Note: I ran all code examples in Python 3.11.0 with Click 8.1.3