argparse fails when called from unittest test

Question

In a file (say parser.py) I have:

import argparse

def parse_cmdline(cmdline=None):
    parser = argparse.ArgumentParser()
    parser.add_argument('--first-param',help="Does foo.")
    parser.add_argument('--second-param',help="Does bar.")

    if cmdline is not None:
        args = parser.parse_args(cmdline)
    else:
        args = parser.parse_args()

    return vars(args)

if __name__=='__main__':
    print parse_cmdline()

Sure enough, when called from the command line it works and give me pretty much what I expect:

$ ./parser.py --first-param 123 --second-param 456
{'first_param': '123', 'second_param': '456'}

But then I want to unittest it, thus I write a test_parser.py file:

import unittest
from parser import parse_cmdline

class TestParser(unittest.TestCase):
    def test_parse_cmdline(self):
        parsed = parse_cmdline("--first-param 123 --second-param 456")

        self.assertEqual(parsed['first_param'],'123')
        self.assertEqual(parsed['second_param'],'456')

if __name__ == '__main__':
    unittest.main()

Then I get the following error:

usage: test_parser.py [-h] [--first-param FIRST_PARAM]
                      [--second-param SECOND_PARAM]
test_parser.py: error: unrecognized arguments: - - f i r s t - p a r a m   1 2 3   - - s e c o n d - p a r a m   4 5 6
E
======================================================================
ERROR: test_parse_cmdline (__main__.TestParser)
----------------------------------------------------------------------
Traceback (most recent call last):
  File "./test_parser.py", line 8, in test_parse_cmdline
    parsed = parse_cmdline("--first-param 123 --second-param 456")
  File "/home/renan/test_argparse/parser.py", line 12, in parse_cmdline
    args = parser.parse_args(cmdline)
  File "/usr/lib/python2.7/argparse.py", line 1691, in parse_args
    self.error(msg % ' '.join(argv))
  File "/usr/lib/python2.7/argparse.py", line 2361, in error
    self.exit(2, _('%s: error: %s\n') % (self.prog, message))
  File "/usr/lib/python2.7/argparse.py", line 2349, in exit
    _sys.exit(status)
SystemExit: 2

----------------------------------------------------------------------
Ran 1 test in 0.004s

FAILED (errors=1)

As can be seen, the command line I specified (--first-param 123 --second-param 456) became - - f i r s t - p a r a m 1 2 3 - - s e c o n d - p a r a m 4 5 6 (each character is separated by a space).

I don't understand why: what am I doing wrong?

score 8 · Accepted Answer · answered Aug 20 '13 at 01:08

argparse wants an "argument vector"—that is, a list of separate arguments—not a "command line" string.

And just calling split won't solve things. For example, this command line from the shell:

python script.py --first-param '123 456' --second-param 789

… will correctly give you 123 456 and 789, but this line of code:

parse_cmdline("--first-param '123 456' --second-param 789")

will not; it will give you '123 and 789, with an extra 456' that it doesn't know how to deal with.

In fact, even this is wrong:

parse_cmdline("--first-param '123' --second-param 789")

… because you'll get '123' instead of 123.

There are two ways to deal with this.

First, if you know what you're trying to pass, you can just pass it as a list instead of a string in the first place, and you don't need to worry about fiddly quoting and splitting details:

parse_cmdline(["--first-param", "123 456", "--second-param", "789"])

Alternatively, if you don't know exactly what you're trying to pass, you just know what it looks like on the shell, you can use shlex:

if cmdline is not None:
    args = parser.parse_args(shlex.split(cmdline))

… and now Python will split your command line the same way as a standard Unix shell, getting the 123 456 as a single parameter.

The shlex-based solution is good enough for me right now. Thanks! — Renan, Aug 20 '13 at 01:10
While it is good to keep the distinction between `split` and `shlex` in mind, it does not test `arpgarse` itself. `test_argparse.py` uses `split` a lot, but does not import `shlex`. If blanks inside the arguments matter, I would just feed `parse_args` a test list of words. — hpaulj, Aug 20 '13 at 15:26

score 1 · Answer 2 · answered Aug 20 '13 at 00:41

1

To answer myself (I realized my mistake a few minutes later):

I am supposed to

if cmdline is not None:
    args = parser.parse_args(cmdline.split())
else:
    args = parser.parse_args()

Now the test passes correctly!

answered Aug 20 '13 at 00:41

Renan

462
7
24

There's a serious problem with this. Try it with `parse_cmdline("--first-param '123 456' --second-param 789")`. That will not do the same thing as if you pass the same thing from the actual shell. – abarnert Aug 20 '13 at 01:04
@abarnert I see; what would be the best solution for this, then? – Renan Aug 20 '13 at 01:08
1

It depends on your exact use case. I've written an answer explaining the two standard solutions. – abarnert Aug 20 '13 at 01:09
Yes, in your original case, `parse_args` wraps your test string in `list()`, turning it into a list of individual characters. Hence the spaces in "- - f i ...". The comment in the code is 'make sure the args are mutable'. Evidently it's more concerned with getting a tuple than a string. Maybe it should object if given a string. – hpaulj Aug 20 '13 at 15:29

argparse fails when called from unittest test

2 Answers2

Linked