testing command line utilities

Question

I'm looking for a way to run tests on command-line utilities written in bash, or any other language.

I'd like to find a testing framework that would have statements like

setup:
    command = 'do_awesome_thing'
    filename = 'testfile'
    args = ['--with', 'extra_win', '--file', filename]
    run_command command args

test_output_was_correct
    assert_output_was 'Creating awesome file "' + filename + '" with extra win.'

test_file_contains_extra_win
    assert_file_contains filename 'extra win'

Presumably the base test case would set up a temp directory in which to run these commands, and remove it at teardown.

I would prefer to use something in Python, since I'm much more familiar with it than with other plausible candidate languages.

I imagine that there could be something using a DSL that would make it effectively language-agnostic (or its own language, depending on how you look at it); however this might be less than ideal, since my testing techniques usually involve writing code that generates tests.

It's a bit difficult to google for this, as there is a lot of information on utilities which run tests, which is sort of the converse of what I'm looking for.

Support for doctests embedded in the output of command --help would be an extra bonus :)

You might get some useful info from a question I asked about unit testing shell scripts: http://stackoverflow.com/questions/971945/unit-testing-for-shell-scripts — gareth_bowles, Jun 21 '10 at 22:28
@gareth_bowles: cool, thanks for the link. I might try using shunit2 if it's bash-compatible. — intuited, Jun 21 '10 at 23:29

score 13 · Accepted Answer · answered Jun 22 '10 at 05:16

Check out ScriptTest :

from scripttest import TestFileEnvironment

env = TestFileEnvironment('./scratch')

def test_script():
    env.reset()
    result = env.run('do_awesome_thing testfile --with extra_win --file %s' % filename)
    # or use a list like ['do_awesome_thing', 'testfile', ...]
    assert result.stdout.startswith('Creating awesome file')
    assert filename in result.files_created

It's reasonably doctest-usable as well.

score 1 · Answer 2 · answered Jan 28 '15 at 15:51

I know that this question is old, but since I was looking for an answer, I figured I would add my own for anyone else who happens along.

Full disclaimer: The project I am mentioning is my own, but it is completely free and open source.

I ran into a very similar problem, and ended up rolling my own solution. The test code will look like this:

from CLITest import CLITest, TestSuite
from subprocess import CalledProcessError


class TestEchoPrintsToScreen(CLITest):
    '''Tests whether the string passed in is the string
    passed out'''

    def test_output_contains_input(self):
        self.assertNotIsInstance(self.output, CalledProcessError)
        self.assertIn("test", self.output)

    def test_ouput_equals_input(self):
        self.assertNotIsInstance(self.output, CalledProcessError)
        self.assertEqual("test", self.output)

suite = TestSuite()

suite.add_test(TestEchoPrintsToScreen("echo test"))

suite.run_tests()

This worked well enough to get me through my issues, but I know it could use some more work to make it as robust as possible (test discovery springs to mind). It may help, and I always love a good pull request.

score 1 · Answer 3 · answered Jun 21 '10 at 21:28

1

Well... What we usually do (and one of the wonders of O.O. languages) is to write all the components of an application before actually make the application. Every component might have an standalone way to be executed, for testing purpose (command line, usually), that also allows you to think in them as complete programs each by each, and use them in future projects. If what you want is to test the integrity of an existing program... well, I think the best way is to learn in deep how it work, or even deeper: read the source. Or even deeper: develop a bot to force-test it :3

Sorry that's what I have .-.

answered Jun 21 '10 at 21:28

Daniel Rodriguez MSFT

83
1
1
9

Definitely.. that's a great way to do things. I gather that what you describe would be termed "unit testing", whereas I want a toolset to do "acceptance testing" or "functional testing" or "integration testing". This is partly to provide an additional means of verifying code that is unit testable, and also to allow for some sort of testing on code that isn't unit-testable: bash scripts (barring the existence of —shudder— a bash unit testing framework); proprietary out-of-house utilities that behave in poorly-documented ways; or code in which a panoply of aspects is mixed in a single routine. – intuited Jun 21 '10 at 21:56
Also: to flow with your terminology, I guess what I'm looking for is a "bot framework". – intuited Jun 21 '10 at 21:58
what I understand you want to do is to find a generic tester for programs with CLI? I find it enourmously hard to realize, because a program so bad documentated is unpredictable. If you face a wrong enough software, and it source is neither available, the best choice seems to be: try searching for similar programs. Or why not? post your trouble in a forum :] Is that ok or I missed something? If I did, try to be more explicit about your personal case. – Daniel Rodriguez MSFT Jun 21 '10 at 22:12
A bot, well that's about making brute force to the program, that could last forever, and also might be dangerous, as you don't know how it works or what it does for sure. – Daniel Rodriguez MSFT Jun 21 '10 at 22:15
A test is always better than no test. – Ian Bicking Jun 22 '10 at 05:21

score 0 · Answer 4 · answered Jun 22 '10 at 15:00

outside of any prepackaged testing framework that may exist but I am unaware of, I would just point out that expect is an awesome and so underutilized tool for this kind of automation, especially if you want to support multistage interaction, which is to say not just send a command and check output but respond to output with more input. If you wind up building your own system, it's worth looking into.

There is also python reimplementation of expect called pexpect.There may be some direct interfaces to the expect library available as well. I'm not a python guy so I couldn't tell you much about them.

testing command line utilities

4 Answers4

Linked