Can't use Python's sh module in Bazel genrule

Question

When I run a python script, which uses the 'sh' module, from bazel genrule, it failed with this:

INFO: Analysed target //src:foo_gen (8 packages loaded).
INFO: Found 1 target...
ERROR: /home/libin11/workspace/test/test/src/BUILD:1:1: Executing genrule //src:foo_gen failed (Exit 1)
Traceback (most recent call last):
  File "src/test.py", line 2, in <module>
    sh.touch("foo.bar")
  File "/usr/local/lib/python2.7/dist-packages/sh.py", line 1427, in __call__
    return RunningCommand(cmd, call_args, stdin, stdout, stderr)
  File "/usr/local/lib/python2.7/dist-packages/sh.py", line 767, in __init__
    self.call_args, pipe, process_assign_lock)
  File "/usr/local/lib/python2.7/dist-packages/sh.py", line 1784, in __init__
    self._stdout_read_fd, self._stdout_write_fd = pty.openpty()
  File "/usr/lib/python2.7/pty.py", line 29, in openpty
    master_fd, slave_name = _open_terminal()
  File "/usr/lib/python2.7/pty.py", line 70, in _open_terminal
    raise os.error, 'out of pty devices'
OSError: out of pty devices
Target //src:foo_gen failed to build
Use --verbose_failures to see the command lines of failed build steps.
INFO: Elapsed time: 2.143s, Critical Path: 0.12s
INFO: 0 processes.
FAILED: Build did NOT complete successfully

I want to integrate a thirdparty project to my own. The third party project is built with a python script, so I would like to build the project with bazel genrule.

Here is an example file list:

.
├── src
│   ├── BUILD
│   └── test.py
└── WORKSPACE

WORKSPACE is empty, BUILD is:

genrule(
    name = "foo_gen",
    srcs = glob(["**/*"]),
    outs = ["foo.bar"],
    cmd = "python $(location test.py)",
)

test.py is:

import sh
sh.touch("foo.bar")

And run:

bazel build //src:foo_gen

OS: Ubuntu 16.04 bazel: release 0.14.1

Answer 1

It looks like if you change the call to sh.touch("foo.bar", _tty_in=False, _tty_out=False) it works, but you'll still need a bit of modification to the genrule otherwise it won't produce output.

I prefer to import pip dependencies using the bazel python rules, so I can create the tool for my genrule. This way, bazel handles the pip requirement install and you don't have to chmod the test.py file.

load("@my_deps//:requirements.bzl", "requirement")

py_binary(
    name = "foo_tool",
    srcs = [
        "test.py",
    ],
    main = "test.py",
    deps = [
        requirement("sh"),
    ],
)

genrule(
    name = "foo_gen",
    outs = ["foo.bar"],
    cmd = """
      python3 $(location //src:foo_tool)
      cp foo.bar $@
    """,
    tools = [":foo_tool"],
)

Note the required copy in the genrule command. It's a bit cleaner if your python script can output to std out, then you can just redirect the output to the file instead of adding a copy command. See this for more info.

My output with these changes:

INFO: Analysed target //src:foo_gen (0 packages loaded).
INFO: Found 1 target...
Target //src:foo_gen up-to-date:
  bazel-genfiles/src/foo.bar
INFO: Elapsed time: 0.302s, Critical Path: 0.00s
INFO: 0 processes.
INFO: Build completed successfully, 1 total action