1

I have my .proto. files defined in a folder workspace_directory/sub_directory/proto_files.

When I run:

protoc --python_out=workspace_directory -I workspace_directory/sub_directory/proto_files workspace_directory/sub_directory/proto_files/*

the output python code is generated in workspace_directory/proto_files, which is what I want.

My main goal is to run that command as part of a build script in Python, so I attempted to use subprocess.run() to achieve that. In Python I run:

subprocess.run(shlex.split("protoc --python_out=workspace_directory -I workspace_directory/sub_directory/proto_files workspace_directory/sub_directory/proto_files/*"))

I get

Could not make proto path relative: workspace_directory/sub_directory/proto_files/*: No such file or directory

Some other things to note:

  • workspace_directory is a fully qualified path. Its like /home/user/workspace_directory, so I don't believe I should be having any relative path problems.
  • I used shlex.split() so I could copy/paste the command, to ensure it was as exact as possible.

It feels like I'm missing something when using the subprocess module, but can't seem to see it.

Hai Vu
  • 37,849
  • 11
  • 66
  • 93
Trashfire
  • 118
  • 1
  • 6
  • are you using `shlex.split` properly? [resource](https://stackoverflow.com/a/34679668/6651840) – Dean Van Greunen Jul 26 '22 at 16:04
  • 2
    @Dean yes, I believe so. There are no single quotes in there so its a very straight forward split. I did also attempt without `shlex.split()`. Is there something in what I posted that makes you think the `shlex.split()` call is incorrect? – Trashfire Jul 26 '22 at 16:33
  • perhaps the values passed to `subprocess.run` is wrong, recheck the args for `subprocess.run` – Dean Van Greunen Jul 26 '22 at 19:48
  • 1
    @Dean I feel confident in the inputs to `subprocess.run`. I have spent time looking at the `CompletedProcess`, comparing the arguments there to the successful command line call. They are identical. – Trashfire Jul 26 '22 at 20:42

2 Answers2

2

protoc --python_out=workspace_directory -I workspace_directory/sub_directory/proto_files workspace_directory/sub_directory/proto_files/*

When you execute this command normally in bash or other shell, the shell detects the * wildcard character and converts it to a list of matching filenames.

Neither subprocess.run() nor shlex.split() do any wildcard handling, so protoc receives the text as-is and tries to open a file literally called *, which it cannot find. The error message is a bit confusing, but contains the reason: No such file or directory.

Calling the command through the shell is a reasonable solution as long as you can trust all the text strings you include in the command. Alternatively you can use the Python glob module to expand the wildcard to a list of paths and include it in the command. That can be safer if the filenames may contain special characters.

(You may also be interested in knowing that protoc is available as a Python module in grpcio-tools and can be called using import grpc_tools.protoc; grpc_tools.protoc(['protoc', '-I...', 'myproto.proto']). This avoids having to go through a subprocess, and can be easier to install using the Python pip package manager.)

jpa
  • 10,351
  • 1
  • 28
  • 45
1

I was able to find a solution. I'm still not positive of why the difference between passing a sequence of arguments and using a string with shell=True

The subprocess docs say

args is required for all calls and should be a string, or a sequence of program arguments. Providing a sequence of arguments is generally preferred, as it allows the module to take care of any required escaping and quoting of arguments (e.g. to permit spaces in file names). If passing a single string, either shell must be True (see below) or else the string must simply name the program to be executed without specifying any arguments.

I had been providing the sequence of arguments and not getting the result I wanted. Passing a string and setting shell=True made it work.

subprocess.run("protoc --python_out=workspace_directory -I workspace_directory/sub_directory/proto_files workspace_directory/sub_directory/proto_files/*", shell=True)

Trashfire
  • 118
  • 1
  • 6