Print only the contents of a string

Question

Here's my code:

#! /usr/bin/env python3
import subprocess
a = subprocess.check_output('echo -n "hello world!"',shell=True)
print("a="+str(a))

output:

a=b'hello world!'

If I include the argument universal_newlines=True in the call to check_output, then I get the desired output:

a=hello world!

For the sake of better understanding the mysterious world of programming with text in the modern (Unicode) age, I would like to know how to generate the second output without specifying universal_newlines=True. In other words, what function do I call to convert a so that it will produce the desired output.

A working example would go a long way. Detailed explanations are nice, but they tend to be a bit confusing for the uninitiated -- maybe due to the use of overloaded terminology, maybe because of differences between Python2 and Python3, or maybe just because I very rarely need to think about text encoding in my line of work -- most of the tools that I work with don't require special handling like this.

Also: I believe the first output is of type bytes, but what is the type of the second output? My guess is str with UTF-8 encoding.

Answer 1

As originally implied by Ignacio's comment, you could use decode:

>>> a = b"hello world!"
>>> print("a="+str(a))
a=b'hello world!'
>>> print("a="+a.decode())
a=hello world!

Answer 2

From subprocess.check_output() docs:

By default, this function will return the data as encoded bytes. The actual encoding of the output data may depend on the command being invoked, so the decoding to text will often need to be handled at the application level.

This behaviour may be overridden by setting universal_newlines to True as described below in Frequently Used Arguments.

If you follow the link to Frequently Used Arguments; it describes what universal_newlines=True does:

If universal_newlines is False the file objects stdin, stdout and stderr will be opened as binary streams, and no line ending conversion is done.

If universal_newlines is True, these file objects will be opened as text streams in universal newlines mode using the encoding returned by locale.getpreferredencoding(False). For stdin, line ending characters '\n' in the input will be converted to the default line separator os.linesep. For stdout and stderr, all line endings in the output will be converted to '\n'. For more information see the documentation of the io.TextIOWrapper class when the newline argument to its constructor is None.

For more details you could look at io.TextIOWrapper() documentation.

To run your echo -n "hello world!" shell command and to return text without check_output() and without using universal_newlines=True:

#!/usr/bin/env python
import locale
from subprocess import Popen, PIPE

charset = locale.getpreferredencoding(False)
with Popen(['echo', 'Hello world!'], stdout=PIPE) as process:
    output = process.communicate()[0].decode(charset).strip()

Here's a couple of code examples that show how subprocess pipes and TextIOWrapper class could be used together.

To understand what is text and what is binary data in Python, read Unicode HOWTO. Here's the most important part: there are two major string types in Python: bytestrings (a sequence of bytes) that represent binary data and Unicode strings (a sequence of Unicode codepoints) that represent human-readable text. It is simple to convert one into another (☯):

unicode_text = bytestring.decode(character_encoding)
bytestring = unicode_text.encode(character_encoding)

Answer 3

Also: I believe the first output is of type bytes, but what is the type of the second output? My guess is str with UTF-8 encoding.

Close, but not quite right. In Python3 the str type is indexed by Unicode code points (note that code points usually, but not always, have a 1:1 correspondence with user perceived characters). Therefore, the underlying encoding is abstracted away when using the str type -- consider it unencoded, even though that is fundamentally not the case. It is the bytes type which is indexed as a simple array of bytes and therefore must use a particular encoding, In this case (as in most similar usages), ASCII would suffice for decoding what was generated by the subprocess script.

Python2 has different defaults for the interpretation of the str type (see here), so string literals will be represented differently in that version of the language (this difference could be a big stumbling block when researching text handling).

As a person who mostly uses C++, I found the following to be extremely enlightening about the practical storage, encoding, and indexing of Unicode text: How do I use 3 and 4-byte Unicode characters with standard C++ strings?

---

So the answer to the first part of the question is bytes.decode():

a = a.decode('ascii') ## convert from `bytes` to 'str' type

although simply using

a = a.decode() ## assumes UTF-8 encoding

will typically produce the same results since ASCII is a subset of UTF-8.

Alternatively, you can use str() like so:

a = str(a,encoding='ascii')

but note that here an encoding must be specified if you want the "contents only" representation -- otherwise it will actually build a str type which internally contains the quote characters (including the 'b' prefix), which is exactly what was happening in the first output shown in the question.

---

subprocess.check_output processes the data in binary mode (returning the raw byte sequence) by default, but the cryptic argument universal_newlines=True basically tells it to decode the string and represent it as text (using the str type). This conversion to the str type is necessary (in Python3) if you want to display the output (and "only the contents") using Python's print function.

The funny thing about this conversion is that, for these purposes, it really doesn't do anything to the data. What happens under the hood is an implementation detail, but if the data is ASCII (as is very typical for this type of program) it essentially just gets copied from one place to another without any meaningful translation. The decode operation is just hoop jumping to change the data type -- and the seemingly pointless nature of the operation further obfuscates the larger vision behind Python's text handling (for the uninitiated). Further, since the docs don't make the return type(s) explicit (by name), it's hard to even know where to start looking for the appropriate conversion function.