42

I have a python script that gave different output when run on a Windows machine and when run on a Mac. On digging deeper, I discovered that it was because when Python read in line breaks on the Mac (from a file), it read in \r\n, while somehow in Windows the \r disappears.

Thus, if I change every \n in the script to \r\n, it works fine on the Mac. But if I do that, it stops working on the Windows PC.

Is there an easy way to fix this problem?

Lennart Regebro
  • 167,292
  • 41
  • 224
  • 251
wrongusername
  • 18,564
  • 40
  • 130
  • 214
  • 1
    You're having this error when READING from a file? What version of python are you on? Where did you get it? Python is usually built with universal newline support – Falmarri Jan 05 '11 at 01:03
  • @Falmarri Python 3... and I'm pretty sure I got it from the official python website – wrongusername Jan 05 '11 at 01:04
  • 4
    Here's another Windows vs Linux inconsistency - on Windows, Python's glob.glob function always returns a list filenames, sorted alphabetically. On Linux, the list of filenames is returned in random order. – PaulMcG Jan 05 '11 at 01:17
  • 2
    Well `glob.glob` doesn't claim to return results in any particular order, so no reason to expect them to be in any particular order – Mark Jun 11 '14 at 16:37

5 Answers5

46

Different platforms have different codes for "new line". Windows have \r\n, Unix has \n, Old macs have \r and yes there are some systems that have \n\r too.

When you open a file in text mode in Python 3, it will convert all newlines to '\n' and be done with it.

infile = open("filename", 'r')

Text mode is default, so if you say nothing, it's text mode. But it's always better to be explicit:

infile = open("filename", 'rt')

If you don't want the translation of line endings to happen, open the file in binary mode:

infile = open("filename", 'rb')

In Python 2 it's different. There this conversion would only happen by default on Windows. If you wanted it to happen on other platforms, you could add the universal newline flag:

infile = open("filename", 'rU')

However, you say that you are on Python 3, and there it happens in text mode on all platforms, so adding the U flag should make no difference.

Lennart Regebro
  • 167,292
  • 41
  • 224
  • 251
  • Thank you so much I had an idea it was like that, it took me a pretty long time to confirm it, but yeah my code worked on python 2.7 but glitched on python 3.2 because of that difference. I needed to "rb" it... thanks, very thorough explanation. – sinekonata Nov 20 '12 at 06:17
  • `os.linesep` is replaced by `\n` while reading in text mode on *all* platforms (not only Windows). If `os.linesep == '\n'` then it is the same as no conversion on Python 2. – jfs Jul 10 '13 at 05:53
  • @J.F.Sebastian: Which in practice means that it only happens on Windows (and Mac OS 9, but that's not supported since Python 2.4). While in Python 3 all newline combinations will be translated to `'\n'` on all platforms. – Lennart Regebro Jul 10 '13 at 07:09
  • and how can you handle this if you use with `codecs.open()`? – Intelligent-Infrastructure Apr 18 '18 at 11:26
  • Codecs.open() also takes file mode flags. – Lennart Regebro Apr 24 '18 at 15:34
26

'U' mode:

Python 2:

I guess it may depend on what you're reading from, but the built-in open() function takes a 'mode' parameter, and if you pass 'U' for the mode, Python 2 will take care of the newlines in a cross-platform way transparently. It requires that Python be built with universal newline support, but test it out!

https://docs.python.org/2/library/functions.html#open

Python 3:

In Python 3, the 'U' mode is the default behaviour, as the docs explain:

There is an additional mode character permitted, 'U', which no longer has any effect, and is considered deprecated. It previously enabled universal newlines in text mode, which became the default behaviour in Python 3.0. Refer to the documentation of the newline parameter for further details.

https://docs.python.org/3/library/functions.html#open

Flimm
  • 136,138
  • 45
  • 251
  • 267
jonesy
  • 3,502
  • 17
  • 23
  • 1
    This is also the default (as far as I know) – Falmarri Jan 05 '11 at 01:08
  • Thanks! It works fine now! Oh and @Falmarri sorry, the only way I knew of how to read from a file was with `'r'`. At least now I know better. – wrongusername Jan 05 '11 at 01:14
  • Hmm, it sounds like that's not the default in python 3. That seems weird. – Falmarri Jan 05 '11 at 01:20
  • 7
    @Falmarri: It *is* default in Python 3, and in fact, the U flag is deprecated in Python 3. – Lennart Regebro Jan 05 '11 at 07:21
  • @Lennart that's interesting... wonder how my mac screwed it up – wrongusername Jan 05 '11 at 08:11
  • @wrongusername: Good question. It *sounds* like you are using Python 2, to be honest. – Lennart Regebro Jan 05 '11 at 08:13
  • @Lennart Heh it does sort of sound like it, but I am using `format` s on strings and prints with parentheses, so it's definitely Python 3 – wrongusername Jan 05 '11 at 08:18
  • 1
    @wrongusername: No, that works fine under Python 2.6 as well. You'll see the Python version when you start the python interpreter. – Lennart Regebro Jan 05 '11 at 08:23
  • @Lennart really? I thought string.format stuff were exclusive to python 3 (http://stackoverflow.com/questions/2450188/string-formatting-error) and get syntax errors when I use print as a keyword? anyways, when I start up the python shell I get "Python 3.1.3 (r313:86882M, Nov 30 2010, 09:55:56) [GCC 4.0.1 (Apple Inc. build 5494)] on darwin" – wrongusername Jan 06 '11 at 01:08
  • @wrongusername: Yes, you get errors when you use print as a keyword in Python 3. But you do *not* get errors because you stick two parenthesis around an expression in Python 2. Hence print(foo) works just fine in Python 2 and has always done. The \r should disappear on OS X as well, unless you open the file in binary mode. – Lennart Regebro Jan 06 '11 at 06:45
  • unfortunately, the documentation indicates that Universal Newlines mode is deprecated. It looks like exactly what I'd want, but doesn't seem to explain why it is deprecated or what the replacement mechanism might be. – MikeB Jul 31 '17 at 14:31
  • 1
    @MikeB the feature itself is not deprecated. It's the default. You don't need to do anything to enable it. When you open a file in text mode, it's opened in Universal Newlines mode. The thing that is deprecated is setting `mode="U"` specifically. If you want to *disable* Universal Newlines then you can pass `newline="\n"` or whatever. See [this question](https://softwareengineering.stackexchange.com/questions/298677/why-is-universal-newlines-mode-deprecated-in-python). – Boris Verkhovskiy Mar 21 '19 at 18:37
8

In Python 3, the Open() method has a newline parameter:

newline controls how universal newlines mode works (it only applies to text mode). It can be None, '', '\n', '\r', and '\r\n'. It works as follows:

When reading input from the stream, if newline is None, universal newlines mode is enabled. Lines in the input can end in '\n', '\r', or '\r\n', and these are translated into '\n' before being returned to the caller. If it is '', universal newlines mode is enabled, but line endings are returned to the caller untranslated. If it has any of the other legal values, input lines are only terminated by the given string, and the line ending is returned to the caller untranslated.

When writing output to the stream, if newline is None, any '\n' characters written are translated to the system default line separator, os.linesep. If newline is '' or '\n', no translation takes place. If newline is any of the other legal values, any '\n' characters written are translated to the given string.

The old way of using U mode specifier has been deprecated in favor of this new way.

'U' universal newlines mode (deprecated)

jdhao
  • 24,001
  • 18
  • 134
  • 273
7

In Python 3, use the keyword argument "newline='\n'" in open() to use a specified line delimiter when writing text files. For more information, please see:

https://pythonconquerstheuniverse.wordpress.com/2011/05/08/newline-conversion-in-python-3/

http://docs.python.org/3/library/functions.html#open

Felix
  • 79
  • 1
  • 2
1

On windows, they both are working fine if i try writing a file with either of the two(\r or \n) python interprets it as a line break in both the cases. While using "\r\n", it is interpreted as a double line break.(Python 3 on windows)

as g
  • 55
  • 6