81

I'm looking for an equivalent to sscanf() in Python. I want to parse /proc/net/* files, in C I could do something like this:

int matches = sscanf(
        buffer,
        "%*d: %64[0-9A-Fa-f]:%X %64[0-9A-Fa-f]:%X %*X %*X:%*X %*X:%*X %*X %*d %*d %ld %*512s\n",
        local_addr, &local_port, rem_addr, &rem_port, &inode);

I thought at first to use str.split, however it doesn't split on the given characters, but the sep string as a whole:

>>> lines = open("/proc/net/dev").readlines()
>>> for l in lines[2:]:
>>>     cols = l.split(string.whitespace + ":")
>>>     print len(cols)
1

Which should be returning 17, as explained above.

Is there a Python equivalent to sscanf (not RE), or a string splitting function in the standard library that splits on any of a range of characters that I'm not aware of?

Andrew Grimm
  • 78,473
  • 57
  • 200
  • 338
Matt Joiner
  • 112,946
  • 110
  • 377
  • 526
  • Is there any reason you are insisting on "not RE"? Regexes are the perfect tool for this job. – Max Shawabkeh Feb 01 '10 at 06:46
  • 8
    If you want to program in C, why not program in C? If you want to program in python, use a regular expression. There's even a helpful hint in the documentation for the re module telling you how to convert scanf formats into regular expressions. http://docs.python.org/library/re.html#simulating-scanf –  Feb 01 '10 at 07:47
  • 9
    @Paul, the last bit would have made a great answer. – Matt Joiner Feb 01 '10 at 10:45
  • @MattJoiner, I think it would be better to request/disallow features than to request/disallow implementations. "I would like to have format strings that specify the type of the output variable, to have the types converted for me, and to assert specific formatting of the input string" rather than "not regex" explains why you have this preference. After all, if someone used regex to build what you wanted, you'd use it, wouldn't you? – interestinglythere Nov 13 '15 at 14:43
  • @interestinglythere: wat – Matt Joiner Nov 14 '15 at 02:27
  • I think it would be best to say which properties of regex you want to avoid rather than to avoid regex altogether. After all, regex might turn out to be the most appropriate tool for the job. It might be possible that someone has made the tool you're looking for, except that it uses regex behind the scenes. In that case I'd imagine you would still want to use that tool. – interestinglythere Nov 15 '15 at 05:06
  • Perhaps [pandas can help](https://pandas.pydata.org/pandas-docs/stable/reference/io.html)? – gerrit Jan 15 '20 at 17:39
  • The Python documentation for its regex module, `re`, includes a section on simulating `scanf`, which I found more useful than any of the answers posted so far. https://docs.python.org/2/library/re.html#simulating-scanf – Jon Dec 19 '16 at 12:38

9 Answers9

100

There is also the parse module.

parse() is designed to be the opposite of format() (the newer string formatting function in Python 2.6 and higher).

>>> from parse import parse
>>> parse('{} fish', '1')
>>> parse('{} fish', '1 fish')
<Result ('1',) {}>
>>> parse('{} fish', '2 fish')
<Result ('2',) {}>
>>> parse('{} fish', 'red fish')
<Result ('red',) {}>
>>> parse('{} fish', 'blue fish')
<Result ('blue',) {}>
Craig McQueen
  • 41,871
  • 30
  • 130
  • 181
74

When I'm in a C mood, I usually use zip and list comprehensions for scanf-like behavior. Like this:

input = '1 3.0 false hello'
(a, b, c, d) = [t(s) for t,s in zip((int,float,strtobool,str),input.split())]
print (a, b, c, d)

Note that for more complex format strings, you do need to use regular expressions:

import re
input = '1:3.0 false,hello'
(a, b, c, d) = [t(s) for t,s in zip((int,float,strtobool,str),re.search('^(\d+):([\d.]+) (\w+),(\w+)$',input).groups())]
print (a, b, c, d)

Note also that you need conversion functions for all types you want to convert. For example, above I used something like:

strtobool = lambda s: {'true': True, 'false': False}[s]
Chris Dellin
  • 849
  • 1
  • 6
  • 3
  • 1
    I really like this approach, especially as my problem was not just a need for `scanf`, but `sscanf`. – JWL Oct 07 '12 at 10:05
  • 2
    This appeared to be a good solution; sadly `bool("false")` returns `True`, because only empty strings evaluate to `False`. However, all is not lost, you could replace `bool` with a custom function which behaves the way you'd like. – Aky Nov 14 '17 at 14:36
  • @Aky Nice catch! I fixed my answer. – Chris Dellin Nov 15 '17 at 16:22
  • @rookiepig `t(s)` gets replaced by `int(substring1)`, `float(substring2)`, `strtobool(substring3)`, and `str(substring4)`, in order. – Blair Houghton Mar 31 '19 at 13:45
  • Thanks for actually addressing the type conversion and not just the string-splitting – dsz Jan 11 '21 at 10:38
38

Python doesn't have an sscanf equivalent built-in, and most of the time it actually makes a whole lot more sense to parse the input by working with the string directly, using regexps, or using a parsing tool.

Probably mostly useful for translating C, people have implemented sscanf, such as in this module: http://hkn.eecs.berkeley.edu/~dyoo/python/scanf/

In this particular case if you just want to split the data based on multiple split characters, re.split is really the right tool.

Mike Graham
  • 73,987
  • 14
  • 101
  • 130
  • 2
    i did say no re, but you justify it nicely – Matt Joiner Feb 01 '10 at 07:00
  • here's a py3k version of the linked implementation: https://gist.github.com/3875529 – Janus Troelsen Oct 11 '12 at 21:28
  • 1
    Its not built in, but there is a library for it here https://pypi.org/project/scanf/ – nimig18 Aug 19 '20 at 16:29
  • "most of the time it actually makes a whole lot more sense to parse the input by working with the string directly, using regexps, or using a parsing tool." This is a false statement. In most cases it makes sense to use (s-)scanf. I do not know about python2, but in python3 they have realized it already. – Student4K Aug 15 '21 at 15:04
24

You can split on a range of characters using the re module.

>>> import re
>>> r = re.compile('[ \t\n\r:]+')
>>> r.split("abc:def  ghi")
['abc', 'def', 'ghi']
Dietrich Epp
  • 205,541
  • 37
  • 345
  • 415
  • it is not a funny to deal with regex on textual float representation – ZAB Aug 31 '13 at 07:58
  • 1
    @ZAB: Nothing funny here. You use the regular expression to split fields, and then you use `float()` to parse it. – Dietrich Epp Aug 31 '13 at 16:20
  • for this speciefic problem, to parse /proc/net/*, this ugly trick will work though – ZAB Dec 03 '13 at 19:52
  • Or, even better, `r = re.compile(r'[\s:]+')`. (It's a good habit to put regular expressions in [raw strings](http://stackoverflow.com/q/2081640/952580) I think, even though it doesn't make any difference in this case.) – Beetle Jun 25 '15 at 09:39
16

You can parse with module re using named groups. It won't parse the substrings to their actual datatypes (e.g. int) but it's very convenient when parsing strings.

Given this sample line from /proc/net/tcp:

line="   0: 00000000:0203 00000000:0000 0A 00000000:00000000 00:00000000 00000000     0        0 335 1 c1674320 300 0 0 0"

An example mimicking your sscanf example with the variable could be:

import re
hex_digit_pattern = r"[\dA-Fa-f]"
pat = r"\d+: " + \
      r"(?P<local_addr>HEX+):(?P<local_port>HEX+) " + \
      r"(?P<rem_addr>HEX+):(?P<rem_port>HEX+) " + \
      r"HEX+ HEX+:HEX+ HEX+:HEX+ HEX+ +\d+ +\d+ " + \
      r"(?P<inode>\d+)"
pat = pat.replace("HEX", hex_digit_pattern)

values = re.search(pat, line).groupdict()

import pprint; pprint values
# prints:
# {'inode': '335',
#  'local_addr': '00000000',
#  'local_port': '0203',
#  'rem_addr': '00000000',
#  'rem_port': '0000'}
orip
  • 73,323
  • 21
  • 116
  • 148
2

There is an example in the official python docs about how to use sscanf from libc:

# import libc
from ctypes import CDLL
if(os.name=="nt"):
    libc = cdll.msvcrt 
else:
    # assuming Unix-like environment
    libc = cdll.LoadLibrary("libc.so.6")
    libc = CDLL("libc.so.6")  # alternative

# allocate vars
i = c_int()
f = c_float()
s = create_string_buffer(b'\000' * 32)

# parse with sscanf
libc.sscanf(b"1 3.14 Hello", "%d %f %s", byref(i), byref(f), s)

# read the parsed values
i.value  # 1
f.value  # 3.14
s.value # b'Hello'
Ryan M
  • 18,333
  • 31
  • 67
  • 74
eadmaster
  • 1,347
  • 13
  • 23
  • replace `from ctypes import CDLL` with `from ctypes import cdll, c_int, c_float, create_string_buffer, byref` or `from ctypes import *` – Dmytro Jun 27 '21 at 13:27
1

you can turn the ":" to space, and do the split.eg

>>> f=open("/proc/net/dev")
>>> for line in f:
...     line=line.replace(":"," ").split()
...     print len(line)

no regex needed (for this case)

ghostdog74
  • 327,991
  • 56
  • 259
  • 343
  • You'd still have to verify that the original string was correct - for example, "abc def ghi" would parse the same as "abc:def:ghi". This distinction may matter. – Kevin May 18 '15 at 17:15
1

You could install pandas and use pandas.read_fwf for fixed width format files. Example using /proc/net/arp:

In [230]: df = pandas.read_fwf("/proc/net/arp")

In [231]: print(df)
       IP address HW type Flags         HW address Mask Device
0   141.38.28.115     0x1   0x2  84:2b:2b:ad:e1:f4    *   eth0
1   141.38.28.203     0x1   0x2  c4:34:6b:5b:e4:7d    *   eth0
2   141.38.28.140     0x1   0x2  00:19:99:ce:00:19    *   eth0
3   141.38.28.202     0x1   0x2  90:1b:0e:14:a1:e3    *   eth0
4    141.38.28.17     0x1   0x2  90:1b:0e:1a:4b:41    *   eth0
5    141.38.28.60     0x1   0x2  00:19:99:cc:aa:58    *   eth0
6   141.38.28.233     0x1   0x2  90:1b:0e:8d:7a:c9    *   eth0
7    141.38.28.55     0x1   0x2  00:19:99:cc:ab:00    *   eth0
8   141.38.28.224     0x1   0x2  90:1b:0e:8d:7a:e2    *   eth0
9   141.38.28.148     0x1   0x0  4c:52:62:a8:08:2c    *   eth0
10  141.38.28.179     0x1   0x2  90:1b:0e:1a:4b:50    *   eth0

In [232]: df["HW address"]
Out[232]:
0     84:2b:2b:ad:e1:f4
1     c4:34:6b:5b:e4:7d
2     00:19:99:ce:00:19
3     90:1b:0e:14:a1:e3
4     90:1b:0e:1a:4b:41
5     00:19:99:cc:aa:58
6     90:1b:0e:8d:7a:c9
7     00:19:99:cc:ab:00
8     90:1b:0e:8d:7a:e2
9     4c:52:62:a8:08:2c
10    90:1b:0e:1a:4b:50

In [233]: df["HW address"][5]
Out[233]: '00:19:99:cc:aa:58'

By default it tries to figure out the format automagically, but there are options you can give for more explicit instructions (see documentation). There are also other IO routines in pandas that are powerful for other file formats.

gerrit
  • 24,025
  • 17
  • 97
  • 170
-2

If the separators are ':', you can split on ':', and then use x.strip() on the strings to get rid of any leading or trailing whitespace. int() will ignore the spaces.

Lennart Regebro
  • 167,292
  • 41
  • 224
  • 251