24

I wrote kind of a test suite which is heavily file intensive. After some time (2h) I get an IOError: [Errno 24] Too many open files: '/tmp/tmpxsqYPm'. I double checked all file handles whether I close them again. But the error still exists.

I tried to figure out the number of allowed file descriptors using resource.RLIMIT_NOFILE and the number of currently opened file desciptors:

def get_open_fds():

    fds = []
    for fd in range(3,resource.RLIMIT_NOFILE):
            try:
                    flags = fcntl.fcntl(fd, fcntl.F_GETFD)
            except IOError:
                    continue

            fds.append(fd)

    return fds

So if I run the following test:

print get_open_fds()
for i in range(0,100):
    f = open("/tmp/test_%i" % i, "w")
    f.write("test")
    print get_open_fds()

I get this output:

[]
/tmp/test_0
[3]
/tmp/test_1
[4]
/tmp/test_2
[3]
/tmp/test_3
[4]
/tmp/test_4
[3]
/tmp/test_5
[4] ...

That's strange, I expected an increasing number of opened file descriptors. Is my script correct?

I'm using python's logger and subprocess. Could that be the reason for my fd leak?

Thanks, Daniel

Paul D. Waite
  • 96,640
  • 56
  • 199
  • 270
dmorlock
  • 1,993
  • 4
  • 18
  • 22

3 Answers3

14

The corrected code is:

import resource
import fcntl
import os

def get_open_fds():
    fds = []
    soft, hard = resource.getrlimit(resource.RLIMIT_NOFILE)
    for fd in range(0, soft):
        try:
            flags = fcntl.fcntl(fd, fcntl.F_GETFD)
        except IOError:
            continue
        fds.append(fd)
    return fds

def get_file_names_from_file_number(fds):
    names = []
    for fd in fds:
        names.append(os.readlink('/proc/self/fd/%d' % fd))
    return names

fds = get_open_fds()
print get_file_names_from_file_number(fds)
blueFast
  • 41,341
  • 63
  • 198
  • 344
  • 4
    It would be more helpful to say what is wrong with the original. Not everyone wants to play 'spot the difference'. – Ian Goldby Mar 26 '18 at 09:17
14

Your test script overwrites f each iteration, which means that the file will get closed each time. Both logging to files and subprocess with pipes use up descriptors, which can lead to exhaustion.

Ignacio Vazquez-Abrams
  • 776,304
  • 153
  • 1,341
  • 1,358
  • my fault, thanks for the hint! It seems that get_open_fds() does its job. But resource.RLIMIT_NOFILE = 7 and my the error occurs after I opened 1024 files. Anyways, I know how to debug my scripts - thanks so far!!! – dmorlock Dec 08 '10 at 10:42
  • 1
    @Informant can you post the fix too? I have the same problem but don't know how to fix it. Thanx. – AliBZ Jun 27 '13 at 22:40
10

resource.RLIMIT_NOFILE is indeed 7, but that's an index into resource.getrlimit(), not the limit itself... resource.getrlimit(resource.RLIMIT_NOFILE) is what you want your top range() to be

Peter Buckner
  • 109
  • 2