logilab-astng #62295 Getting "OSError: Too many open files" when running pylint repeatedly on huge number of python scripts [validation pending]

The issue is that running pylint repeatedly on about 500 files, I get OSError: Too many open files.

I use pylint like such:

import os
import sys
from pylint import lint
args = sys.argv[1:]
for root, dirs, files in os.walk(args[0]):
for file in files:
if not file.endswith('.py'): continue
try:
lint.Run(args)
except OSError, e:
raise
except SystemExit, e:
returncode = e.code

I installed pylint-0.23.0.tar.gz and ran it using python 2.6.4. I found the workaround by increasing the max file descriptors with ulimit to more than 4000.

priorityimportant
typebug
done in0.23.1
load0.100
load left0.000
closed by#df946b92d8fe closes #62295: avoid "OSError: Too many open files"

Workflow history

from state (3)to statecommentdateUser
donevalidation pending2011/12/08 15:54 UTCsthenault
in-progressdone2011/12/08 13:33 UTC 
openin-progress2011/10/26 07:37 UTC 

Comments

  • 2011/02/15 23:29, written by anon

    This is the OS info:

    OSX 10.5: 9.8.0 Darwin Kernel Version 9.8.0: Wed Jul 15 16:55:01 PDT 2009; root:xnu-1228.15.4~1/RELEASE_I386 i386

  • 2011/05/11 10:01, written by anon

    Same problem for 0.21.0 and 0.23.0 under Windows XP pro x64 SP2 using Python 2.6.2 (x86).
    It doesn't seem to have anything to do with pylint directly, the problem is somewhere in common-0.55.2 or astng-0.21.1. Running pylint using the further versions common-0.50.2 and astng-0.20.1 don't have this problem.

  • 2011/08/10 21:09, written by anon

    Same here, for
    logilab-astng==0.22.0
    logilab-common==0.56.1
    pylint==0.24.0

  • 2011/10/25 18:51, written by patrick
    Here is the patch that we are using with pylint 0.25.0. Our source tree has over 1000 .py files, and this change works for us. 
    Index: lint.py
===================================================================
--- lint.py     (revision 3500)
+++ lint.py     (working copy)
@@ -494,6 +494,7 @@
             # if it's actually a c extension)
             self.current_file = astng.file
             self.check_astng_module(astng, walker, rawcheckers)
+            astng.file_stream.close()
         # notify global end
         self.set_current_module('')
         self.stats['statement'] = walker.nbstatements
  • 2011/11/21 18:02, written by anon

    Having this same problem. Applied the patch from Patrick to 0.25.0 and it worked like a charm.

  • 2011/11/21 21:45, written by anon

    Thanks Patrick. That's a useful one-liner. :-)

  • 2012/12/03 19:33, written by andres.riancho

    Applied Patrick's fix to my pylint installation [0] and it also fixed the too many open files on my side. Given that his patch is from Oct 2011, shouldn't this be fixed in ubuntu already? Should I use pylint from pip instead of apt?

    [0] pylint 0.25.0,
    astng 0.23.0, common 0.57.1
    Python 2.7.3 (default, Aug 1 2012, 05:14:39)
    [GCC 4.6.3]

add comment