logilab-common #8796 support setuptools namespace packages [validation pending]

Comments

• 2009/03/30 13:24, written by anon

  The minimal sample below creates this output ("source lint_ns.sh"), note that the code itself works (prints the paste module's repr), but pylint can't see said module:
  

  <module 'paste.script.templates' from '~/tmp/lint_ns/lib/python2.5/site-packages/PasteScript-1.7.3-py2.5.egg/paste/script/templates.pyc'>
  pylint 0.18.0,
  astng 0.19.0, common 0.39.0
  Python 2.5.2 (r252:60911, Nov 14 2008, 19:46:32)
  [GCC 4.3.2]
  ************* Module paste.lint_ns
  C: 1: Missing docstring
  E: 1: No name 'script' in module 'paste'
  F: 1: Unable to import 'paste.script' (No module named script)
  

  I could not reproduce the "TypeError: '_Yes' object is unindexable" I get with a more complex project, let's hope that disappears when all modules can be found.
  

  ==> lint_ns.sh <==
  # must be sourced!
  if test ! -d bin; then
  virtualenv2.5 --no-site-packages .
  . bin/activate
  PYTHONPATH=. bin/python ./setup.py develop -U
  fi
  bin/python -m paste.lint_ns
  bin/pylint --version
  bin/pylint -rn paste
  

  ==> setup.py <==
  from setuptools import setup
  

  setup(
  name = "lint_ns",
  version = "0.1",
  packages = ["paste.lint_ns"],
  namespace_packages = ["paste"],
  

  install_requires = [
  "setuptools==0.6c9",
  "PasteScript>=1.6.2",
  "pylint==0.18.0",
  #"logilab.pylintinstaller==0.1.1",
  ],
  )
  

  ==> paste/__init__.py <==
  """Namespace package."""
  __import__('pkg_resources').declare_namespace(__name__)
  

  ==> paste/lint_ns/__init__.py <==
  from paste.script import templates
  if __name__ == "__main__":
  print repr(templates)
  
  • 2009/03/30 13:29, written by sthenault

    huum that's the one I was mainly interested in :(
    
  • 2010/01/06 10:46, written by kevingill

    I experienced this problem using the zope system. Zope uses large numbers or namespace eggs.

    I work around it by modifying _module_file in modutils.py

    before the line :

    
    

        while modpath:
    

    
    

    insert:

    
    

        try:
            _, mp_filename, mp_desc = find_module(os.sep.join(modpath), path)
        except ImportError:
            if checkeggs:
                return _search_zip(modpath, pic)[:2]
            raise
        mtype = mp_desc[2]
        return mtype, mp_filename
    
    

    
    

    This returns the correct name for imports from namespace eggs. It may mess up .zip eggs. It lowers the quality of the import error message.
  • 2011/01/21 13:04, written by anon

    Has there been any progress or workaround for this issue?
    
• 2011/01/22 15:13, written by dsully

  Also looking for an update here.
  

  It looks like the code from kevingill above has been added to modutils.py already running logilab_common 0.53.0, but it doesn't fix the E0611 errors from Pylint.
  

  Thanks
  
• 2011/06/14 03:37, written by anon

  I have a few namespaced modules as well. as it stands i cannot use pylint because of the 300+ false positives it gives due to this error. Any updates?
  
• 2011/11/03 10:53, written by anon

  I have a similar problem with pkgutil.extend_path. I added code to reproduce it to GitHub
• 2012/04/26 15:53, written by jd

  This is preliminary because it only supports top-level namespace packages, but generalizing won't be too hard.

  In logilab.common.modutils, import pkg_resources. Then in _module_file() insert the following lines just after while modpath:

  while modpath:

  if modpath[0] in pkg_resources._namespace_packages and len(modpath) > 1:

  module = sys.modules[modpath.pop(0)] path = module.__path__

  With this I don't errors related to namespace packages anymore.
• 2012/12/04 15:49, written by rbarrois

  A common issue appears when using pylint on one namespace-enabled package using data from other parts of the same namespace.

  Typical issue:

  • foo.bar and foo.baz are two packages belonging to the "foo" namespace
  • In foo.baz.blah, there is "from . import mymod"

  The above patch will try to import "foo.baz.mymod" from the site-packages installed foo/ folder (brought in by foo.bar) instead of looking for foo.baz.blah.

  I have written a patch that works fine on our setup:

  diff --git a/modutils.py b/modutils.py
  index 2cae005..b2e8c6b 100644
  --- a/modutils.py
  +++ b/modutils.py
  @@ -35,6 +35,7 @@ from os.path import splitext, join, abspath, isdir, dirname, exists, basename
   from imp import find_module, load_module, C_BUILTIN, PY_COMPILED, PKG_DIRECTORY
   from distutils.sysconfig import get_config_var, get_python_lib, get_python_version
   from distutils.errors import DistutilsPlatformError
  +import pkg_resources
  
   try:
       import zipimport
  @@ -545,13 +546,40 @@ def _file_from_modpath(modpath, path=None, context=None):
       documentation for more information
       """
       assert len(modpath) > 0
  +
  +    # List of paths to test, as (modpath, path).
  +    # The 'modpath' is needed for namespaced package handling.
  +    paths = []
  +
       if context is not None:
  +        paths.append((modpath, [context]))
  +
  +    paths.append((modpath, path))
  +
  +    if len(modpath) > 1 and modpath[0] in pkg_resources._namespace_packages:
  +        # Looks like a namespaced package, try system-wide package too
  +        module = modpath[0]
  +        module = sys.modules[module]
  +        namespaced_path = module.__path__
  +
  +        # We need to lookup modpath[1:] within that module
  +        paths.append((modpath[1:], namespaced_path))
  +
  +    # Iterate over all (modpath, path) combination until a valid one is found
  +    mtype = mp_filename = None
  +    found = False
  +    for _modpath, _path in paths:
           try:
  -            mtype, mp_filename = _module_file(modpath, [context])
  +            mtype, mp_filename = _module_file(_modpath, _path)
  +            found = True
  +            break
           except ImportError:
  -            mtype, mp_filename = _module_file(modpath, path)
  -    else:
  -        mtype, mp_filename = _module_file(modpath, path)
  +            pass
  +
  +    if not found:
  +        raise ImportError("No module named %s in %s" % (
  +            '.'.join(modpath), [p for _modpath, p in paths]))
  +
       if mtype == PY_COMPILED:
           try:
               return get_source_file(mp_filename)
  @@ -604,6 +632,10 @@ def _module_file(modpath, path=None):
           checkeggs = True
       except AttributeError:
           checkeggs = False
  +
  +    # We'll alter modpath, so let's make a copy
  +    modpath = list(modpath)
  +
       imported = []
       while modpath:
           try:
  
  • 2012/12/20 21:09, written by jd

    Unfortunately your patch doesn't work for me (logilab-common==0.58.3): namespace packages aren't found, just like with no patch.
    

    This bug deserves some love from the package maintainers.
    
    • 2013/01/21 23:01, written by rbarrois

      Would you have an example of such setup? I think my patch would fail with second-level namespaces (i.e where "foo.bar" is the namespace).
      
• 2013/01/21 22:04, written by anon

  I am also working with a package hierarchy that is heavily namespace'd. I too have many error false positives on modules that Pylint can't import. I use Pylint and flymake to make my Emacs a Python IDE; the false errors reported by failure to import namespace'd packages masks bigger errors frequently. We catch them later, usually, but we could be catching them earlier if Pylint would properly import our packages.
  
• 2013/01/31 09:43, written by anon

  I solved the problem when using pkgutil.extend_path.

  It works for me with pylint 0.25.1, astng 0.23.1, common 0.57.1, Python 2.7.3

  May be someone may test ist with other versions.

  ReST / HTML errors:System Message: ERROR/3 (&lt;string&gt; , line 11)</p>

  Error in "highlight" directive: maximum 1 argument(s) allowed, 316 supplied.

  .. highlight:: python
     588a589
     >     path = (path is None and sys.path or path)
     592,594c593,594
     <         _path = (path is None and sys.path or path)
     <         for __path in _path:
     <             if not __path in pic:
     ---
     >         for _path in path:
     >             if not _path in pic:
     596c596
     <                     pic[__path] = zipimport.zipimporter(__path)
     ---
     >                     pic[_path] = zipimport.zipimporter(_path)
     598c598
     <                     pic[__path] = None
     ---
     >                     pic[_path] = None
     604,606c604,611
     <         try:
     <             _, mp_filename, mp_desc = find_module(modpath[0], path)
     <         except ImportError:
     ---
     >         mp_filenames = []
     >         for _path in path:
     >             try:
     >                 _, mp_filename, mp_desc = find_module(modpath[0], [_path])
     >                 mp_filenames.append((mp_filename, mp_desc))
     >             except ImportError:
     >                 pass
     >         if not mp_filenames:
     609,624c614,629
     <             raise
     <         else:
     <             if checkeggs:
     <                 fullabspath = [abspath(x) for x in _path]
     <                 try:
     <                     pathindex = fullabspath.index(dirname(abspath(mp_filename)))
     <                     emtype, emp_filename, zippath = _search_zip(modpath, pic)
     <                     if pathindex > _path.index(zippath):
     <                         # an egg takes priority
     <                         return emtype, emp_filename
     <                 except ValueError:
     <                     # XXX not in _path
     <                     pass
     <                 except ImportError:
     <                     pass
     <                 checkeggs = False
     ---
     >             raise ImportError('No module %s in %s' % ('.'.join(modpath),
     >                                                       '.'.join(imported)))
     >         if checkeggs:
     >             fullabspath = [abspath(x) for x in path]
     >             try:
     >                 pathindex = fullabspath.index(dirname(abspath(mp_filename)))
     >                 emtype, emp_filename, zippath = _search_zip(modpath, pic)
     >                 if pathindex > _path.index(zippath):
     >                     # an egg takes priority
     >                     return emtype, emp_filename
     >             except ValueError:
     >                 # XXX not in _path
     >                 pass
     >             except ImportError:
     >                 pass
     >             checkeggs = False
     626d630
     <         mtype = mp_desc[2]
     628,631c632,639
     <             if mtype != PKG_DIRECTORY:
     <                 raise ImportError('No module %s in %s' % ('.'.join(modpath),
     <                                                           '.'.join(imported)))
     <             path = [mp_filename]
     ---
     >             for mtype in [p[1][2] for p in mp_filenames]:
     >                 if mtype != PKG_DIRECTORY:
     >                     raise ImportError('No module %s in %s' % ('.'.join(modpath),
     >                                                               '.'.join(imported)))
     >             path = [p[0] for p in mp_filenames]
     >         else:
     >             mp_filename, mp_desc = mp_filenames[0]
     >             mtype = mp_desc[2]