setuptoos introduce the notion of 'namespace' package allowing to have sub-parts of a same package splitted into several eggs (eg what's used to build proper eggs for logilab'subpackages...) this should be understood by pylint/astng? | |
| priority | important |
|---|---|
| type | enhancement |
| appeared in | <not specified> |
| done in | <not specified> |
| load | 2.000 |
| load left | 2.000 |
| closed by | <not specified> |
[register] [login]
#8774 iterator / generator / next method
Comments
The minimal sample below creates this output ("source lint_ns.sh"), note that the code itself works (prints the paste module's repr), but pylint can't see said module:
<module 'paste.script.templates' from '~/tmp/lint_ns/lib/python2.5/site-packages/PasteScript-1.7.3-py2.5.egg/paste/script/templates.pyc'>
pylint 0.18.0,
astng 0.19.0, common 0.39.0
Python 2.5.2 (r252:60911, Nov 14 2008, 19:46:32)
[GCC 4.3.2]
************* Module paste.lint_ns
C: 1: Missing docstring
E: 1: No name 'script' in module 'paste'
F: 1: Unable to import 'paste.script' (No module named script)
I could not reproduce the "TypeError: '_Yes' object is unindexable" I get with a more complex project, let's hope that disappears when all modules can be found.
==> lint_ns.sh <==
# must be sourced!
if test ! -d bin; then
virtualenv2.5 --no-site-packages .
. bin/activate
PYTHONPATH=. bin/python ./setup.py develop -U
fi
bin/python -m paste.lint_ns
bin/pylint --version
bin/pylint -rn paste
==> setup.py <==
from setuptools import setup
setup(
name = "lint_ns",
version = "0.1",
packages = ["paste.lint_ns"],
namespace_packages = ["paste"],
install_requires = [
"setuptools==0.6c9",
"PasteScript>=1.6.2",
"pylint==0.18.0",
#"logilab.pylintinstaller==0.1.1",
],
)
==> paste/__init__.py <==
"""Namespace package."""
__import__('pkg_resources').declare_namespace(__name__)
==> paste/lint_ns/__init__.py <==
from paste.script import templates
if __name__ == "__main__":
print repr(templates)
huum that's the one I was mainly interested in :(
I experienced this problem using the zope system. Zope uses large numbers or namespace eggs.
I work around it by modifying _module_file in modutils.py
before the line :
while modpath:insert:
try: _, mp_filename, mp_desc = find_module(os.sep.join(modpath), path) except ImportError: if checkeggs: return _search_zip(modpath, pic)[:2] raise mtype = mp_desc[2] return mtype, mp_filenameThis returns the correct name for imports from namespace eggs. It may mess up .zip eggs. It lowers the quality of the import error message.
Has there been any progress or workaround for this issue?
Also looking for an update here.
It looks like the code from kevingill above has been added to modutils.py already running logilab_common 0.53.0, but it doesn't fix the E0611 errors from Pylint.
Thanks
I have a few namespaced modules as well. as it stands i cannot use pylint because of the 300+ false positives it gives due to this error. Any updates?
I have a similar problem with pkgutil.extend_path. I added code to reproduce it to GitHub
This is preliminary because it only supports top-level namespace packages, but generalizing won't be too hard.
In logilab.common.modutils, import pkg_resources. Then in _module_file() insert the following lines just after while modpath:
With this I don't errors related to namespace packages anymore.
A common issue appears when using pylint on one namespace-enabled package using data from other parts of the same namespace.
Typical issue:
The above patch will try to import "foo.baz.mymod" from the site-packages installed foo/ folder (brought in by foo.bar) instead of looking for foo.baz.blah.
I have written a patch that works fine on our setup:
diff --git a/modutils.py b/modutils.py index 2cae005..b2e8c6b 100644 --- a/modutils.py +++ b/modutils.py @@ -35,6 +35,7 @@ from os.path import splitext, join, abspath, isdir, dirname, exists, basename from imp import find_module, load_module, C_BUILTIN, PY_COMPILED, PKG_DIRECTORY from distutils.sysconfig import get_config_var, get_python_lib, get_python_version from distutils.errors import DistutilsPlatformError +import pkg_resources try: import zipimport @@ -545,13 +546,40 @@ def _file_from_modpath(modpath, path=None, context=None): documentation for more information """ assert len(modpath) > 0 + + # List of paths to test, as (modpath, path). + # The 'modpath' is needed for namespaced package handling. + paths = [] + if context is not None: + paths.append((modpath, [context])) + + paths.append((modpath, path)) + + if len(modpath) > 1 and modpath[0] in pkg_resources._namespace_packages: + # Looks like a namespaced package, try system-wide package too + module = modpath[0] + module = sys.modules[module] + namespaced_path = module.__path__ + + # We need to lookup modpath[1:] within that module + paths.append((modpath[1:], namespaced_path)) + + # Iterate over all (modpath, path) combination until a valid one is found + mtype = mp_filename = None + found = False + for _modpath, _path in paths: try: - mtype, mp_filename = _module_file(modpath, [context]) + mtype, mp_filename = _module_file(_modpath, _path) + found = True + break except ImportError: - mtype, mp_filename = _module_file(modpath, path) - else: - mtype, mp_filename = _module_file(modpath, path) + pass + + if not found: + raise ImportError("No module named %s in %s" % ( + '.'.join(modpath), [p for _modpath, p in paths])) + if mtype == PY_COMPILED: try: return get_source_file(mp_filename) @@ -604,6 +632,10 @@ def _module_file(modpath, path=None): checkeggs = True except AttributeError: checkeggs = False + + # We'll alter modpath, so let's make a copy + modpath = list(modpath) + imported = [] while modpath: try:Unfortunately your patch doesn't work for me (logilab-common==0.58.3): namespace packages aren't found, just like with no patch.
This bug deserves some love from the package maintainers.
Would you have an example of such setup? I think my patch would fail with second-level namespaces (i.e where "foo.bar" is the namespace).
I am also working with a package hierarchy that is heavily namespace'd. I too have many error false positives on modules that Pylint can't import. I use Pylint and flymake to make my Emacs a Python IDE; the false errors reported by failure to import namespace'd packages masks bigger errors frequently. We catch them later, usually, but we could be catching them earlier if Pylint would properly import our packages.
I solved the problem when using pkgutil.extend_path.
It works for me with pylint 0.25.1, astng 0.23.1, common 0.57.1, Python 2.7.3
May be someone may test ist with other versions.
Unknown directive type "highlight".
.. highlight:: python 588a589 > path = (path is None and sys.path or path) 592,594c593,594 < _path = (path is None and sys.path or path) < for __path in _path: < if not __path in pic: --- > for _path in path: > if not _path in pic: 596c596 < pic[__path] = zipimport.zipimporter(__path) --- > pic[_path] = zipimport.zipimporter(_path) 598c598 < pic[__path] = None --- > pic[_path] = None 604,606c604,611 < try: < _, mp_filename, mp_desc = find_module(modpath[0], path) < except ImportError: --- > mp_filenames = [] > for _path in path: > try: > _, mp_filename, mp_desc = find_module(modpath[0], [_path]) > mp_filenames.append((mp_filename, mp_desc)) > except ImportError: > pass > if not mp_filenames: 609,624c614,629 < raise < else: < if checkeggs: < fullabspath = [abspath(x) for x in _path] < try: < pathindex = fullabspath.index(dirname(abspath(mp_filename))) < emtype, emp_filename, zippath = _search_zip(modpath, pic) < if pathindex > _path.index(zippath): < # an egg takes priority < return emtype, emp_filename < except ValueError: < # XXX not in _path < pass < except ImportError: < pass < checkeggs = False --- > raise ImportError('No module %s in %s' % ('.'.join(modpath), > '.'.join(imported))) > if checkeggs: > fullabspath = [abspath(x) for x in path] > try: > pathindex = fullabspath.index(dirname(abspath(mp_filename))) > emtype, emp_filename, zippath = _search_zip(modpath, pic) > if pathindex > _path.index(zippath): > # an egg takes priority > return emtype, emp_filename > except ValueError: > # XXX not in _path > pass > except ImportError: > pass > checkeggs = False 626d630 < mtype = mp_desc[2] 628,631c632,639 < if mtype != PKG_DIRECTORY: < raise ImportError('No module %s in %s' % ('.'.join(modpath), < '.'.join(imported))) < path = [mp_filename] --- > for mtype in [p[1][2] for p in mp_filenames]: > if mtype != PKG_DIRECTORY: > raise ImportError('No module %s in %s' % ('.'.join(modpath), > '.'.join(imported))) > path = [p[0] for p in mp_filenames] > else: > mp_filename, mp_desc = mp_filenames[0] > mtype = mp_desc[2]