Importing types from namespace packages emits false positives without `from ... import ...`

[chrisphilip322]

Importing type hints from namespace packages only works when using the from ... import ... form.

Using the --namespace-packages flag works properly when the typedpkg_namespace-alpha is not installed but mypy is executed from the directory containing the package contents (as expected) but only for imports using the from ... import ... style. import ... does not work as expected. Likewise the same is true without the --namespace-packages flag if typedpkg_namespace-alpha is installed.

Specifically, the type hints are correctly imported but a false positive error message is still emitted of Cannot find module named 'typedpkg_namespace'.

Repro steps

• Virtual env state

$ pip list
Package         Version
--------------- --------------------------------------------------
mypy            0.640+dev.e12be3ba9e6be3e9242e3a81ffaa9b3a136876fe
mypy-extensions 0.4.1
pip             18.1
setuptools      40.4.3
typed-ast       1.1.0

• Repro file structure

$ find . \( -path ./.mypy_cache -o -path ./venv \) -prune -o -print
.
./typedpkg_namespace
./typedpkg_namespace/alpha
./typedpkg_namespace/alpha/__init__.py
./typedpkg_namespace/alpha/py.typed
./typedpkg_namespace/alpha/alpha_module.py
./not_from_imp.py
./from_imp.py
./setup.py

• Repro files
  • setup.py

  setup(
      name='typedpkg_namespace.alpha',
      version='1.0.0',
      packages=find_namespace_packages(),
      namespace_packages=['typedpkg_namespace'],
      zip_safe=False,
      package_data={'typedpkg_namespace.alpha': ['py.typed']}
  )
  

  • typedpkg_namespace/alpha/alpha_module.py

  def alpha_func(a: bool) -> bool:
      return not a
  

  • from_imp.py

  from typedpkg_namespace.alpha.alpha_module import alpha_func
  alpha_func(False)
  alpha_func(2)
  

  • not_from_imp.py

  import typedpkg_namespace.alpha.alpha_module as am; alpha_func = am.alpha_func
  alpha_func(False)
  alpha_func(2)
  

  • typedpkg_namespace/alpha/__init__.py and typedpkg_namespace/alpha/py.typed are empty

Repro Example

• using --namespace-packages

$ mypy from_imp.py  --namespace-packages
from_imp.py:3: error: Argument 1 to "alpha_func" has incompatible type "int"; expected "bool"

$ mypy not_from_imp.py --namespace-packages
not_from_imp.py:1: error: Cannot find module named 'typedpkg_namespace'
not_from_imp.py:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
not_from_imp.py:3: error: Argument 1 has incompatible type "int"; expected "bool"

• using installed package

$ pip install .
Looking in indexes: http://artifactory.factset.com/artifactory/api/pypi/python/simple/
Processing /home/user/cphilip/side_projects/test_namespace/typedpkg_namespace-alpha
Installing collected packages: typedpkg-namespace.alpha
  Running setup.py install for typedpkg-namespace.alpha ... done
Successfully installed typedpkg-namespace.alpha-1.0.0

$ pip list
Package                  Version
------------------------ --------------------------------------------------
mypy                     0.640+dev.e12be3ba9e6be3e9242e3a81ffaa9b3a136876fe
mypy-extensions          0.4.1
pip                      18.1
setuptools               40.4.3
typed-ast                1.1.0
typedpkg-namespace.alpha 1.0.0

$ mypy from_imp.py
from_imp.py:3: error: Argument 1 to "alpha_func" has incompatible type "int"; expected "bool"

$ mypy not_from_imp.py
not_from_imp.py:1: error: Cannot find module named 'typedpkg_namespace'
not_from_imp.py:1: note: (Perhaps setting MYPYPATH or using the "--ignore-missing-imports" flag would help)
not_from_imp.py:3: error: Argument 1 has incompatible type "int"; expected "bool"

[gvanrossum]

Yup, I can repro this. There must be a corner case that I don't handle correctly yet.

[gvanrossum]

Can you try with #5762?

[chrisphilip322]

I tested #5762 locally at it looks like it fixes the first part of this issue, using --namespace-packages when the typedpkg_namespace source directory is in the current working directory.

When the package is installed, and the current working directory does not contain the package source directory, I get the same false positive error with or without --namespace-packages.

Too be clear, I get the same error when the package is installed with a file structure like this

$ find . \( -path *.mypy_cache -o -path ./typedpkg_namespace-alpha/venv \) -prune -o -print
.
./not_from_imp.py
./typedpkg_namespace-alpha
./typedpkg_namespace-alpha/typedpkg_namespace
./typedpkg_namespace-alpha/typedpkg_namespace/alpha
./typedpkg_namespace-alpha/typedpkg_namespace/alpha/__init__.py
./typedpkg_namespace-alpha/typedpkg_namespace/alpha/py.typed
./typedpkg_namespace-alpha/typedpkg_namespace/alpha/alpha_module.py
./typedpkg_namespace-alpha/not_from_imp.py
./typedpkg_namespace-alpha/from_imp.py
./typedpkg_namespace-alpha/setup.py
./from_imp.py

[gvanrossum]

So this seems specific to when it's installed. Could you make those files available as a throwaway GitHub repo so I can test with it? (I'm too lazy to try and copy/paste the entire setup from your initial message.)

[chrisphilip322]

I made https://github.com/macbeth322/mypy-namespace-bug-repro for reproducing the issue. I also noticed that if you do import .. as .. it works but with false positive output, but without the as .. it actually doesn't find the type hints.

I was looking through the code and I believe it may be related to https://github.com/python/mypy/blob/master/mypy/build.py#L577 and the fact that when you do import foo.bar.baz, foo, foo.bar and foo.bar.baz all become available in the current scope. In the case where you have foo/bar/py.typed the foo package is never going to have type hints. In most cases foo will be a namespace package so there shouldn't be any code there to have type hints so there might be a workaround to check if a import name is missing type hints and it is a namespace package.

[gvanrossum]

Thanks for making the repro! I'm still banging my head against the wall but you're definitely right that it's got something to do with the ancestors. Unfortunately I don't think it's as simple as dropping that code (or even dropping it when the ancestor is a toplevel namespace), because later references to e.g. foo.bar.baz are going to expect foo to be a module object, and then do a getattr() on that to get baz, and so on. I'll try to continue the research tomorrow, but if you have time to experiment with some careful surgery of the code don't let me stop you!