
## Summary This PR adds support for the `__all__` module variable. Reference spec: https://typing.python.org/en/latest/spec/distributing.html#library-interface-public-and-private-symbols This PR adds a new `dunder_all_names` query that returns a set of `Name`s defined in the `__all__` variable of the given `File`. The query works by implementing the `StatementVisitor` and collects all the names by recognizing the supported idioms as mentioned in the spec. Any idiom that's not recognized are ignored. The current implementation is minimum to what's required for us to remove all the false positives that this is causing. Refer to the "Follow-ups" section below to see what we can do next. I'll a open separate issue to keep track of them. Closes: astral-sh/ty#106 Closes: astral-sh/ty#199 ### Follow-ups * Diagnostics: * Add warning diagnostics for unrecognized `__all__` idioms, `__all__` containing non-string element * Add an error diagnostic for elements that are present in `__all__` but not defined in the module. This could lead to runtime error * Maybe we should return `<type>` instead of `Unknown | <type>` for `module.__all__`. For example: https://playknot.ruff.rs/2a6fe5d7-4e16-45b1-8ec3-d79f2d4ca894 * Mark a symbol that's mentioned in `__all__` as used otherwise it could raise (possibly in the future) "unused-name" diagnostic Supporting diagnostics will require that we update the return type of the query to be something other than `Option<FxHashSet<Name>>`, something that behaves like a result and provides a way to check whether a name exists in `__all__`, loop over elements in `__all__`, loop over the invalid elements, etc. ## Ecosystem analysis The following are the maximum amount of diagnostics **removed** in the ecosystem: * "Type <module '...'> has no attribute ..." * `collections.abc` - 14 * `numpy` - 35534 * `numpy.ma` - 296 * `numpy.char` - 37 * `numpy.testing` - 175 * `hashlib` - 311 * `scipy.fft` - 2 * `scipy.stats` - 38 * "Module '...' has no member ..." * `collections.abc` - 85 * `numpy` - 508 * `numpy.testing` - 741 * `hashlib` - 36 * `scipy.stats` - 68 * `scipy.interpolate` - 7 * `scipy.signal` - 5 The following modules have dynamic `__all__` definition, so `ty` assumes that `__all__` doesn't exists in that module: * `scipy.stats` (95a5d6ea8b/scipy/stats/__init__.py (L665)
) * `scipy.interpolate` (95a5d6ea8b/scipy/interpolate/__init__.py (L221)
) * `scipy.signal` (indirectly via95a5d6ea8b/scipy/signal/_signal_api.py (L30)
) * `numpy.testing` (de784cd6ee/numpy/testing/__init__.py (L16-L18)
) ~There's this one category of **false positives** that have been added:~ Fixed the false positives by also ignoring `__all__` from a module that uses unrecognized idioms. <details><summary>Details about the false postivie:</summary> <p> The `scipy.stats` module has dynamic `__all__` and it imports a bunch of symbols via star imports. Some of those modules have a mix of valid and invalid `__all__` idioms. For example, in95a5d6ea8b/scipy/stats/distributions.py (L18-L24)
, 2 out of 4 `__all__` idioms are invalid but currently `ty` recognizes two of them and says that the module has a `__all__` with 5 values. This leads to around **2055** newly added false positives of the form: ``` Type <module 'scipy.stats'> has no attribute ... ``` I think the fix here is to completely ignore `__all__`, not only if there are invalid elements in it, but also if there are unrecognized idioms used in the module. </p> </details> ## Test Plan Add a bunch of test cases using the new `ty_extensions.dunder_all_names` function to extract a module's `__all__` names. Update various test cases to remove false positives around `*` imports and re-export convention. Add new test cases for named import behavior as `*` imports covers all of it already (thanks Alex!).
6.7 KiB
Import conventions
This document describes the conventions for importing symbols.
Reference:
Builtins scope
When looking up for a name, ty will fallback to using the builtins scope if the name is not found in
the global scope. The builtins.pyi
file, that will be used to resolve any symbol in the builtins
scope, contains multiple symbols from other modules (e.g., typing
) but those are not re-exported.
# These symbols are being imported in `builtins.pyi` but shouldn't be considered as being
# available in the builtins scope.
# error: "Name `Literal` used when not defined"
reveal_type(Literal) # revealed: Unknown
# error: "Name `sys` used when not defined"
reveal_type(sys) # revealed: Unknown
Builtins import
Similarly, trying to import the symbols from the builtins module which aren't re-exported should also raise an error.
# error: "Module `builtins` has no member `Literal`"
# error: "Module `builtins` has no member `sys`"
from builtins import Literal, sys
reveal_type(Literal) # revealed: Unknown
reveal_type(sys) # revealed: Unknown
# error: "Module `math` has no member `Iterable`"
from math import Iterable
reveal_type(Iterable) # revealed: Unknown
Re-exported symbols in stub files
When a symbol is re-exported, importing it should not raise an error. This tests both import ...
and from ... import ...
forms.
Note: Submodule imports in import ...
form doesn't work because it's a syntax error. For example,
in import os.path as os.path
the os.path
is not a valid identifier.
from b import Any, Literal, foo
reveal_type(Any) # revealed: typing.Any
reveal_type(Literal) # revealed: typing.Literal
reveal_type(foo) # revealed: <module 'foo'>
b.pyi
:
import foo as foo
from typing import Any as Any, Literal as Literal
foo.py
:
Non-exported symbols in stub files
Here, none of the symbols are being re-exported in the stub file.
# error: 15 [unresolved-import] "Module `b` has no member `foo`"
# error: 20 [unresolved-import] "Module `b` has no member `Any`"
# error: 25 [unresolved-import] "Module `b` has no member `Literal`"
from b import foo, Any, Literal
reveal_type(Any) # revealed: Unknown
reveal_type(Literal) # revealed: Unknown
reveal_type(foo) # revealed: Unknown
b.pyi
:
import foo
from typing import Any, Literal
foo.pyi
:
Nested non-exports
Here, a chain of modules all don't re-export an import.
# error: "Module `a` has no member `Any`"
from a import Any
reveal_type(Any) # revealed: Unknown
a.pyi
:
# error: "Module `b` has no member `Any`"
from b import Any
reveal_type(Any) # revealed: Unknown
b.pyi
:
# error: "Module `c` has no member `Any`"
from c import Any
reveal_type(Any) # revealed: Unknown
c.pyi
:
from typing import Any
reveal_type(Any) # revealed: typing.Any
Nested mixed re-export and not
But, if the symbol is being re-exported explicitly in one of the modules in the chain, it should not raise an error at that step in the chain.
# error: "Module `a` has no member `Any`"
from a import Any
reveal_type(Any) # revealed: Unknown
a.pyi
:
from b import Any
reveal_type(Any) # revealed: Unknown
b.pyi
:
# error: "Module `c` has no member `Any`"
from c import Any as Any
reveal_type(Any) # revealed: Unknown
c.pyi
:
from typing import Any
reveal_type(Any) # revealed: typing.Any
Exported as different name
The re-export convention only works when the aliased name is exactly the same as the original name.
# error: "Module `a` has no member `Foo`"
from a import Foo
reveal_type(Foo) # revealed: Unknown
a.pyi
:
from b import AnyFoo as Foo
reveal_type(Foo) # revealed: <class 'AnyFoo'>
b.pyi
:
class AnyFoo: ...
Exported using __all__
Here, the symbol is re-exported using the __all__
variable.
from a import Foo
reveal_type(Foo) # revealed: <class 'Foo'>
a.pyi
:
from b import Foo
__all__ = ['Foo']
b.pyi
:
class Foo: ...
Re-exports with __all__
If a symbol is re-exported via redundant alias but is not included in __all__
, it shouldn't raise
an error when using named import.
named_import.py
:
from a import Foo
reveal_type(Foo) # revealed: <class 'Foo'>
a.pyi
:
from b import Foo as Foo
__all__ = []
b.pyi
:
class Foo: ...
However, a star import would raise an error.
star_import.py
:
from a import *
# error: [unresolved-reference] "Name `Foo` used when not defined"
reveal_type(Foo) # revealed: Unknown
Re-exports in __init__.pyi
Similarly, for an __init__.pyi
(stub) file, importing a non-exported name should raise an error
but the inference would be Unknown
.
# error: 15 "Module `a` has no member `Foo`"
# error: 20 "Module `a` has no member `c`"
from a import Foo, c, foo
reveal_type(Foo) # revealed: Unknown
reveal_type(c) # revealed: Unknown
reveal_type(foo) # revealed: <module 'a.foo'>
a/__init__.pyi
:
from .b import c
from .foo import Foo
a/foo.pyi
:
class Foo: ...
a/b/__init__.pyi
:
a/b/c.pyi
:
Conditional re-export in stub file
The following scenarios are when a re-export happens conditionally in a stub file.
Global import
# error: "Member `Foo` of module `a` is possibly unbound"
from a import Foo
reveal_type(Foo) # revealed: str
a.pyi
:
from b import Foo
def coinflip() -> bool: ...
if coinflip():
Foo: str = ...
reveal_type(Foo) # revealed: <class 'Foo'> | str
b.pyi
:
class Foo: ...
Both branch is an import
Here, both the branches of the condition are import statements where one of them re-exports while the other does not.
# error: "Member `Foo` of module `a` is possibly unbound"
from a import Foo
reveal_type(Foo) # revealed: <class 'Foo'>
a.pyi
:
def coinflip() -> bool: ...
if coinflip():
from b import Foo
else:
from b import Foo as Foo
reveal_type(Foo) # revealed: <class 'Foo'>
b.pyi
:
class Foo: ...
Re-export in one branch
# error: "Member `Foo` of module `a` is possibly unbound"
from a import Foo
reveal_type(Foo) # revealed: <class 'Foo'>
a.pyi
:
def coinflip() -> bool: ...
if coinflip():
from b import Foo as Foo
b.pyi
:
class Foo: ...
Non-export in one branch
# error: "Module `a` has no member `Foo`"
from a import Foo
reveal_type(Foo) # revealed: Unknown
a.pyi
:
def coinflip() -> bool: ...
if coinflip():
from b import Foo
b.pyi
:
class Foo: ...