## Summary
Temporarily modify `UseDefMapBuilder::reachability` for star imports in
order for new definitions to pick up the right reachability. This was
already working for `UseDefMapBuilder::place_states`, but not for
`UseDefMapBuilder::reachable_definitions`.
closes https://github.com/astral-sh/ty/issues/728
## Test Plan
Regression test
## Summary
Evaluate `TYPE_CHECKING` to `ALWAYS_TRUE` and `not TYPE_CHECKING` to
`ALWAYS_FALSE` during semantic index building. This is a follow-up to
https://github.com/astral-sh/ruff/pull/18998 and is in principle just a
performance optimization. We see some (favorable) ecosystem changes
because we can eliminate definitely-unreachable branches early now and
retain narrowing constraints without solving
https://github.com/astral-sh/ty/issues/690 first.
## Summary
This just replaces one temporary solution to recursive protocols (the
`SelfReference` mechanism) with another one (track seen types when
recursively descending in `normalize` and replace recursive references
with `Any`). But this temporary solution can handle mutually-recursive
types, not just self-referential ones, and it's sufficient for the
primer ecosystem and some other projects we are testing on to no longer
stack overflow.
The follow-up here will be to properly handle these self-references
instead of replacing them with `Any`.
We will also eventually need cycle detection on more recursive-descent
type transformations and tests.
## Test Plan
Existing tests (including recursive-protocol tests) and primer.
Added mdtest for mutually-recursive protocols that stack-overflowed
before this PR.
## Summary
Simplifies literal `True` and `False` conditions to `ALWAYS_TRUE` /
`ALWAYS_FALSE` during semantic index building. This allows us to eagerly
evaluate more constraints, which should help with performance (looks
like there is a tiny 1% improvement in instrumented benchmarks), but
also allows us to eliminate definitely-unreachable branches in
control-flow merging. This can lead to better type inference in some
cases because it allows us to retain narrowing constraints without
solving https://github.com/astral-sh/ty/issues/690 first:
```py
def _(c: int | None):
if c is None:
assert False
reveal_type(c) # int, previously: int | None
```
closes https://github.com/astral-sh/ty/issues/713
## Test Plan
* Regression test for https://github.com/astral-sh/ty/issues/713
* Made sure that all ecosystem diffs trace back to removed false
positives
## Summary
This PR adds diagnostic for invalid binary operators in type
expressions. It should close https://github.com/astral-sh/ty/issues/706
if merged.
Please feel free to suggest better wordings for the diagnostic message.
## Test Plan
I modified `mdtest/annotations/invalid.md` and added a test for each
binary operator, and fixed tests that was broken by the new diagnostic.
This PR extracts a lot of the complex logic in the `match_parameters`
and `check_types` methods of our call binding machinery into separate
helper types. This is setup for #18996, which will update this logic to
handle variadic arguments. To do so, it is helpful to have the
per-argument logic extracted into a method that we can call repeatedly
for each _element_ of a variadic argument.
This should be a pure refactoring, with no behavioral changes.
This PR updates our unpacking assignment logic to use the new tuple
machinery. As a result, we can now unpack variable-length tuples
correctly.
As part of this, the `TupleSpec` classes have been renamed to `Tuple`,
and can now contain any element (Rust) type, not just `Type<'db>`. The
unpacker uses a tuple of `UnionBuilder`s to maintain the types that will
be assigned to each target, as we iterate through potentially many union
elements on the rhs. We also add a new consuming iterator for tuples,
and update the `all_elements` methods to wrap the result in an enum
(similar to `itertools::Position`) letting you know which part of the
tuple each element appears in. I also added a new
`UnionBuilder::try_build`, which lets you specify a different fallback
type if the union contains no elements.
## Summary
Ensure that we correctly infer calls such as `tuple((1, 2))`,
`tuple(range(42))`, etc. Ensure that we emit errors on invalid calls
such as `tuple[int, str]()`.
## Test Plan
Mdtests
Most of the work here was doing some light refactoring to facilitate
sensible testing. That is, we don't want to list every builtin included
in most tests, so we add some structure to the completion type returned.
Tests can now filter based on whether a completion is a builtin or not.
Otherwise, builtins are found using the existing infrastructure for
`object.attr` completions (where we hard-code the module name
`builtins`).
I did consider changing the sort order based on whether a completion
suggestion was a builtin or not. In particular, it seemed like it might
be a good idea to sort builtins after other scope based completions,
but before the dunder and sunder attributes. Namely, it seems likely
that there is an inverse correlation between the size of a scope and
the likelihood of an item in that scope being used at any given point.
So it *might* be a good idea to prioritize the likelier candidates in
the completions returned.
Additionally, the number of items introduced by adding builtins is quite
large. So I wondered whether mixing them in with everything else would
become too noisy.
However, it's not totally clear to me that this is the right thing to
do. Right now, I feel like there is a very obvious lexicographic
ordering that makes "finding" the right suggestion to activate
potentially easier than if the ranking mechanism is less clear.
(Technically, the dunder and sunder attributes are not sorted
lexicographically, but I'd put forward that most folks don't have an
intuitive understanding of where `_` ranks lexicographically with
respect to "regular" letters. Moreover, since dunder and sunder
attributes are all grouped together, I think the ordering here ends up
being very obvious after even a quick glance.)
## Summary
Setting `TY_MEMORY_REPORT=full` will generate and print a memory usage
report to the CLI after a `ty check` run:
```
=======SALSA STRUCTS=======
`Definition` metadata=7.24MB fields=17.38MB count=181062
`Expression` metadata=4.45MB fields=5.94MB count=92804
`member_lookup_with_policy_::interned_arguments` metadata=1.97MB fields=2.25MB count=35176
...
=======SALSA QUERIES=======
`File -> ty_python_semantic::semantic_index::SemanticIndex`
metadata=11.46MB fields=88.86MB count=1638
`Definition -> ty_python_semantic::types::infer::TypeInference`
metadata=24.52MB fields=86.68MB count=146018
`File -> ruff_db::parsed::ParsedModule`
metadata=0.12MB fields=69.06MB count=1642
...
=======SALSA SUMMARY=======
TOTAL MEMORY USAGE: 577.61MB
struct metadata = 29.00MB
struct fields = 35.68MB
memo metadata = 103.87MB
memo fields = 409.06MB
```
Eventually, we should integrate these numbers into CI in some form. The
one limitation currently is that heap allocations in salsa structs (e.g.
interned values) are not tracked, but memoized values should have full
coverage. We may also want a peak memory usage counter (that accounts
for non-salsa memory), but that is relatively simple to profile manually
(e.g. `time -v ty check`) and would require a compile-time option to
avoid runtime overhead.
## Summary
Format conflicting declared types as
```
`str`, `int` and `bytes`
```
Thanks to @AlexWaygood for the initial draft.
@dcreager, looking forward to your one-character follow-up PR.
## Summary
This PR includes a behavioral change to how we infer types for public
uses of symbols within a module. Where we would previously use the type
that a use at the end of the scope would see, we now consider all
reachable bindings and union the results:
```py
x = None
def f():
reveal_type(x) # previously `Unknown | Literal[1]`, now `Unknown | None | Literal[1]`
f()
x = 1
f()
```
This helps especially in cases where the the end of the scope is not
reachable:
```py
def outer(x: int):
def inner():
reveal_type(x) # previously `Unknown`, now `int`
raise ValueError
```
This PR also proposes to skip the boundness analysis of public uses.
This is consistent with the "all reachable bindings" strategy, because
the implicit `x = <unbound>` binding is also always reachable, and we
would have to emit "possibly-unresolved" diagnostics for every public
use otherwise. Changing this behavior allows common use-cases like the
following to type check without any errors:
```py
def outer(flag: bool):
if flag:
x = 1
def inner():
print(x) # previously: possibly-unresolved-reference, now: no error
```
closes https://github.com/astral-sh/ty/issues/210
closes https://github.com/astral-sh/ty/issues/607
closes https://github.com/astral-sh/ty/issues/699
## Follow up
It is now possible to resolve the following TODO, but I would like to do
that as a follow-up, because it requires some changes to how we treat
implicit attribute assignments, which could result in ecosystem changes
that I'd like to see separately.
315fb0f3da/crates/ty_python_semantic/src/semantic_index/builder.rs (L1095-L1117)
## Ecosystem analysis
[**Full report**](https://shark.fish/diff-public-types.html)
* This change obviously removes a lot of `possibly-unresolved-reference`
diagnostics (7818) because we do not analyze boundness for public uses
of symbols inside modules anymore.
* As the primary goal here, this change also removes a lot of
false-positive `unresolved-reference` diagnostics (231) in scenarios
like this:
```py
def _(flag: bool):
if flag:
x = 1
def inner():
x
raise
```
* This change also introduces some new false positives for cases like:
```py
def _():
x = None
x = "test"
def inner():
x.upper() # Attribute `upper` on type `Unknown | None | Literal["test"]`
is possibly unbound
```
We have test cases for these situations and it's plausible that we can
improve this in a follow-up.
## Test Plan
New Markdown tests
## Summary
This function is huge, and hugely indented. This PR breaks most of it
out into two helper functions: `KnownFunction::check_call()` and
`KnownClass::check_call`.
My immediate motivation is that we need to add yet more special cases to
this function in order to properly handle `tuple` instantiations and
instantiations of tuple subclasses. But I really don't relish the
thought of doing that with the function's current structure 😆
## Test Plan
Existing tests all pass. No new ones are added; this is a pure refactor
that should have no functional change.
## Summary
Add type narrowing inside comprehensions:
```py
def _(xs: list[int | None]):
[reveal_type(x) for x in xs if x is not None] # revealed: int
```
closes https://github.com/astral-sh/ty/issues/680
## Test Plan
* New Markdown tests
* Made sure the example from https://github.com/astral-sh/ty/issues/680
now checks without errors
* Made sure that all removed ecosystem diagnostics were actually false
positives
## Summary
Having a recursive type method to check whether a type is fully static
is inefficient, unnecessary, and makes us overly strict about subtyping
relations.
It's inefficient because we end up re-walking the same types many times
to check for fully-static-ness.
It's unnecessary because we can check relations involving the dynamic
type appropriately, depending whether the relation is subtyping or
assignability.
We use the subtyping relation to simplify unions and intersections. We
can usefully consider that `S <: T` for gradual types also, as long as
it remains true that `S | T` is equivalent to `T` and `S & T` is
equivalent to `S`.
One conservative definition (implemented here) that satisfies this
requirement is that we consider `S <: T` if, for every possible pair of
materializations `S'` and `T'`, `S' <: T'`. Or put differently the top
materialization of `S` (`S+` -- the union of all possible
materializations of `S`) is a subtype of the bottom materialization of
`T` (`T-` -- the intersection of all possible materializations of `T`).
In the most basic cases we can usefully say that `Any <: object` and
that `Never <: Any`, and we can handle more complex cases inductively
from there.
This definition of subtyping for gradual subtypes is not reflexive
(`Any` is not a subtype of `Any`).
As a corollary, we also remove `is_gradual_equivalent_to` --
`is_equivalent_to` now has the meaning that `is_gradual_equivalent_to`
used to have. If necessary, we could restore an
`is_fully_static_equivalent_to` or similar (which would not do an
`is_fully_static` pre-check of the types, but would instead pass a
relation-kind enum down through a recursive equivalence check, similar
to `has_relation_to`), but so far this doesn't appear to be necessary.
Credit to @JelleZijlstra for the observation that `is_fully_static` is
unnecessary and overly restrictive on subtyping.
There is another possible definition of gradual subtyping: instead of
requiring that `S+ <: T-`, we could instead require that `S+ <: T+` and
`S- <: T-`. In other words, instead of requiring all materializations of
`S` to be a subtype of every materialization of `T`, we just require
that every materialization of `S` be a subtype of _some_ materialization
of `T`, and that every materialization of `T` be a supertype of some
materialization of `S`. This definition also preserves the core
invariant that `S <: T` implies that `S | T = T` and `S & T = S`, and it
restores reflexivity: under this definition, `Any` is a subtype of
`Any`, and for any equivalent types `S` and `T`, `S <: T` and `T <: S`.
But unfortunately, this definition breaks transitivity of subtyping,
because nominal subclasses in Python use assignability ("consistent
subtyping") to define acceptable overrides. This means that we may have
a class `A` with `def method(self) -> Any` and a subtype `B(A)` with
`def method(self) -> int`, since `int` is assignable to `Any`. This
means that if we have a protocol `P` with `def method(self) -> Any`, we
would have `B <: A` (from nominal subtyping) and `A <: P` (`Any` is a
subtype of `Any`), but not `B <: P` (`int` is not a subtype of `Any`).
Breaking transitivity of subtyping is not tenable, so we don't use this
definition of subtyping.
## Test Plan
Existing tests (modified in some cases to account for updated
semantics.)
Stable property tests pass at a million iterations:
`QUICKCHECK_TESTS=1000000 cargo test -p ty_python_semantic -- --ignored
types::property_tests::stable`
### Changes to property test type generation
Since we no longer have a method of categorizing built types as
fully-static or not-fully-static, I had to add a previously-discussed
feature to the property tests so that some tests can build types that
are known by construction to be fully static, because there are still
properties that only apply to fully-static types (for example,
reflexiveness of subtyping.)
## Changes to handling of `*args, **kwargs` signatures
This PR "discovered" that, once we allow non-fully-static types to
participate in subtyping under the above definitions, `(*args: Any,
**kwargs: Any) -> Any` is now a subtype of `() -> object`. This is true,
if we take a literal interpretation of the former signature: all
materializations of the parameters `*args: Any, **kwargs: Any` can
accept zero arguments, making the former signature a subtype of the
latter. But the spec actually says that `*args: Any, **kwargs: Any`
should be interpreted as equivalent to `...`, and that makes a
difference here: `(...) -> Any` is not a subtype of `() -> object`,
because (unlike a literal reading of `(*args: Any, **kwargs: Any)`),
`...` can materialize to _any_ signature, including a signature with
required positional arguments.
This matters for this PR because it makes the "any two types are both
assignable to their union" property test fail if we don't implement the
equivalence to `...`. Because `FunctionType.__call__` has the signature
`(*args: Any, **kwargs: Any) -> Any`, and if we take that at face value
it's a subtype of `() -> object`, making `FunctionType` a subtype of `()
-> object)` -- but then a function with a required argument is also a
subtype of `FunctionType`, but not a subtype of `() -> object`. So I
went ahead and implemented the equivalence to `...` in this PR.
## Ecosystem analysis
* Most of the ecosystem report are cases of improved union/intersection
simplification. For example, we can now simplify a union like `bool |
(bool & Unknown) | Unknown` to simply `bool | Unknown`, because we can
now observe that every possible materialization of `bool & Unknown` is
still a subtype of `bool` (whereas before we would set aside `bool &
Unknown` as a not-fully-static type.) This is clearly an improvement.
* The `possibly-unresolved-reference` errors in sockeye, pymongo,
ignite, scrapy and others are true positives for conditional imports
that were formerly silenced by bogus conflicting-declarations (which we
currently don't issue a diagnostic for), because we considered two
different declarations of `Unknown` to be conflicting (we used
`is_equivalent_to` not `is_gradual_equivalent_to`). In this PR that
distinction disappears and all equivalence is gradual, so a declaration
of `Unknown` no longer conflicts with a declaration of `Unknown`, which
then results in us surfacing the possibly-unbound error.
* We will now issue "redundant cast" for casting from a typevar with a
gradual bound to the same typevar (the hydra-zen diagnostic). This seems
like an improvement.
* The new diagnostics in bandersnatch are interesting. For some reason
primer in CI seems to be checking bandersnatch on Python 3.10 (not yet
sure why; this doesn't happen when I run it locally). But bandersnatch
uses `enum.StrEnum`, which doesn't exist on 3.10. That makes the `class
SimpleDigest(StrEnum)` a class that inherits from `Unknown` (and
bypasses our current TODO handling for accessing attributes on enum
classes, since we don't recognize it as an enum class at all). This PR
improves our understanding of assignability to classes that inherit from
`Any` / `Unknown`, and we now recognize that a string literal is not
assignable to a class inheriting `Any` or `Unknown`.
Add property test generators for the new variable-length tuples. This
covers homogeneous tuples as well.
The property tests did their job! This identified several fixes we
needed to make to various type property methods.
cf https://github.com/astral-sh/ruff/pull/18600#issuecomment-2993764471
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
## Summary
Previously, the checks for implicit attribute assignments didn't
properly account for method decorators. This PR fixes that by:
- Adding a decorator check in `implicit_instance_attribute`. This allows
it to filter out methods with mismatching decorators when analyzing
attribute assignments.
- Adding attribute search for implicit class attributes: if an attribute
can't be found directly in the class body, the
`ClassLiteral::own_class_member` function will now search in
classmethods.
- Adding `staticmethod`: it has been added into `KnownClass` and
together with the new decorator check, it will no longer expose
attributes when the assignment target name is the same as the first
method name.
If accepted, it should fix https://github.com/astral-sh/ty/issues/205
and https://github.com/astral-sh/ty/issues/207.
## Test Plan
This is tested with existing mdtest suites and is able to get most of
the TODO marks for implicit assignments in classmethods and
staticmethods removed.
However, there's one specific test case I failed to figure out how to
correctly resolve:
b279508bdc/crates/ty_python_semantic/resources/mdtest/attributes.md (L754-L755)
I tried to add `instance_member().is_unbound()` check in this [else
branch](b279508bdc/crates/ty_python_semantic/src/types/infer.rs (L3299-L3301))
but it causes tests with class attributes defined in class body to fail.
While it's possible to implicitly add `ClassVar` to qualifiers to make
this assignment fail and keep everything else passing, it doesn't feel
like the right solution.
## Summary
This PR fixesastral-sh/ty#185 by avoiding to infer the value expression
for an unpacking.
This is done simply by only inferring the value expression in a
non-unpacking branch for assignment statement, for statement, with
statement and comprehensions.
This is a simpler alternative to
https://github.com/astral-sh/ruff/pull/18890 which I only realized in
hindsight! Ideally, the solution would to consider the "unpack" as it's
own region and do all of the inference of every expressions involved in
an unpacking inside the unpack query and then merge the results in the
outer query. This would require access to the `Unpack` ingredient which
is stored on the `Definition`. And, this would require create the said
`Definition`s for all attributes and subscript expressions. It does
simplify the target inference logic by streamlining it into a single
`infer_target` method instead of the `infer_target`/`infer_target_impl`
split.
Additionally, #18890 also solves a couple of TODOs around raising errors
around attribute / subscript assignment.
## Test Plan
Update the existing test, go through a couple of ecosystem diagnostic.
## Summary
Note this modifies the diagnostics a bit. Previously performing
subscript access on something like `NotSubscriptable1 |
NotSubscriptable2` would report the full type as not being
subscriptable:
```
[non-subscriptable] "Cannot subscript object of type `NotSubscriptable1 | NotSubscriptable2` with no `__getitem__` method"
```
Now each erroneous constituent has a separate error:
```
[non-subscriptable] "Cannot subscript object of type `NotSubscriptable2` with no `__getitem__` method"
[non-subscriptable] "Cannot subscript object of type `NotSubscriptable1` with no `__getitem__` method"
```
Closes https://github.com/astral-sh/ty/issues/625
## Test Plan
mdtest
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
There were two main challenges in this PR.
The first was mostly just figuring out how to get the symbols
corresponding to `module`. It turns out that we do this in a couple
of places in ty already, but through different means. In one approach,
we use [`exported_names`]. In another approach, we get a `Type`
corresponding to the module. We take the latter approach here, which is
consistent with how we do completions elsewhere. (I looked into
factoring this logic out into its own function, but it ended up being
pretty constrained. e.g., There's only one other place where we want to
go from `ast::StmtImportFrom` to a module `Type`, and that code also
wants the module name.)
The second challenge was recognizing the `from module import <CURSOR>`
pattern in the code. I initially started with some fixed token patterns
to get a proof of concept working. But I ended up switching to mini
state machine over tokens. I looked at the parser for `StmtImportFrom`
to determine what kinds of tokens we can expect.
[`exported_names`]:
23a3b6ef23/crates/ty_python_semantic/src/semantic_index/re_exports.rs (L47)
## Summary
As far as I can tell, the two existing tests did the exact same thing.
Remove the redundant test, and add tests for all combinations of
declared/not-declared and local/"public" use of the name.
Proposing this as a separate PR before the behavior might change via
https://github.com/astral-sh/ruff/pull/18750
## Summary
The code in the `Variable` branch of
`VariableLengthTupleSpec::has_relation_to` made the incorrect assumption
that if you zip two possibly-different-length iterators together and
iterate over the resulting zip iterator, the original two iterators will
only have their common elements consumed. But in fact, the zip iterator
detects that it is done when it receives a `None` from one iterator and
`Some()` element from the other iterator, which means that it consumes
one additional element from the longer iterator. This meant that we
failed to detect mismatched types on this extra consumed element,
because we never compared it to the variable type of the other tuple.
Use `zip_longest` from itertools as an alternative, which allows us to
combine all the handling into just two `zip_longest`, one for prefixes
and one for suffixes.
Marking this PR internal since it fixes a bug in a commit that wasn't
released yet.
## Test Plan
Added mdtests that failed before this fix and pass after it.
## Summary
Fixes https://github.com/astral-sh/ty/issues/640. If a user passes
`--python=<some-virtual-environment>/bin/python`, we must avoid
canonicalizing the path until we've traversed upwards to find the
`sys.prefix` directory (`<some-virtual-environment>`). On Unix systems,
`<sys.prefix>/bin/python` is often a symlink to a system interpreter; if
we resolve the symlink too easily then we'll add the system
interpreter's `site-packages` directory as a search path rather than the
virtual environment's directory.
## Test Plan
I added an integration test to
`crates/ty/tests/cli/python_environment.rs` which fails on `main`. I
also manually tested locally that running `cargo run -p ty check foo.py
--python=.venv/bin/python -vv` now prints this log to the terminal
```
2025-06-20 18:35:24.57702 DEBUG Resolved site-packages directories for this virtual environment are: SitePackagesPaths({"/Users/alexw/dev/ruff/.venv/lib/python3.13/site-packages"})
```
Whereas it previously resolved `site-packages` to my system
intallation's `site-packages` directory
We already had support for homogeneous tuples (`tuple[int, ...]`). This
PR extends this to also support mixed tuples (`tuple[str, str,
*tuple[int, ...], str str]`).
A mixed tuple consists of a fixed-length (possibly empty) prefix and
suffix, and a variable-length portion in the middle. Every element of
the variable-length portion must be of the same type. A homogeneous
tuple is then just a mixed tuple with an empty prefix and suffix.
The new data representation uses different Rust types for a fixed-length
(aka heterogeneous) tuple. Another option would have been to use the
`VariableLengthTuple` representation for all tuples, and to wrap the
"variable + suffix" portion in an `Option`. I don't think that would
simplify the method implementations much, though, since we would still
have a 2×2 case analysis for most of them.
One wrinkle is that the definition of the `tuple` class in the typeshed
has a single typevar, and canonically represents a homogeneous tuple.
When getting the class of a tuple instance, that means that we have to
summarize our detailed mixed tuple type information into its
"homogeneous supertype". (We were already doing this for heterogeneous
types.)
A similar thing happens when concatenating two mixed tuples: the
variable-length portion and suffix of the LHS, and the prefix and
variable-length portion of the RHS, all get unioned into the
variable-length portion of the result. The LHS prefix and RHS suffix
carry through unchanged.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
## Summary
Add support for `@staticmethod`s. Overall, the changes are very similar
to #16305.
#18587 will be dependent on this PR for a potential fix of
https://github.com/astral-sh/ty/issues/207.
mypy_primer will look bad since the new code allows ty to check more
code.
## Test Plan
Added new markdown tests. Please comment if there's any missing tests
that I should add in, thank you.
