## Summary
Avoid literal promotion when a literal type annotation is provided, e.g.,
```py
x: list[Literal[1]] = [1]
```
Resolves https://github.com/astral-sh/ty/issues/1198. This does not fix
issue https://github.com/astral-sh/ty/issues/1284, but it does make it
more relevant because after this change, it is possible to directly
instantiate a generic type with a literal specialization.
## Summary
Respect parameters such as `frozen_default` for metaclass-based
`@dataclass_transformer` models.
Related to: https://github.com/astral-sh/ty/issues/1260
## Typing conformance changes
Those are all correct (new true positives)
## Test Plan
New Markdown tests
## Summary
- Add support for eq, kw_only, and frozen parameter overrides in
@dataclass_transform
- Previously only order parameter override was supported
- Update test documentation to reflect fixed behavior
- Resolves issue where kw_only_default and frozen_default could not be
overridden
closes https://github.com/astral-sh/ty/issues/1260
## Test Plan
New Markdown tests
---------
Co-authored-by: David Peter <mail@david-peter.de>
## Summary
Even disambiguating classes using their fully qualified names is not
enough for some diagnostics. We've seen real-world examples in the
ecosystem (and https://github.com/astral-sh/ruff/pull/20368 introduces
some more!) where two types can be different, but can still have the
same fully qualified name. In these cases, our disambiguation machinery
needs to print the file path and line number of the class in order to
disambiguate classes with similar names in our diagnostics.
Helps with https://github.com/astral-sh/ty/issues/1306
## Test Plan
Mdtests
## Summary
This adds a couple of new test cases related to
https://github.com/astral-sh/ty/issues/1067 and beyond that. For now,
they are just documenting the current (problematic) behavior. Since the
topic has some subtleties, I'd like to merge this prior to the actual
bugfix(es) in order to evaluate the changes in an easier way.
## Summary
We have the following test in `protocols.md`:
```py
class HasX(Protocol):
x: int
# […]
class Foo:
x: int
# […]
class FooBool(Foo):
x: bool
static_assert(not is_subtype_of(FooBool, HasX))
static_assert(not is_assignable_to(FooBool, HasX))
```
If `Foo` was indeed intended to be a base class of `FooBool`, then `x:
bool` should be reported as a Liskov violation. And then it's a matter
of definition whether or not these assertions should hold true or not
(should the incorrect override take precedence or not?). So it looks to
me like this is just an oversight, probably a copy-paste error from
another test right before it, where `FooSub` is indeed intended to be a
subclass of `Foo`.
I am fixing this because this test started to fail on a branch of mine
that changes how attribute lookup in inheritance chains works.
## Summary
Fixes [astral-sh/ty#1307](https://github.com/astral-sh/ty/issues/1307)
Unions with length <= 5 are unaffected to minimize test churn
Unions with length > 5 will only display the first 3 elements + "...
omitted x union elements"
Here "length" is defined as the number of elements after condensation to
literals
Edit: we no longer truncate in revel case.
Before:
> info: Attempted to call union type `(def f1() -> int) | (def f2(name:
str) -> int) | (def f3(a: int, b: int) -> int) | (def f4[T](x: T@f4) ->
int) | Literal[5] | (Overload[() -> None, (x: str) -> str]) |
(Overload[() -> None, (x: str, y: str) -> str]) | PossiblyNotCallable`
After:
> info: Attempted to call union type `(def f1() -> int) | (def f2(name:
str) -> int) | (def f3(a: int, b: int) -> int) | ... omitted 5 union
elements`
The below comparisons are outdated, but left here as a reference.
Before:
```reveal_type(x) # revealed: Literal[1, 2] | A | B | C | D | E | F | G```
```reveal_type(x) # revealed: Result1A | Result1B | Result2A | Result2B
| Result3 | Result4```
After:
```reveal_type(x) # revealed: Literal[1, 2] | A | B | ... omitted 5 union elements```
```reveal_type(x) # revealed: Result1A | Result1B | Result2A | ...
omitted 3 union elements```
This formatting is consistent with
`crates/ty_python_semantic/src/types/call/bind.rs` line 2992
## Test Plan
Cosmetic only, covered and verified by changes in mdtest
## Summary
Bump the latest supported Python version of ty to 3.14 and updates some
references from 3.13 to 3.14.
This also fixes a bug with `dataclasses.field` on 3.14 (which adds a new
keyword-only parameter to that function, breaking our previously naive
matching on the parameter structure of that function).
## Test Plan
A `ty check` on a file with template strings (without any further
configuration) doesn't raise errors anymore.
## Summary
Typevar attributes (bound/constraints/default) can be either lazily
evaluated or eagerly evaluated. Currently they are lazily evaluated for
PEP 695 typevars, and eager for legacy and synthetic typevars.
https://github.com/astral-sh/ruff/pull/20598 will make them lazy also
for legacy typevars, and the ecosystem report on that PR surfaced the
issue fixed here (because legacy typevars are much more common in the
ecosystem than PEP 695 typevars.)
Applying a transform to a typevar (normalization, materialization, or
mark-inferable) will reify all lazy attributes and create a new typevar
with eager attributes. In terms of Salsa identity, this transformed
typevar will be considered different from the original typevar, whether
or not the attributes were actually transformed.
In general, this is not a problem, since all typevars in a given generic
context will be transformed, or not, together.
The exception to this was implicit-self vs explicit Self annotations.
The typevar we created for implicit self was created initially using
inferable typevars, whereas an explicit Self annotation is initially
non-inferable, then transformed via mark-inferable when accessed as part
of a function signature. If the containing class (which becomes the
upper bound of `Self`) is generic, and has e.g. a lazily-evaluated
default, then the explicit-Self annotation will reify that default in
the upper bound, and the implicit-self would not, leading them to be
treated as different typevars, and causing us to fail to solve a call to
a method such as `def method(self) -> Self` correctly.
The fix here is to treat implicit-self more like explicit-Self,
initially creating it as non-inferable and then using the mark-inferable
transform on it. This is less efficient, but restores the invariant that
all typevars in a given generic context are transformed together, or
not, fixing the bug.
In the improved-constraint-solver work, the separation of typevars into
"inferable" and "non-inferable" is expected to disappear, along with the
mark-inferable transform, which would render both this bug and the fix
moot. So this fix is really just temporary until that lands.
There is a performance regression, but not a huge one: 1-2% on most
projects, 5% on one outlier. This seems acceptable, given that it should
be fully recovered by removing the mark-inferable transform.
## Test Plan
Added mdtests that failed before this change.
This PR adds a specialization inference special case that lets us handle
the following examples better:
```py
def f[T](t: T | None) -> T: ...
def g[T](t: T | int | None) -> T | int: ...
def _(x: str | None):
reveal_type(f(x)) # revealed: str (previously str | None)
def _(y: str | int | None):
reveal_type(g(x)) # revealed: str | int (previously str | int | None)
```
We already have a special case for when the formal is a union where one
element is a typevar, but it maps the entire actual type to the typevar
(as you can see in the "previously" results above).
The new special case kicks in when the actual is also a union. Now, we
filter out any actual union elements that are already subtypes of the
formal, and only bind whatever types remain to the typevar. (The `|
None` pattern appears quite often in the ecosystem results, but it's
more general and works with any number of non-typevar union elements.)
The new constraint solver should handle this case as well, but it's
worth adding this heuristic now with the old solver because it
eliminates some false positives from the ecosystem report, and makes the
ecosystem report less noisy on the other constraint solver PRs.
Summary
--
Closes#19467 and also removes the warning about using Python 3.14
without
preview enabled.
I also bumped `PythonVersion::default` to 3.9 because it reaches EOL
this month,
but we could also defer that for now if we wanted.
The first three commits are related to the `latest` bump to 3.14; the
fourth commit
bumps the default to 3.10.
Note that this PR also bumps the default Python version for ty to 3.10
because
there was a test asserting that it stays in sync with
`ast::PythonVersion`.
Test Plan
--
Existing tests
I spot-checked the ecosystem report, and I believe these are all
expected. Inbits doesn't specify a target Python version, so I guess
we're applying the default. UP007, UP035, and UP045 all use the new
default value to emit new diagnostics.
## Summary
Use the type annotation of function parameters as bidirectional type
context when inferring the argument expression. For example, the
following example now type-checks:
```py
class TD(TypedDict):
x: int
def f(_: TD): ...
f({ "x": 1 })
```
Part of https://github.com/astral-sh/ty/issues/168.
## Summary
The union `T | U` can be validly simplified to `U` iff:
1. `T` is a subtype of `U` OR
2. `T` is equivalent to `U` OR
3. `U` is a union and contains a type that is equivalent to `T` OR
4. `T` is an intersection and contains a type that is equivalent to `U`
(In practice, the only situation in which 2, 3 or 4 would be true when
(1) was not true would be if `T` or `U` is a dynamic type.)
Currently we achieve these simplifications in the union builder by doing
something along the lines of `t.is_subtype_of(db, u) ||
t.is_equivalent_to_(db, u) ||
t.into_intersection().is_some_and(|intersection|
intersection.positive(db).contains(&u)) ||
u.into_union().is_some_and(|union| union.elements(db).contains(&t))`.
But this is both slow and misses some cases (it doesn't simplify the
union `Any | (Unknown & ~None)` to `Any`, for example). We can improve
the consistency and performance of our union simplifications by adding a
third type relation that sits in between `TypeRelation::Subtyping` and
`TypeRelation::Assignability`: `TypeRelation::UnionSimplification`.
This change leads to simpler, more user-friendly types due to the more
consistent simplification. It also lead to a pretty huge performance
improvement!
## Test Plan
Existing tests, plus some new ones.
## Summary
closes: https://github.com/astral-sh/ty/issues/247
This PR adds support for variadic arguments to overload call evaluation.
This basically boils down to making sure that the overloads are not
filtered out incorrectly during the step 5 in the overload call
evaluation algorithm. For context, the step 5 tries to filter out the
remaining overloads after finding an overload where the materialization
of argument types are assignable to the parameter types.
The issue with the previous implementation was that it wouldn't unpack
the variadic argument and wouldn't consider the many-to-one (multiple
arguments mapping to a single variadic parameter) correctly. This PR
fixes that.
## Test Plan
Update existing test cases and resolve the TODOs.
## Summary
Currently we do not emit an error on this code:
```py
from ty_extensions import Not
def f[T](x: T, y: Not[T]) -> T:
x = y
return x
```
But we should do! `~T` should never be assignable to `T`.
This fixes a small regression introduced in
14fe1228e7 (diff-8049ab5af787dba29daa389bbe2b691560c15461ef536f122b1beab112a4b48aR1443-R1446),
where a branch that previously returned `false` was replaced with a
branch that returns `C::always_satisfiable` -- the opposite of what it
used to be! The regression occurred because we didn't have any tests for
this -- so I added some tests in this PR that fail on `main`. I only
spotted the problem because I was going through the code of
`has_relation_to_impl` with a fine toothcomb for
https://github.com/astral-sh/ruff/pull/20602😄
## Summary
Quoting from the newly added comment:
Module-level globals can be mutated externally. A `MY_CONSTANT = 1`
global might be changed to `"some string"` from code outside of the
module that we're looking at, and so from a gradual-guarantee
perspective, it makes sense to infer a type of `Literal[1] | Unknown`
for global symbols. This allows the code that does the mutation to type
check correctly, and for code that uses the global, it accurately
reflects the lack of knowledge about the type.
External modifications (or modifications through `global` statements)
that would require a wider type are relatively rare. From a practical
perspective, we can therefore achieve a better user experience by
trusting the inferred type. Users who need the external mutation to work
can always annotate the global with the wider type. And everyone else
benefits from more precise type inference.
I initially implemented this by applying literal promotion to the type
of the unannotated module globals (as suggested in
https://github.com/astral-sh/ty/issues/1069), but the ecosystem impact
showed a lot of problems (https://github.com/astral-sh/ruff/pull/20643).
I fixed/patched some of these problems, but this PR seems like a good
first step, and it seems sensible to apply the literal promotion change
in a second step that can be evaluated separately.
closes https://github.com/astral-sh/ty/issues/1069
## Ecosystem impact
This seems like an (unexpectedly large) net positive with 650 fewer
diagnostics overall.. even though this change will certainly catch more
true positives.
* There are 666 removed `type-assertion-failure` diagnostics, where we
were previously used the correct type already, but removing the
`Unknown` now leads to an "exact" match.
* 1464 of the 1805 total new diagnostics are `unresolved-attribute`
errors, most (1365) of which were previously
`possibly-missing-attribute` errors. So they could also be counted as
"changed" diagnostics.
* For code that uses constants like
```py
IS_PYTHON_AT_LEAST_3_10 = sys.version_info >= (3, 10)
```
where we would have previously inferred a type of `Literal[True/False] |
Unknown`, removing the `Unknown` now allows us to do reachability
analysis on branches that use these constants, and so we get a lot of
favorable ecosystem changes because of that.
* There is code like the following, where we previously emitted
`conflicting-argument-forms` diagnostics on calls to the aliased
`assert_type`, because its type was `Unknown | def …` (and the call to
`Unknown` "used" the type form argument in a non type-form way):
```py
if sys.version_info >= (3, 11):
import typing
assert_type = typing.assert_type
else:
import typing_extensions
assert_type = typing_extensions.assert_type
```
* ~100 new `invalid-argument-type` false positives, due to missing
`**kwargs` support (https://github.com/astral-sh/ty/issues/247)
## Typing conformance
```diff
+protocols_modules.py:25:1: error[invalid-assignment] Object of type `<module '_protocols_modules1'>` is not assignable to `Options1`
```
This diagnostic should apparently not be there, but it looks like we
also fail other tests in that file, so it seems to be a limitation that
was previously hidden by `Unknown` somehow.
## Test Plan
Updated tests and relatively thorough ecosystem analysis.
## Summary
Reformulation of the public symbol type inference test suite to use
class scopes instead of module scopes. This is in preparation for an
upcoming change to module-global scopes (#20664).
## Test Plan
Updated tests
This doesn't seem to be flaky in the sense of tests failing
non-deterministically, but they are flaky in the sense of unrelated
changes causing testing failures from the clauses of a constraint set
being rendered in different orders. This flakiness is because we're
using Salsa IDs to determine the order in which typevars appear in a
constraint set BDD, and those IDs are assigned non-deterministically.
The fix is ham-fisted but effective: sort the constraints in each
clause, and the clauses in each set, as part of the rendering process.
Constraint sets are only rendered in our test cases, so we don't need to
over-optimize this.
## Summary
Not sure if this was the original intention, but it looks to me like the
previous `Type::literal_promotion_type` was more of an implementation
detail for the actual operation of promoting all literals in a
possibly-nested position of a type.
This is not a pure refactor, as I'm technically changing the behavior
for that protocols diagnostic message suggestion.
## Test Plan
New Markdown test
## Summary
Add two simple tests that we recently discussed with @dcreager. They
demonstrate that the `TypeMapping::MarkTypeVarsInferable` operation
really does need to keep track of the binding context.
## Test Plan
Made sure that those tests fail if we create
`TypeMapping::MarkTypeVarsInferable(None)`s everywhere.
## Summary
Modify the (external) signature of instance methods such that the first
parameter uses `Self` unless it is explicitly annotated. This allows us
to correctly type-check more code, and allows us to infer correct return
types for many functions that return `Self`. For example:
```py
from pathlib import Path
from datetime import datetime, timedelta
reveal_type(Path(".config") / ".ty") # now Path, previously Unknown
def _(dt: datetime, delta: timedelta):
reveal_type(dt - delta) # now datetime, previously Unknown
```
part of https://github.com/astral-sh/ty/issues/159
## Performance
I ran benchmarks locally on `attrs`, `freqtrade` and `colour`, the
projects with the largest regressions on CodSpeed. I see much smaller
effects locally, but can definitely reproduce the regression on `attrs`.
From looking at the profiling results (on Codspeed), it seems that we
simply do more type inference work, which seems plausible, given that we
now understand much more return types (of many stdlib functions). In
particular, whenever a function uses an implicit `self` and returns
`Self` (without mentioning `Self` anywhere else in its signature), we
will now infer the correct type, whereas we would previously return
`Unknown`. This also means that we need to invoke the generics solver in
more cases. Comparing half a million lines of log output on attrs, I can
see that we do 5% more "work" (number of lines in the log), and have a
lot more `apply_specialization` events (7108 vs 4304). On freqtrade, I
see similar numbers for `apply_specialization` (11360 vs 5138 calls).
Given these results, I'm not sure if it's generally worth doing more
performance work, especially since none of the code modifications
themselves seem to be likely candidates for regressions.
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|:---|---:|---:|---:|---:|
| `./ty_main check /home/shark/ecosystem/attrs` | 92.6 ± 3.6 | 85.9 |
102.6 | 1.00 |
| `./ty_self check /home/shark/ecosystem/attrs` | 101.7 ± 3.5 | 96.9 |
113.8 | 1.10 ± 0.06 |
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|:---|---:|---:|---:|---:|
| `./ty_main check /home/shark/ecosystem/freqtrade` | 599.0 ± 20.2 |
568.2 | 627.5 | 1.00 |
| `./ty_self check /home/shark/ecosystem/freqtrade` | 607.9 ± 11.5 |
594.9 | 626.4 | 1.01 ± 0.04 |
| Command | Mean [ms] | Min [ms] | Max [ms] | Relative |
|:---|---:|---:|---:|---:|
| `./ty_main check /home/shark/ecosystem/colour` | 423.9 ± 17.9 | 394.6
| 447.4 | 1.00 |
| `./ty_self check /home/shark/ecosystem/colour` | 426.9 ± 24.9 | 373.8
| 456.6 | 1.01 ± 0.07 |
## Test Plan
New Markdown tests
## Ecosystem report
* apprise: ~300 new diagnostics related to problematic stubs in apprise
😩
* attrs: a new true positive, since [this
function](4e2c89c823/tests/test_make.py (L2135))
is missing a `@staticmethod`?
* Some legitimate true positives
* sympy: lots of new `invalid-operator` false positives in [matrix
multiplication](cf9f4b6805/sympy/matrices/matrixbase.py (L3267-L3269))
due to our limited understanding of [generic `Callable[[Callable[[T1,
T2], T3]], Callable[[T1, T2], T3]]` "identity"
types](cf9f4b6805/sympy/core/decorators.py (L83-L84))
of decorators. This is not related to type-of-self.
## Typing conformance results
The changes are all correct, except for
```diff
+generics_self_usage.py:50:5: error[invalid-assignment] Object of type `def foo(self) -> int` is not assignable to `(typing.Self, /) -> int`
```
which is related to an assignability problem involving type variables on
both sides:
```py
class CallableAttribute:
def foo(self) -> int:
return 0
bar: Callable[[Self], int] = foo # <- we currently error on this assignment
```
---------
Co-authored-by: Shaygan Hooshyari <sh.hooshyari@gmail.com>
## Summary
Fixes a bug observed by @AlexWaygood where `C[Any] <: C[object]` should
hold for a class that is covariant in its type parameter (and similar
subtyping relations involving dynamic types for other variance
configurations).
## Test Plan
New and updated Markdown tests
While working on #20093, I kept running into test failures due to
constraint sets not simplifying as much as they could, and therefore not
being easily testable against "always true" and "always false".
This PR updates our constraint set representation to use BDDs. Because
BDDs are reduced and ordered, they are canonical — equivalent boolean
formulas are represented by the same interned BDD node.
That said, there is a wrinkle, in that the "variables" that we use in
these BDDs — the individual constraints like `Lower ≤ T ≤ Upper` are not
always independent of each other.
As an example, given types `A ≤ B ≤ C ≤ D` and a typevar `T`, the
constraints `A ≤ T ≤ C` and `B ≤ T ≤ D` "overlap" — their intersection
is non-empty. So we should be able to simplify
```
(A ≤ T ≤ C) ∧ (B ≤ T ≤ D) == (B ≤ T ≤ C)
```
That's not a simplification that the BDD structure can perform itself,
since those three constraints are modeled as separate BDD variables, and
are therefore "opaque" to the BDD algorithms.
That means we need to perform this kind of simplification ourselves. We
look at pairs of constraints that appear in a BDD and see if they can be
simplified relative to each other, and if so, replace the pair with the
simplification. A large part of the toil of getting this PR to work was
identifying all of those patterns and getting that substitution logic
correct.
With this new representation, all existing tests pass, as well as some
new ones that represent test failures that were occuring on #20093.
---------
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
Closes: https://github.com/astral-sh/ty/issues/551
This PR adds support for step 4 of the overload call evaluation
algorithm which states that:
> If the argument list is compatible with two or more overloads,
determine whether one or more of the overloads has a variadic parameter
(either `*args` or `**kwargs`) that maps to a corresponding argument
that supplies an indeterminate number of positional or keyword
arguments. If so, eliminate overloads that do not have a variadic
parameter.
And, with that, the overload call evaluation algorithm has been
implemented completely end to end as stated in the typing spec.
## Test Plan
Expand the overload call test suite.
## Summary
This removes a hack in the protocol satisfiability check that was
previously needed to work around missing assignability-modeling of
inferable type variables. Assignability of type variables is not
implemented fully, but some recent changes allow us to remove that hack
with limited impact on the ecosystem (and the test suite). The change in
the typing conformance test is favorable.
## Test Plan
* Adapted Markdown tests
* Made sure that this change works in combination with
https://github.com/astral-sh/ruff/pull/20517
## Summary
Closes: https://github.com/astral-sh/ty/issues/1236
This PR fixes a bug where the variadic argument wouldn't match against
the variadic parameter in certain scenarios.
This was happening because I didn't realize that the `all_elements`
iterator wouldn't keep on returning the variable element (which is
correct, I just didn't realize it back then).
I don't think we can use the `resize` method here because we don't know
how many parameters this variadic argument is matching against as this
is where the actual parameter matching occurs.
## Test Plan
Expand test cases to consider a few more combinations of arguments and
parameters which are variadic.
## Summary
This applies the trick that we use for `builtins.open` to similar
functions that have the same problem. The reason is that the problem
would otherwise become even more pronounced once we add understanding of
the implicit type of `self` parameters, because then something like
`(base_path / "test.bin").open("rb")` also leads to a wrong return type
and can result in false positives.
## Test Plan
New Markdown tests
## Summary
I found this bug while working on #20528.
The minimum reproducible code is:
```python
from __future__ import annotations
from typing import NamedTuple
from ty_extensions import is_disjoint_from, static_assert
class Path(NamedTuple):
prev: Path | None
key: str
static_assert(not is_disjoint_from(Path, Path))
```
A stack overflow occurs when a nominal instance type inherits from
`NamedTuple` and is defined recursively.
This PR fixes this bug.
## Test Plan
mdtest updated
### Summary
This PR includes two changes, both of which are necessary to resolve
https://github.com/astral-sh/ty/issues/1196:
* For a generic class `C[T]`, we previously used `C[Unknown]` as the
upper bound of the `Self` type variable. There were two problems with
this. For one, when `Self` appeared in contravariant position, we would
materialize its upper bound to `Bottom[C[Unknown]]` (which might
simplify to `C[Never]` if `C` is covariant in `T`) when accessing
methods on `Top[C[Unknown]]`. This would result in `invalid-argument`
errors on the `self` parameter. Also, using an upper bound of
`C[Unknown]` would mean that inside methods, references to `T` would be
treated as `Unknown`. This could lead to false negatives. To fix this,
we now use `C[T]` (with a "nested" typevar) as the upper bound for
`Self` on `C[T]`.
* In order to make this work, we needed to allow assignability/subtyping
of inferable typevars to other types, since we now check assignability
of e.g. `C[int]` to `C[T]` (when checking assignability to the upper
bound of `Self`) when calling an instance-method on `C[int]` whose
`self` parameter is annotated as `self: Self` (or implicitly `Self`,
following https://github.com/astral-sh/ruff/pull/18007).
closes https://github.com/astral-sh/ty/issues/1196
closes https://github.com/astral-sh/ty/issues/1208
### Test Plan
Regression tests for both issues.
## Summary
@ibraheemdev notes this example failed
```py
from typing import Callable
class X:
...
def f(callable: Callable[[], X]) -> X:
return callable()
x = f(X)
```
Resolves https://github.com/astral-sh/ty/issues/1210
The issue was that we set the `Self` to the class type instead of the
instance type of the class.
## Test Plan
Fix tests in `is_subtype_of.md`
