## Summary
Use `Type::Divergent` to avoid "too many iterations" panic on an
infinitely-nested tuple in an implicit instance attribute.
The regression here is from checking all tuple elements to see if they
contain a Divergent type. It's 5% on one project, 1% on another, and
zero on the rest. I spent some time looking into eliminating this
regression by tracking a flag on inference results to note if they could
possibly contain any Divergent type, but this doesn't really work --
there are too many different ways a type containing a Divergent type
could enter an inference result. Still thinking about whether there are
other ways to reduce this. One option is if we see certain kinds of
non-atomic types that are commonly expensive to check for Divergent, we
could make `has_divergent_type` a Salsa query on those types.
## Test Plan
Added mdtest.
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Reverts astral-sh/ruff#20156. As @sharkdp noted in his post-merge
review, there were several issues with that PR that I didn't spot before
merging — but I'm out for four days now, and would rather not leave
things in an inconsistent state for that long. I'll revisit this on
Wednesday.
## Summary
This error is about assigning to attributes rather than reading
attributes, so I think `invalid-assignment` makes more sense than
`invalid-attribute-access`
## Test Plan
existing mdtests updated
## Summary
Decrease the maximum number of literals in a union before we collapse to
the supertype. The better fix for this will be
https://github.com/astral-sh/ty/issues/957, but it is very tempting to
solve this for now by simply decreasing the limit by one, to get below
the salsa limit of 200.
closes https://github.com/astral-sh/ty/issues/660
## Test Plan
Added a regression test that would previously lead to a "too many cycle
iterations" panic.
## Summary
With this PR, we stop performing boundness analysis for implicit
instance attributes:
```py
class C:
def __init__(self):
if False:
self.x = 1
C().x # would previously show an error, with this PR we pretend the attribute exists
```
This PR is potentially just a temporary measure until we find a better
fix. But I have already invested a lot of time trying to find the root
cause of https://github.com/astral-sh/ty/issues/758 (and [this
example](https://github.com/astral-sh/ty/issues/758#issuecomment-3206108262),
which I'm not entirely sure is related) and I still don't understand
what is going on. This PR fixes the performance problems in both of
these problems (in a rather crude way).
The impact of the proposed change on the ecosystem is small, and the
three new diagnostics are arguably true positives (previously hidden
because we considered the code unreachable, based on e.g. `assert`ions
that depended on implicit instance attributes). So this seems like a
reasonable fix for now.
Note that we still support cases like these:
```py
class D:
if False: # or any other expression that statically evaluates to `False`
x: int = 1
D().x # still an error
class E:
if False: # or any other expression that statically evaluates to `False`
def f(self):
self.x = 1
E().x # still an error
```
closes https://github.com/astral-sh/ty/issues/758
## Test Plan
Updated tests, benchmark results
## Summary
closes https://github.com/astral-sh/ty/issues/692
If the expression (or any child expressions) is not definitely bound the
reachability constraint evaluation is determined as ambiguous.
This fixes the infinite cycles panic in the following code:
```py
from typing import Literal
class Toggle:
def __init__(self: "Toggle"):
if not self.x:
self.x: Literal[True] = True
```
Credit of this solution is for David.
## Test Plan
- Added a test case with too many cycle iterations panic.
- Previous tests.
---------
Co-authored-by: David Peter <mail@david-peter.de>
## Summary
We currently infer a `@Todo` type whenever we access an attribute on an
intersection type with negative components. This can happen very
naturally. Consequently, this `@Todo` type is rather pervasive and hides
a lot of true positives that ty could otherwise detect:
```py
class Foo:
attr: int = 1
def _(f: Foo | None):
if f:
reveal_type(f) # Foo & ~AlwaysFalsy
reveal_type(f.attr) # now: int, previously: @Todo
```
The changeset here proposes to handle member access on these
intersection types by simply ignoring all negative contributions. This
is not always ideal: a negative contribution like `~<Protocol with
members 'attr'>` could be a hint that `.attr` should not be accessible
on the full intersection type. The behavior can certainly be improved in
the future, but this seems like a reasonable initial step to get rid of
this unnecessary `@Todo` type.
## Ecosystem analysis
There are quite a few changes here. I spot-checked them and found one
bug where attribute access on pure negation types (`~P == object & ~P`)
would not allow attributes on `object` to be accessed. After that was
fixed, I only see true positives and known problems. The fact that a lot
of `unused-ignore-comment` diagnostics go away are also evidence for the
fact that this touches a sensitive area, where static analysis clashes
with dynamically adding attributes to objects:
```py
… # type: ignore # Runtime attribute access
```
## Test Plan
Updated tests.
## Summary
Implement expansion of enums into unions of enum literals (and the
reverse operation). For the enum below, this allows us to understand
that `Color = Literal[Color.RED, Color.GREEN, Color.BLUE]`, or that
`Color & ~Literal[Color.RED] = Literal[Color.GREEN, Color.BLUE]`. This
helps in exhaustiveness checking, which is why we see some removed
`assert_never` false positives. And since exhaustiveness checking also
helps with understanding terminal control flow, we also see a few
removed `invalid-return-type` and `possibly-unresolved-reference` false
positives. This PR also adds expansion of enums in overload resolution
and type narrowing constructs.
```py
from enum import Enum
from typing_extensions import Literal, assert_never
from ty_extensions import Intersection, Not, static_assert, is_equivalent_to
class Color(Enum):
RED = 1
GREEN = 2
BLUE = 3
type Red = Literal[Color.RED]
type Green = Literal[Color.GREEN]
type Blue = Literal[Color.BLUE]
static_assert(is_equivalent_to(Red | Green | Blue, Color))
static_assert(is_equivalent_to(Intersection[Color, Not[Red]], Green | Blue))
def color_name(color: Color) -> str: # no error here (we detect that this can not implicitly return None)
if color is Color.RED:
return "Red"
elif color is Color.GREEN:
return "Green"
elif color is Color.BLUE:
return "Blue"
else:
assert_never(color) # no error here
```
## Performance
I avoided an initial regression here for large enums, but the
`UnionBuilder` and `IntersectionBuilder` parts can certainly still be
optimized. We might want to use the same technique that we also use for
unions of other literals. I didn't see any problems in our benchmarks so
far, so this is not included yet.
## Test Plan
Many new Markdown tests
## Summary
Add a new `Type::EnumLiteral(…)` variant and infer this type for member
accesses on enums.
**Example**: No more `@Todo` types here:
```py
from enum import Enum
class Answer(Enum):
YES = 1
NO = 2
def is_yes(self) -> bool:
return self == Answer.YES
reveal_type(Answer.YES) # revealed: Literal[Answer.YES]
reveal_type(Answer.YES == Answer.NO) # revealed: Literal[False]
reveal_type(Answer.YES.is_yes()) # revealed: bool
```
## Test Plan
* Many new Markdown tests for the new type variant
* Added enum literal types to property tests, ran property tests
## Ecosystem analysis
Summary:
Lots of false positives removed. All of the new diagnostics are
either new true positives (the majority) or known problems. Click for
detailed analysis</summary>
Details:
```diff
AutoSplit (https://github.com/Toufool/AutoSplit)
+ error[call-non-callable] src/capture_method/__init__.py:137:9: Method `__getitem__` of type `bound method CaptureMethodDict.__getitem__(key: Never, /) -> type[CaptureMethodBase]` is not callable on object of type `CaptureMethodDict`
+ error[call-non-callable] src/capture_method/__init__.py:147:9: Method `__getitem__` of type `bound method CaptureMethodDict.__getitem__(key: Never, /) -> type[CaptureMethodBase]` is not callable on object of type `CaptureMethodDict`
+ error[call-non-callable] src/capture_method/__init__.py:148:1: Method `__getitem__` of type `bound method CaptureMethodDict.__getitem__(key: Never, /) -> type[CaptureMethodBase]` is not callable on object of type `CaptureMethodDict`
```
New true positives. That `__getitem__` method is apparently annotated
with `Never` to prevent developers from using it.
```diff
dd-trace-py (https://github.com/DataDog/dd-trace-py)
+ error[invalid-assignment] ddtrace/vendor/psutil/_common.py:29:5: Object of type `None` is not assignable to `Literal[AddressFamily.AF_INET6]`
+ error[invalid-assignment] ddtrace/vendor/psutil/_common.py:33:5: Object of type `None` is not assignable to `Literal[AddressFamily.AF_UNIX]`
```
Arguably true positives:
e0a772c28b/ddtrace/vendor/psutil/_common.py (L29)
```diff
ignite (https://github.com/pytorch/ignite)
+ error[invalid-argument-type] tests/ignite/engine/test_custom_events.py:190:34: Argument to bound method `__call__` is incorrect: Expected `((...) -> Unknown) | None`, found `Literal["123"]`
+ error[invalid-argument-type] tests/ignite/engine/test_custom_events.py:220:37: Argument to function `default_event_filter` is incorrect: Expected `Engine`, found `None`
+ error[invalid-argument-type] tests/ignite/engine/test_custom_events.py:220:43: Argument to function `default_event_filter` is incorrect: Expected `int`, found `None`
+ error[call-non-callable] tests/ignite/engine/test_custom_events.py:561:9: Object of type `CustomEvents` is not callable
+ error[invalid-argument-type] tests/ignite/metrics/test_frequency.py:50:38: Argument to bound method `attach` is incorrect: Expected `Events`, found `CallableEventWithFilter`
```
All true positives. Some of them are inside `pytest.raises(TypeError,
…)` blocks 🙃
```diff
meson (https://github.com/mesonbuild/meson)
+ error[invalid-argument-type] unittests/internaltests.py:243:51: Argument to bound method `__init__` is incorrect: Expected `bool`, found `Literal[MachineChoice.HOST]`
+ error[invalid-argument-type] unittests/internaltests.py:271:51: Argument to bound method `__init__` is incorrect: Expected `bool`, found `Literal[MachineChoice.HOST]`
```
New true positives. Enum literals can not be assigned to `bool`, even if
their value types are `0` and `1`.
```diff
poetry (https://github.com/python-poetry/poetry)
+ error[invalid-assignment] src/poetry/console/exceptions.py:101:5: Object of type `Literal[""]` is not assignable to `InitVar[str]`
```
New false positive, missing support for `InitVar`.
```diff
prefect (https://github.com/PrefectHQ/prefect)
+ error[invalid-argument-type] src/integrations/prefect-dask/tests/test_task_runners.py:193:17: Argument is incorrect: Expected `StateType`, found `Literal[StateType.COMPLETED]`
```
This is confusing. There are two definitions
([one](74d8cd93ee/src/prefect/client/schemas/objects.py (L89-L100)),
[two](https://github.com/PrefectHQ/prefect/blob/main/src/prefect/server/schemas/states.py#L40))
of the `StateType` enum. Here, we're trying to assign one to the other.
I don't think that should be allowed, so this is a true positive (?).
```diff
python-htmlgen (https://github.com/srittau/python-htmlgen)
+ error[invalid-assignment] test_htmlgen/form.py:51:9: Object of type `str` is not assignable to attribute `autocomplete` of type `Autocomplete | None`
+ error[invalid-assignment] test_htmlgen/video.py:38:9: Object of type `str` is not assignable to attribute `preload` of type `Preload | None`
```
True positives. [The stubs are
wrong](01e3b911ac/htmlgen/form.pyi (L8-L10)).
These should not contain type annotations, but rather just `OFF = ...`.
```diff
rotki (https://github.com/rotki/rotki)
+ error[invalid-argument-type] rotkehlchen/tests/unit/test_serialization.py:62:30: Argument to bound method `deserialize` is incorrect: Expected `str`, found `Literal[15]`
```
New true positive.
```diff
vision (https://github.com/pytorch/vision)
+ error[unresolved-attribute] test/test_extended_models.py:302:17: Type `type[WeightsEnum]` has no attribute `DEFAULT`
+ error[unresolved-attribute] test/test_extended_models.py:302:58: Type `type[WeightsEnum]` has no attribute `DEFAULT`
```
Also new true positives. No `DEFAULT` member exists on `WeightsEnum`.
## Summary
Related:
- https://github.com/astral-sh/ty/issues/111
- https://github.com/astral-sh/ruff/pull/17974#discussion_r2108527106
Previously, when validating an attribute assignment, a `__setattr__`
call check was only done if the attribute wasn't found as either a class
member or instance member
This PR changes the `__setattr__` call check to be attempted first,
prior to the "[normal
mechanism](https://docs.python.org/3/reference/datamodel.html#object.__setattr__)",
as a defined `__setattr__` should take precedence over setting an
attribute on the instance dictionary directly.
if the return type of `__setattr__` is `Never`, an `invalid-assignment`
diagnostic is emitted
Once this is merged, a subsequent PR will synthesize a `__setattr__`
method with a `Never` return type for frozen dataclasses.
## Test Plan
Existing tests + mypy_primer
---------
Co-authored-by: David Peter <mail@david-peter.de>
## Summary
It was recently clarified in the [typing
spec](https://typing.python.org/en/latest/spec/class-compat.html#classvar)
that bare `ClassVar` annotations are allowed. For annotated assignments
with a right hand side value, the spec requires type checkers to infer
the type as something "to which [the] value is assignable". For a value
of `2`, the spec suggests `int`, `Literal[2]`, or `Any` as examples.
Here, we choose `Unknown | Literal[2]` instead, conforming with out
usual treatment of attribute types.
closes https://github.com/astral-sh/ty/issues/211
## Summary
Following ty issue [#698](https://github.com/astral-sh/ty/issues/698)
this PR adds support for declarations.
closes#698
## Test Plan
Tested against mdtest (specifically attributes).
---------
Co-authored-by: David Peter <mail@david-peter.de>
## Summary
Allow declared-only class-level attributes to be accessed on the class:
```py
class C:
attr: int
C.attr # this is now allowed
```
closes https://github.com/astral-sh/ty/issues/384
closes https://github.com/astral-sh/ty/issues/553
## Ecosystem analysis
* We see many removed `unresolved-attribute` false-positives for code
that makes use of sqlalchemy, as expected (see changes for `prefect`)
* We see many removed `call-non-callable` false-positives for uses of
`pytest.skip` and similar, as expected
* Most new diagnostics seem to be related to cases like the following,
where we previously inferred `int` for `Derived().x`, but now we infer
`int | None`. I think this should be a
conflicting-declarations/bad-override error anyway? The new behavior may
even be preferred here?
```py
class Base:
x: int | None
class Derived(Base):
def __init__(self):
self.x: int = 1
```
## Summary
Remove a hack in control flow modeling that was treating `return`
statements at the end of function bodies in a special way (basically
considering the state *just before* the `return` statement as the
end-of-scope state). This is not needed anymore now that #18750 has been
merged.
In order to make this work, we now use *all reachable bindings* for
purposes of finding implicit instance attribute assignments as well as
for deferred lookups of symbols. Both would otherwise be affected by
this change:
```py
def C:
def f(self):
self.x = 1 # a reachable binding that is not visible at the end of the scope
return
```
```py
def f():
class X: ... # a reachable binding that is not visible at the end of the scope
x: "X" = X() # deferred use of `X`
return
```
Implicit instance attributes also required another change. We previously
kept track of possibly-unbound instance attributes in some cases, but we
now give up on that completely and always consider *implicit* instance
attributes to be bound if we see a reachable binding in a reachable
method. The previous behavior was somewhat inconsistent anyway because
we also do not consider attributes possibly-unbound in other scenarios:
we do not (and can not) keep track of whether or not methods are called
that define these attributes.
closes https://github.com/astral-sh/ty/issues/711
## Ecosystem analysis
I think this looks very positive!
* We see an unsurprising drop in `possibly-unbound-attribute`
diagnostics (599), mostly for classes that define attributes in `try …
except` blocks, `for` loops, or `if … else: raise …` constructs. There
might obviously also be true positives that got removed, but the vast
majority should be false positives.
* There is also a drop in `possibly-unresolved-reference` /
`unresolved-reference` diagnostics (279+13) from the change to deferred
lookups.
* Some `invalid-type-form` false positives got resolved (13), because we
can now properly look up the names in the annotations.
* There are some new *true* positives in `attrs`, since we understand
the `Attribute` annotation that was previously inferred as `Unknown`
because of a re-assignment after the class definition.
## Test Plan
The existing attributes.md test suite has sufficient coverage here.
## Summary
Previously, the checks for implicit attribute assignments didn't
properly account for method decorators. This PR fixes that by:
- Adding a decorator check in `implicit_instance_attribute`. This allows
it to filter out methods with mismatching decorators when analyzing
attribute assignments.
- Adding attribute search for implicit class attributes: if an attribute
can't be found directly in the class body, the
`ClassLiteral::own_class_member` function will now search in
classmethods.
- Adding `staticmethod`: it has been added into `KnownClass` and
together with the new decorator check, it will no longer expose
attributes when the assignment target name is the same as the first
method name.
If accepted, it should fix https://github.com/astral-sh/ty/issues/205
and https://github.com/astral-sh/ty/issues/207.
## Test Plan
This is tested with existing mdtest suites and is able to get most of
the TODO marks for implicit assignments in classmethods and
staticmethods removed.
However, there's one specific test case I failed to figure out how to
correctly resolve:
b279508bdc/crates/ty_python_semantic/resources/mdtest/attributes.md (L754-L755)
I tried to add `instance_member().is_unbound()` check in this [else
branch](b279508bdc/crates/ty_python_semantic/src/types/infer.rs (L3299-L3301))
but it causes tests with class attributes defined in class body to fail.
While it's possible to implicitly add `ClassVar` to qualifiers to make
this assignment fail and keep everything else passing, it doesn't feel
like the right solution.
We already had support for homogeneous tuples (`tuple[int, ...]`). This
PR extends this to also support mixed tuples (`tuple[str, str,
*tuple[int, ...], str str]`).
A mixed tuple consists of a fixed-length (possibly empty) prefix and
suffix, and a variable-length portion in the middle. Every element of
the variable-length portion must be of the same type. A homogeneous
tuple is then just a mixed tuple with an empty prefix and suffix.
The new data representation uses different Rust types for a fixed-length
(aka heterogeneous) tuple. Another option would have been to use the
`VariableLengthTuple` representation for all tuples, and to wrap the
"variable + suffix" portion in an `Option`. I don't think that would
simplify the method implementations much, though, since we would still
have a 2×2 case analysis for most of them.
One wrinkle is that the definition of the `tuple` class in the typeshed
has a single typevar, and canonically represents a homogeneous tuple.
When getting the class of a tuple instance, that means that we have to
summarize our detailed mixed tuple type information into its
"homogeneous supertype". (We were already doing this for heterogeneous
types.)
A similar thing happens when concatenating two mixed tuples: the
variable-length portion and suffix of the LHS, and the prefix and
variable-length portion of the RHS, all get unioned into the
variable-length portion of the result. The LHS prefix and RHS suffix
carry through unchanged.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
## Summary
This PR closesastral-sh/ty#164.
This PR introduces a basic type narrowing mechanism for
attribute/subscript expressions.
Member accesses, int literal subscripts, string literal subscripts are
supported (same as mypy and pyright).
## Test Plan
New test cases are added to `mdtest/narrow/complex_target.md`.
---------
Co-authored-by: David Peter <mail@david-peter.de>
## Summary
This PR partially solves https://github.com/astral-sh/ty/issues/164
(derived from #17643).
Currently, the definitions we manage are limited to those for simple
name (symbol) targets, but we expand this to track definitions for
attribute and subscript targets as well.
This was originally planned as part of the work in #17643, but the
changes are significant, so I made it a separate PR.
After merging this PR, I will reflect this changes in #17643.
There is still some incomplete work remaining, but the basic features
have been implemented, so I am publishing it as a draft PR.
Here is the TODO list (there may be more to come):
* [x] Complete rewrite and refactoring of documentation (removing
`Symbol` and replacing it with `Place`)
* [x] More thorough testing
* [x] Consolidation of duplicated code (maybe we can consolidate the
handling related to name, attribute, and subscript)
This PR replaces the current `Symbol` API with the `Place` API, which is
a concept that includes attributes and subscripts (the term is borrowed
from Rust).
## Test Plan
`mdtest/narrow/assignment.md` is added.
---------
Co-authored-by: David Peter <sharkdp@users.noreply.github.com>
Co-authored-by: Carl Meyer <carl@astral.sh>
## Summary
Closes https://github.com/astral-sh/ty/issues/502.
In the following example:
```py
class Foo:
x: int
def method(self):
y = x
```
The user may intended to use `y = self.x` in `method`.
This is now added as a subdiagnostic in the following form :
`info: An attribute with the same name as 'x' is defined, consider using
'self.x'`
## Test Plan
Added mdtest with snapshot diagnostics.
## Summary
`Type::member_lookup_with_policy` now falls back to calling
`__getattribute__` when a member cannot be found as a second fallback
after `__getattr__`.
closes https://github.com/astral-sh/ty/issues/441
## Test Plan
Added markdown tests.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: David Peter <mail@david-peter.de>
## Summary
Fix a bug that involved writes to attributes on union/intersection types
that included modules as elements.
This is a prerequisite to avoid some ecosystem false positives in
https://github.com/astral-sh/ruff/pull/18312
## Test Plan
Added regression test
## Summary
Make sure that the following definitions all lead to the same outcome
(bug originally noticed by @AlexWaygood)
```py
from typing import ClassVar
class Descriptor:
def __get__(self, instance, owner) -> int:
return 42
class C:
a: ClassVar[Descriptor]
b: Descriptor = Descriptor()
c: ClassVar[Descriptor] = Descriptor()
reveal_type(C().a) # revealed: int (previously: int | Descriptor)
reveal_type(C().b) # revealed: int
reveal_type(C().c) # revealed: int
```
## Test Plan
New Markdown tests
## Summary
Dunder methods are never looked up on instances. We do this implicitly
in `try_call_dunder`, but the corresponding flag was missing in the
instance-construction code where we use `member_lookup_with_policy`
directly.
fixes https://github.com/astral-sh/ty/issues/322
## Test Plan
Added regression test.
It's possible for a typevar to list another typevar as its default
value:
```py
class C[T, U = T]: ...
```
When specializing this class, if a type isn't provided for `U`, we would
previously use the default as-is, leaving an unspecialized `T` typevar
in the specialization. Instead, we want to use what `T` is mapped to as
the type of `U`.
```py
reveal_type(C()) # revealed: C[Unknown, Unknown]
reveal_type(C[int]()) # revealed: C[int, int]
reveal_type(C[int, str]()) # revealed: C[int, str]
```
This is especially important for the `slice` built-in type.
