## Summary
Support `dataclass_transform` when used on a (base) class.
## Typing conformance
* The changes in `dataclasses_transform_class.py` look good, just a few
mistakes due to missing `alias` support.
* I didn't look closely at the changes in
`dataclasses_transform_converter.py` since we don't support `converter`
yet.
## Ecosystem impact
The impact looks huge, but it's concentrated on a single project (ibis).
Their setup looks more or less like this:
* the real `Annotatable`:
d7083c2c96/ibis/common/grounds.py (L100-L101)
* the real `DataType`:
d7083c2c96/ibis/expr/datatypes/core.py (L161-L179)
* the real `Array`:
d7083c2c96/ibis/expr/datatypes/core.py (L1003-L1006)
```py
from typing import dataclass_transform
@dataclass_transform()
class Annotatable:
pass
class DataType(Annotatable):
nullable: bool = True
class Array[T](DataType):
value_type: T
```
They expect something like `Array([1, 2])` to work, but ty, pyright,
mypy, and pyrefly would all expect there to be a first argument for the
`nullable` field on `DataType`. I don't really understand on what
grounds they expect the `nullable` field to be excluded from the
signature, but this seems to be the main reason for the new diagnostics
here. Not sure if related, but it looks like their typing setup is not
really complete
(https://github.com/ibis-project/ibis/issues/6844#issuecomment-1868274770,
this thread also mentions `dataclass_transform`).
## Test Plan
Update pre-existing tests.
Detect legacy namespace packages and treat them like namespace packages
when looking them up as the *parent* of the module we're interested in.
In all other cases treat them like a regular package.
(This PR is coauthored by @MichaReiser in a shared coding session)
Fixes https://github.com/astral-sh/ty/issues/838
---------
Co-authored-by: Micha Reiser <micha@reiser.io>
## Summary
Prefer the declared type for collection literals, e.g.,
```py
x: list[Any] = [1, "2", (3,)]
reveal_type(x) # list[Any]
```
This solves a large part of https://github.com/astral-sh/ty/issues/136
for invariant generics, where respecting the declared type is a lot more
important. It also means that annotated dict literals with `dict[_,
Any]` is a way out of https://github.com/astral-sh/ty/issues/1248.
We have to track whether a typevar appears in a position where it's
inferable or not. In a non-inferable position (in the body of the
generic class or function that binds it), assignability must hold for
every possible specialization of the typevar. In an inferable position,
it only needs to hold for _some_ specialization.
https://github.com/astral-sh/ruff/pull/20093 is working on using
constraint sets to model assignability of typevars, and the constraint
sets that we produce will be the same for inferable vs non-inferable
typevars; what changes is what we _compare_ that constraint set to. (For
a non-inferable typevar, the constraint set must equal the set of valid
specializations; for an inferable typevar, it must not be `never`.)
When I first added support for tracking inferable vs non-inferable
typevars, it seemed like it would be easiest to have separate `Type`
variants for each. The alternative (which lines up with the Δ set in
[POPL15](https://doi.org/10.1145/2676726.2676991)) would be to
explicitly plumb through a list of inferable typevars through our type
property methods. That seemed cumbersome.
In retrospect, that was the wrong decision. We've had to jump through
hoops to translate types between the inferable and non-inferable
variants, which has been quite brittle. Combined with the original point
above, that much of the assignability logic will become more identical
between inferable and non-inferable, there is less justification for the
two `Type` variants. And plumbing an extra `inferable` parameter through
all of these methods turns out to not be as bad as I anticipated.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
## Summary
Use the declared type of variables as type context for the RHS of assignment expressions, e.g.,
```py
x: list[int | str]
x = [1]
reveal_type(x) # revealed: list[int | str]
```
## Summary
Ignore the type context when specializing a generic call if it leads to
an unnecessarily wide return type. For example, [the example mentioned
here](https://github.com/astral-sh/ruff/pull/20796#issuecomment-3403319536)
works as expected after this change:
```py
def id[T](x: T) -> T:
return x
def _(i: int):
x: int | None = id(i)
y: int | None = i
reveal_type(x) # revealed: int
reveal_type(y) # revealed: int
```
I also added extended our usage of `filter_disjoint_elements` to tuple
and typed-dict inference, which resolves
https://github.com/astral-sh/ty/issues/1266.
## Summary
Add support for the `field_specifiers` parameter on
`dataclass_transform` decorator calls.
closes https://github.com/astral-sh/ty/issues/1068
## Conformance test results
All true positives ✔️
## Ecosystem analysis
* `trio`: this is the kind of change that I would expect from this PR.
The code makes use of a dataclass `Outcome` with a `_unwrapped: bool =
attr.ib(default=False, eq=False, init=False)` field that is excluded
from the `__init__` signature, so we now see a bunch of
constructor-call-related errors going away.
* `home-assistant/core`: They have a `domain: str = attr.ib(init=False,
repr=False)` field and then use
```py
@domain.default
def _domain_default(self) -> str:
# …
```
This accesses the `default` attribute on `dataclasses.Field[…]` with a
type of `default: _T | Literal[_MISSING_TYPE.MISSING]`, so we get those
"Object of type `_MISSING_TYPE` is not callable" errors. I don't really
understand how that is supposed to work. Even if `_MISSING_TYPE` would
be absent from that union, what does this try to call? pyright also
issues an error and it doesn't seem to work at runtime? So this looks
like a true positive?
* `attrs`: Similar here. There are some new diagnostics on code that
tries to access `.validator` on a field. This *does* work at runtime,
but I'm not sure how that is supposed to type-check (without a [custom
plugin](2c6c395935/mypy/plugins/attrs.py (L575-L602))).
pyright errors on this as well.
* A handful of new false positives because we don't support `alias` yet
## Test Plan
Updated tests.
Summary
--
This PR unifies the two different ways Ruff and ty construct syntax
errors. Ruff has been storing the primary message in the diagnostic
itself, while ty attached the message to the primary annotation:
```
> ruff check try.py
invalid-syntax: name capture `x` makes remaining patterns unreachable
--> try.py:2:10
|
1 | match 42:
2 | case x: ...
| ^
3 | case y: ...
|
Found 1 error.
> uvx ty check try.py
WARN ty is pre-release software and not ready for production use. Expect to encounter bugs, missing features, and fatal errors.
Checking ------------------------------------------------------------ 1/1 files
error[invalid-syntax]
--> try.py:2:10
|
1 | match 42:
2 | case x: ...
| ^ name capture `x` makes remaining patterns unreachable
3 | case y: ...
|
Found 1 diagnostic
```
I think there are benefits to both approaches, and I do like ty's
version, but I feel like we should pick one (and it might help with
#20901 eventually). I slightly prefer Ruff's version, so I went with
that. Hopefully this isn't too controversial, but I'm happy to close
this if it is.
Note that this shouldn't change any other diagnostic formats in ty
because
[`Diagnostic::primary_message`](98d27c4128/crates/ruff_db/src/diagnostic/mod.rs (L177))
was already falling back to the primary annotation message if the
diagnostic message was empty. As a result, I think this change will
partially resolve the FIXME therein.
Test Plan
--
Existing tests with updated snapshots
This is the ultra-minimal implementation of
* https://github.com/astral-sh/ty/issues/296
that was previously discussed as a good starting point. In particular we
don't actually bother trying to figure out the exact python versions,
but we still mention "hey btw for No Reason At All... you're on python
3.10" when you try to access something that has a definition rooted in
the stdlib that we believe exists sometimes.
This is a drive-by improvement that I stumbled backwards into while
looking into
* https://github.com/astral-sh/ty/issues/296
I was writing some simple tests for "thing not in old version of stdlib"
diagnostics and checked what was added in 3.14, and saw
`compression.zstd` and to my surprise discovered that `import
compression.zstd` and `from compression import zstd` had completely
different quality diagnostics.
This is because `compression` and `compression.zstd` were *both*
introduced in 3.14, and so per VERSIONS policy only an entry for
`compression` was added, and so we don't actually have any definite info
on `compression.zstd` and give up on producing a diagnostic. However the
`from compression import zstd` form fails on looking up `compression`
and we *do* have an exact match for that, so it gets a better
diagnostic!
(aside: I have now learned about the VERSIONS format and I *really* wish
they would just enumerate all the submodules but, oh well!)
The fix is, when handling an import failure, if we fail to find an exact
match *we requery with the parent module*. In cases like
`compression.zstd` this lets us at least identify that, hey, not even
`compression` exists, and luckily that fixes the whole issue. In cases
where the parent module and submodule were introduced at different times
then we may discover that the parent module is in-range and that's fine,
we don't produce the richer stdlib diagnostic.
## Summary
`dataclasses.field` and field-specifier functions of commonly used
libraries like `pydantic`, `attrs`, and `SQLAlchemy` all return the
default type for the field (or `Any`) instead of an actual `Field`
instance, even if this is not what happens at runtime. Let's make use of
this fact and assume that *all* field specifiers return the type of the
default value of the field.
For standard dataclasses, this leads to more or less the same outcome
(see test diff for details), but this change is important for 3rd party
dataclass-transformers.
## Test Plan
Tested the consequences of this change on the field-specifiers branch as
well.
## Summary
Resolves https://github.com/astral-sh/ty/issues/1349.
Fix match statement value patterns to use equality comparison semantics
instead of incorrectly narrowing to literal types directly. Value
patterns use equality for matching, and equality can be overridden, so
we can't always narrow to the matched literal.
## Test Plan
Updated match.md with corrected expected types and an additional example
with explanation
---------
Co-authored-by: David Peter <mail@david-peter.de>
## Summary
Part of astral-sh/ty#1341
The following changes will be made to `Place`.
* Introduce `TypeOrigin`
* `Place::Type` -> `Place::Defined`
* `Place::Unbound` -> `Place::Undefined`
* `Boundness` -> `Definedness`
`TypeOrigin::Declared`+`Definedness::PossiblyUndefined` are patterns
that weren't considered before, but this PR doesn't address them yet,
only refactors.
## Test Plan
Refactoring
A large part of the diff on #20677 just involves threading a new
`inferable` parameter through all of the type property methods. In the
interests of making that PR easier to review, I've pulled that bit out
into here, so that it can be reviewed in isolation. This should be a
pure refactoring, with no logic changes or behavioral changes.
## Summary
If a function is decorated with a decorator that returns a union of
`Callable`s, also treat it as a union of function-like `Callable`s.
Labeling as `internal`, since the previous change has not been released
yet.
## Test Plan
New regression test.
## Summary
Rename "unwrapping" methods on `Type` from e.g.
`Type::into_class_literal` to `Type::as_class_literal`. I personally
find that name more intuitive, since no transformation of any kind is
happening. We are just unwrapping from certain enum variants. An
alternative would be `try_as_class_literal`, which would follow the
[`strum` naming
scheme](https://docs.rs/strum/latest/strum/derive.EnumTryAs.html), but
is slightly longer.
Also rename `Type::into_callable` to `Type::try_upcast_to_callable`.
Note that I intentionally kept names like
`FunctionType::into_callable_type`, because those return `CallableType`,
not `Option<Type<…>>`.
## Test Plan
Pure refactoring
As part of #20598, we added `is_identical_to` methods to
`TypeVarInstance` and `BoundTypeVarInstance`, which compare when two
typevar instances refer to "the same" underlying typevar, even if we
have forced their lazy bounds/constraints as part of marking typevars as
inferable. (Doing so results in a different salsa interned struct ID,
since we've changed the contents of the `bounds_or_constraints` field.)
It turns out that marking typevars as inferable is not the only way that
we might force lazy bounds/constraints; it also happens when we
materialize a type containing a typevar. This surfaced as ecosystem
report failures on #20677.
That means that we need a more long-term fix to this problem.
(`is_identical_to`, and its underlying `original` field, were meant to
be a temporary fix until we removed the `MarkTypeVarsInferable` type
mapping.)
This PR extracts out a separate type (`TypeVarIdentity`) that only
includes the fields that actually inform whether two typevars are "the
same". All other properties of the typevar (default, bounds/constraints,
etc) still live in `TypeVarInstance`. Call sites that care about typevar
identity can now either store just `TypeVarIdentity` (if they never need
access to those other properties), or continue to store
`TypeVarInstance` but pull out its `identity` when performing those "are
they the same typevar" comparisons. (All of this also applies
respectively to `BoundTypeVar{Identity,Instance}`.) In particular,
constraint sets now work on `BoundTypeVarIdentity`, and generic contexts
still _store_ a `BoundTypeVarInstance` (since we might need access to
defaults when specializing), but are keyed on `BoundTypeVarIdentity`.
Generic classes are not allowed to bind or reference a typevar from an
enclosing scope:
```py
def f[T](x: T, y: T) -> None:
class Ok[S]: ...
# error: [invalid-generic-class]
class Bad1[T]: ...
# error: [invalid-generic-class]
class Bad2(Iterable[T]): ...
class C[T]:
class Ok1[S]: ...
# error: [invalid-generic-class]
class Bad1[T]: ...
# error: [invalid-generic-class]
class Bad2(Iterable[T]): ...
```
It does not matter if the class uses PEP 695 or legacy syntax. It does
not matter if the enclosing scope is a generic class or function. The
generic class cannot even _reference_ an enclosing typevar in its base
class list.
This PR adds diagnostics for these cases.
In addition, the PR adds better fallback behavior for generic classes
that violate this rule: any enclosing typevars are not included in the
class's generic context. (That ensures that we don't inadvertently try
to infer specializations for those typevars in places where we
shouldn't.) The `dulwich` ecosystem project has [examples of
this](d912eaaffd/dulwich/config.py (L251))
that were causing new false positives on #20677.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
<!--
Thank you for contributing to Ruff/ty! To help us out with reviewing,
please consider the following:
- Does this pull request include a summary of the change? (See below.)
- Does this pull request include a descriptive title? (Please prefix
with `[ty]` for ty pull
requests.)
- Does this pull request include references to any relevant issues?
-->
## Summary
<!-- What's the purpose of the change? What does it do, and why? -->
This PR implements https://docs.astral.sh/ruff/rules/break-outside-loop/
(F701) as a semantic syntax error.
## Test Plan
<!-- How was it tested? -->
---------
Signed-off-by: 11happy <soni5happy@gmail.com>
Co-authored-by: Brent Westbrook <brentrwestbrook@gmail.com>
## Summary
Treat `Callable`s as bound-method descriptors if `Callable` is the
return type of a decorator that is applied to a function definition. See
the [rendered version of the new test
file](https://github.com/astral-sh/ruff/blob/david/callables-as-descriptors/crates/ty_python_semantic/resources/mdtest/call/callables_as_descriptors.md)
for the full description of this new heuristic.
I could imagine that we want to treat `Callable`s as bound-method
descriptors in other cases as well, but this seems like a step in the
right direction. I am planning to add other "use cases" from
https://github.com/astral-sh/ty/issues/491 to this test suite.
partially addresses https://github.com/astral-sh/ty/issues/491
closes https://github.com/astral-sh/ty/issues/1333
## Ecosystem impact
All positive
* 2961 removed `unsupported-operator` diagnostics on `sympy`, which was
one of the main motivations for implementing this change
* 37 removed `missing-argument` diagnostics, and no added call-error
diagnostics, which is an indicator that this heuristic shouldn't cause
many false positives
* A few removed `possibly-missing-attribute` diagnostics when accessing
attributes like `__name__` on decorated functions. The two added
`unused-ignore-comment` diagnostics are also cases of this.
* One new `invalid-assignment` diagnostic on `dd-trace-py`, which looks
suspicious, but only because our `invalid-assignment` diagnostics are
not great. This is actually a "Implicit shadowing of function"
diagnostic that hides behind the `invalid-assignment` diagnostic,
because a module-global function is being patched through a
`module.func` attribute assignment.
## Test Plan
New Markdown tests.
## Summary
Move the `class_member` function to the `member` module. This allows us
to move the `member` module into the `types` module and to reduce the
visibility of its contents to `pub(super)`. The drawback is that we need
to make `place::place_by_id` public.
## Test Plan
Pure refactoring.
## Summary
When accessing an (instance) attribute on a given class, we were
previously traversing its MRO, and building a union of types (if the
attribute was available on multiple classes in the MRO) until we found a
*definitely bound* symbol. The idea was that possibly unbound symbols in
a subclass might only partially shadow the underlying base class
attribute.
This behavior was problematic for two reasons:
* if the attribute was definitely bound on a class (e.g. `self.x =
None`), we would have stopped iterating, even if there might be a `x:
str | None` declaration in a base class (the bug reported in
https://github.com/astral-sh/ty/issues/1067).
* if the attribute originated from an implicit instance attribute
assignment (e.g. `self.x = 1` in method `Sub.foo`), we might stop
looking and miss another implicit instance attribute assignment in a
base class method (e.g. `self.x = 2` in method `Base.bar`).
With this fix, we still iterate the MRO of the class, but we only stop
iterating if we find a *definitely declared* symbol. In this case, we
only return the declared attribute type. Otherwise, we keep building a
union of inferred attribute types.
The implementation here seemed to be the easiest fix for
https://github.com/astral-sh/ty/issues/1067 that also kept the ecosystem
impact low (the changes that I see all look correct). However, as the
Markdown tests show, there are other things to fix in this area. For
example, we should do a similar thing for *class attributes*. This is
more involved, though (affects many different areas and probably
involves a change to our descriptor protocol implementation), so I'd
like to postpone this to a follow-up.
closes https://github.com/astral-sh/ty/issues/1067
## Test Plan
Updated Markdown tests, including a regression test for
https://github.com/astral-sh/ty/issues/1067.
## Summary
Implements bidirectional type inference using function return type
annotations.
This PR was originally proposed to solve astral-sh/ty#1167, but this
does not fully resolve it on its own.
Additionally, I believe we need to allow dataclasses to generate their
own `__new__` methods, [use constructor return types for
inference](5844c0103d/crates/ty_python_semantic/src/types.rs (L5326-L5328)),
and a mechanism to discard type narrowing like `& ~AlwaysFalsy` if
necessary (at a more general level than this PR).
## Test Plan
`mdtest/bidirectional.md` is added.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
Co-authored-by: Ibraheem Ahmed <ibraheem@ibraheem.ca>
## Summary
Adds a set of basic new tests corresponding to open points in
https://github.com/astral-sh/ty/issues/1327, to document the state of
support for `dataclass_transform`.
## Summary
Type annotations are deferred by default starting with Python 3.14. No
`from __future__ import annotations` import is necessary.
## Test Plan
New Markdown test
## Summary
Simplify and fix the implementation of
`ty_extensions.CallableTypeOf[..]`.
closes https://github.com/astral-sh/ty/issues/1331
## Test Plan
Added regression test.
## Summary
This allows us to handle self-referential bounds/constraints/defaults
without panicking.
Handles more cases from https://github.com/astral-sh/ty/issues/256
This also changes the way we infer the types of legacy TypeVars. Rather
than understanding a constructor call to `typing[_extension].TypeVar`
inside of any (arbitrarily nested) expression, and having to use a
special `assigned_to` field of the semantic index to try to best-effort
figure out what name the typevar was assigned to, we instead understand
the creation of a legacy `TypeVar` only in the supported syntactic
position (RHS of a simple un-annotated assignment with one target). In
any other position, we just infer it as creating an opaque instance of
`typing.TypeVar`. (This behavior matches all other type checkers.)
So we now special-case TypeVar creation in `TypeInferenceBuilder`, as a
special case of an assignment definition, rather than deeper inside call
binding. This does mean we re-implement slightly more of
argument-parsing, but in practice this is minimal and easy to handle
correctly.
This is easier to implement if we also make the RHS of a simple (no
unpacking) one-target assignment statement no longer a standalone
expression. Which is fine to do, because simple one-target assignments
don't need to infer the RHS more than once. This is a bonus performance
(0-3% across various projects) and significant memory-usage win, since
most assignment statements are simple one-target assignment statements,
meaning we now create many fewer standalone-expression salsa
ingredients.
This change does mean that inference of manually-constructed
`TypeAliasType` instances can no longer find its Definition in
`assigned_to`, which regresses go-to-definition for these aliases. In a
future PR, `TypeAliasType` will receive the same treatment that
`TypeVar` did in this PR (moving its special-case inference into
`TypeInferenceBuilder` and supporting it only in the correct syntactic
position, and lazily inferring its value type to support recursion),
which will also fix the go-to-definition regression. (I decided a
temporary edge-case regression is better in this case than doubling the
size of this PR.)
This PR also tightens up and fixes various aspects of the validation of
`TypeVar` creation, as seen in the tests.
We still (for now) treat all typevars as instances of `typing.TypeVar`,
even if they were created using `typing_extensions.TypeVar`. This means
we'll wrongly error on e.g. `T.__default__` on Python 3.11, even if `T`
is a `typing_extensions.TypeVar` instance at runtime. We share this
wrong behavior with both mypy and pyrefly. It will be easier to fix
after we pull in https://github.com/python/typeshed/pull/14840.
There are some issues that showed up here with typevar identity and
`MarkTypeVarsInferable`; the fix here (using the new `original` field
and `is_identical_to` methods on `BoundTypeVarInstance` and
`TypeVarInstance`) is a bit kludgy, but it can go away when we eliminate
`MarkTypeVarsInferable`.
## Test Plan
Added and updated mdtests.
### Conformance suite impact
The impact here is all positive:
* We now correctly error on a legacy TypeVar with exactly one constraint
type given.
* We now correctly error on a legacy TypeVar with both an upper bound
and constraints specified.
### Ecosystem impact
Basically none; in the setuptools case we just issue slightly different
errors on an invalid TypeVar definition, due to the modified validation
code.
---------
Co-authored-by: Alex Waygood <Alex.Waygood@Gmail.com>
## Summary
Avoid literal promotion when a literal type annotation is provided, e.g.,
```py
x: list[Literal[1]] = [1]
```
Resolves https://github.com/astral-sh/ty/issues/1198. This does not fix
issue https://github.com/astral-sh/ty/issues/1284, but it does make it
more relevant because after this change, it is possible to directly
instantiate a generic type with a literal specialization.
