mirror of
https://github.com/astral-sh/ruff.git
synced 2025-10-27 18:36:35 +00:00
11 commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
Dhruv Manilawala
|
4ca38b2974
|
[ty] Unpack variadic argument type in specialization (#20130)
Some checks are pending
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / cargo fuzz build (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / Fuzz for new ty panics (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks-instrumented (push) Blocked by required conditions
CI / benchmarks-walltime (push) Blocked by required conditions
[ty Playground] Release / publish (push) Waiting to run
## Summary This PR fixes various TODOs around overload call when a variadic argument is used. The reason this bug existed is because the specialization wouldn't account for unpacking the type of the variadic argument. This is fixed by expanding `MatchedArgument` to contain the type of that argument _only_ when it is a variadic argument. The reason is that there's a split for when the argument type is inferred -- the non-variadic arguments are inferred using `infer_argument_types` _after_ parameter matching while the variadic argument type is inferred _during_ the parameter matching. And, the `MatchedArgument` is populated _during_ parameter matching which means the unpacking would need to happen during parameter matching. This split seems a bit inconsistent but I don't want to spend a lot of time on trying to merge them such that all argument type inference happens in a single place. I might look into it while adding support for `**kwargs`. ## Test Plan Update existing tests by resolving the todos. The ecosystem changes looks correct to me except for the `slice` call but it seems that it's unrelated to this PR as we infer `slice[Any, Any, Any]` for a `slice(1, 2, 3)` call on `main` as well ([playground](https://play.ty.dev/9eacce00-c7d5-4dd5-a932-4265cb2bb4f6)). |
||
|
Alex Waygood
|
ce1dc21e7e
|
[ty] Fix the inferred interface of specialized generic protocols (#19866) | ||
|
Dhruv Manilawala
|
376e3ff395
|
[ty] Limit argument expansion size for overload call evaluation (#20041)
## Summary This PR limits the argument type expansion size for an overload call evaluation to 512. The limit chosen is arbitrary but I've taken the 256 limit from Pyright into account and bumped it x2 to start with. Initially, I actually started out by trying to refactor the entire argument type expansion to be lazy. Currently, expanding a single argument at any position eagerly creates the combination (argument lists) and returns that (`Vec<CallArguments>`) but I thought we could make it lazier by converting the return type of `expand` from `Iterator<Item = Vec<CallArguments>>` to `Iterator<Item = Iterator<Item = CallArguments>>` but that's proving to be difficult to implement mainly because we **need** to maintain the previous expansion to generate the next expansion which is the main reason to use `std::iter::successors` in the first place. Another approach would be to eagerly expand all the argument types and then use the `combinations` from `itertools` to generate the combinations but we would need to find the "boundary" between arguments lists produced from expanding argument at position 1 and position 2 because that's important for the algorithm. Closes: https://github.com/astral-sh/ty/issues/868 ## Test Plan Add test case to demonstrate the limit along with the diagnostic snapshot stating that the limit has been reached. |
||
|
Dhruv Manilawala
|
d43a3d34dd
|
[ty] Avoid unnecessary argument type expansion (#19999)
## Summary Part of: https://github.com/astral-sh/ty/issues/868 This PR adds a heuristic to avoid argument type expansion if it's going to eventually lead to no matching overload. This is done by checking whether the non-expandable argument types are assignable to the corresponding annotated parameter type. If one of them is not assignable to all of the remaining overloads, then argument type expansion isn't going to help. ## Test Plan Add mdtest that would otherwise take a long time because of the number of arguments that it would need to expand (30). |
||
|
Dhruv Manilawala
|
f019cfd15f
|
[ty] Use specialized parameter type for overload filter (#19964)
## Summary Closes: https://github.com/astral-sh/ty/issues/669 (This turned out to be simpler that I thought :)) ## Test Plan Update existing test cases. ### Ecosystem report Most of them are basically because ty has now started inferring more precise types for the return type to an overloaded call and a lot of the types are defined using type aliases, here's some examples: <details><summary>Details</summary> <p> > attrs (https://github.com/python-attrs/attrs) > + tests/test_make.py:146:14: error[unresolved-attribute] Type `Literal[42]` has no attribute `default` > - Found 555 diagnostics > + Found 556 diagnostics This is accurate now that we infer the type as `Literal[42]` instead of `Unknown` (Pyright infers it as `int`) > optuna (https://github.com/optuna/optuna) > + optuna/_gp/search_space.py:181:53: error[invalid-argument-type] Argument to function `_round_one_normalized_param` is incorrect: Expected `tuple[int | float, int | float]`, found `tuple[Unknown | ndarray[Unknown, <class 'float'>], Unknown | ndarray[Unknown, <class 'float'>]]` > + optuna/_gp/search_space.py:181:83: error[invalid-argument-type] Argument to function `_round_one_normalized_param` is incorrect: Expected `int | float`, found `Unknown | ndarray[Unknown, <class 'float'>]` > + tests/gp_tests/test_search_space.py:109:13: error[invalid-argument-type] Argument to function `_unnormalize_one_param` is incorrect: Expected `tuple[int | float, int | float]`, found `Unknown | ndarray[Unknown, <class 'float'>]` > + tests/gp_tests/test_search_space.py:110:13: error[invalid-argument-type] Argument to function `_unnormalize_one_param` is incorrect: Expected `int | float`, found `Unknown | ndarray[Unknown, <class 'float'>]` > - Found 559 diagnostics > + Found 563 diagnostics Same as above where ty is now inferring a more precise type like `Unknown | ndarray[tuple[int, int], <class 'float'>]` instead of just `Unknown` as before > jinja (https://github.com/pallets/jinja) > + src/jinja2/bccache.py:298:39: error[invalid-argument-type] Argument to bound method `write_bytecode` is incorrect: Expected `IO[bytes]`, found `_TemporaryFileWrapper[str]` > - Found 186 diagnostics > + Found 187 diagnostics This requires support for type aliases to match the correct overload. > hydra-zen (https://github.com/mit-ll-responsible-ai/hydra-zen) > + src/hydra_zen/wrapper/_implementations.py:945:16: error[invalid-return-type] Return type does not match returned value: expected `DataClass_ | type[@Todo(type[T] for protocols)] | ListConfig | DictConfig`, found `@Todo(unsupported type[X] special form) | (((...) -> Any) & dict[Unknown, Unknown]) | (DataClass_ & dict[Unknown, Unknown]) | dict[Any, Any] | (ListConfig & dict[Unknown, Unknown]) | (DictConfig & dict[Unknown, Unknown]) | (((...) -> Any) & list[Unknown]) | (DataClass_ & list[Unknown]) | list[Any] | (ListConfig & list[Unknown]) | (DictConfig & list[Unknown])` > + tests/annotations/behaviors.py:60:28: error[call-non-callable] Object of type `Path` is not callable > + tests/annotations/behaviors.py:64:21: error[call-non-callable] Object of type `Path` is not callable > + tests/annotations/declarations.py:167:17: error[call-non-callable] Object of type `Path` is not callable > + tests/annotations/declarations.py:524:17: error[unresolved-attribute] Type `<class 'int'>` has no attribute `_target_` > - Found 561 diagnostics > + Found 566 diagnostics Same as above, this requires support for type aliases to match the correct overload. > paasta (https://github.com/yelp/paasta) > + paasta_tools/utils.py:4188:19: warning[redundant-cast] Value is already of type `list[str]` > - Found 888 diagnostics > + Found 889 diagnostics This is correct. > colour (https://github.com/colour-science/colour) > + colour/plotting/diagrams.py:448:13: error[invalid-argument-type] Argument to bound method `__init__` is incorrect: Expected `Sequence[@Todo(Support for `typing.TypeAlias`)]`, found `ndarray[tuple[int, int, int], dtype[Unknown]]` > + colour/plotting/diagrams.py:462:13: error[invalid-argument-type] Argument to bound method `__init__` is incorrect: Expected `Sequence[@Todo(Support for `typing.TypeAlias`)]`, found `ndarray[tuple[int, int, int], dtype[Unknown]]` > + colour/plotting/models.py:419:13: error[invalid-argument-type] Argument to bound method `__init__` is incorrect: Expected `Sequence[@Todo(Support for `typing.TypeAlias`)]`, found `ndarray[tuple[int, int, int], dtype[Unknown]]` > + colour/plotting/temperature.py:230:9: error[invalid-argument-type] Argument to bound method `__init__` is incorrect: Expected `Sequence[@Todo(Support for `typing.TypeAlias`)]`, found `ndarray[tuple[int, int, int], dtype[Unknown]]` > + colour/plotting/temperature.py:474:13: error[invalid-argument-type] Argument to bound method `__init__` is incorrect: Expected `Sequence[@Todo(Support for `typing.TypeAlias`)]`, found `ndarray[tuple[int, int, int], dtype[Unknown]]` > + colour/plotting/temperature.py:495:17: error[invalid-argument-type] Argument to bound method `__init__` is incorrect: Expected `Sequence[@Todo(Support for `typing.TypeAlias`)]`, found `ndarray[tuple[int, int, int], dtype[Unknown]]` > + colour/plotting/temperature.py:513:13: error[invalid-argument-type] Argument to bound method `text` is incorrect: Expected `int | float`, found `ndarray[@Todo(Support for `typing.TypeAlias`), dtype[Unknown]]` > + colour/plotting/temperature.py:514:13: error[invalid-argument-type] Argument to bound method `text` is incorrect: Expected `int | float`, found `ndarray[@Todo(Support for `typing.TypeAlias`), dtype[Unknown]]` > - Found 480 diagnostics > + Found 488 diagnostics Most of them are correct except for the last two diagnostics which I'm not sure what's happening, it's trying to index into an `np.ndarray` type (which is inferred correctly) but I think it might be picking up an incorrect overload for the `__getitem__` method. Scipy's diagnostics also requires support for type alises to pick the correct overload. </p> </details> |
||
|
Douglas Creager
|
7673d46b71
|
[ty] Splat variadic arguments into parameter list (#18996)
This PR updates our call binding logic to handle splatted arguments. Complicating matters is that we have separated call bind analysis into two phases: parameter matching and type checking. Parameter matching looks at the arity of the function signature and call site, and assigns arguments to parameters. Importantly, we don't yet know the type of each argument! This is needed so that we can decide whether to infer the type of each argument as a type form or value form, depending on the requirements of the parameter that the argument was matched to. This is an issue when splatting an argument, since we need to know how many elements the splatted argument contains to know how many positional parameters to match it against. And to know how many elements the splatted argument has, we need to know its type. To get around this, we now make the assumption that splatted arguments can only be used with value-form parameters. (If you end up splatting an argument into a type-form parameter, we will silently pass in its value-form type instead.) That allows us to preemptively infer the (value-form) type of any splatted argument, so that we have its arity available during parameter matching. We defer inference of non-splatted arguments until after parameter matching has finished, as before. We reuse a lot of the new tuple machinery to make this happen — in particular resizing the tuple spec representing the number of arguments passed in with the tuple length representing the number of parameters the splat was matched with. This work also shows that we might need to change how we are performing argument expansion during overload resolution. At the moment, when we expand parameters, we assume that each argument will still be matched to the same parameters as before, and only retry the type-checking phase. With splatted arguments, this is no longer the case, since the inferred arity of each union element might be different than the arity of the union as a whole, which can affect how many parameters the splatted argument is matched to. See the regression test case in `mdtest/call/function.md` for more details. |
||
|
David Peter
|
dc66019fbc
|
[ty] Expansion of enums into unions of literals (#19382)
## Summary
Implement expansion of enums into unions of enum literals (and the
reverse operation). For the enum below, this allows us to understand
that `Color = Literal[Color.RED, Color.GREEN, Color.BLUE]`, or that
`Color & ~Literal[Color.RED] = Literal[Color.GREEN, Color.BLUE]`. This
helps in exhaustiveness checking, which is why we see some removed
`assert_never` false positives. And since exhaustiveness checking also
helps with understanding terminal control flow, we also see a few
removed `invalid-return-type` and `possibly-unresolved-reference` false
positives. This PR also adds expansion of enums in overload resolution
and type narrowing constructs.
```py
from enum import Enum
from typing_extensions import Literal, assert_never
from ty_extensions import Intersection, Not, static_assert, is_equivalent_to
class Color(Enum):
RED = 1
GREEN = 2
BLUE = 3
type Red = Literal[Color.RED]
type Green = Literal[Color.GREEN]
type Blue = Literal[Color.BLUE]
static_assert(is_equivalent_to(Red | Green | Blue, Color))
static_assert(is_equivalent_to(Intersection[Color, Not[Red]], Green | Blue))
def color_name(color: Color) -> str: # no error here (we detect that this can not implicitly return None)
if color is Color.RED:
return "Red"
elif color is Color.GREEN:
return "Green"
elif color is Color.BLUE:
return "Blue"
else:
assert_never(color) # no error here
```
## Performance
I avoided an initial regression here for large enums, but the
`UnionBuilder` and `IntersectionBuilder` parts can certainly still be
optimized. We might want to use the same technique that we also use for
unions of other literals. I didn't see any problems in our benchmarks so
far, so this is not included yet.
## Test Plan
Many new Markdown tests
|
||
|
David Peter
|
a1edb69ea5
|
[ty] Enum literal types (#19328)
## Summary
Add a new `Type::EnumLiteral(…)` variant and infer this type for member
accesses on enums.
**Example**: No more `@Todo` types here:
```py
from enum import Enum
class Answer(Enum):
YES = 1
NO = 2
def is_yes(self) -> bool:
return self == Answer.YES
reveal_type(Answer.YES) # revealed: Literal[Answer.YES]
reveal_type(Answer.YES == Answer.NO) # revealed: Literal[False]
reveal_type(Answer.YES.is_yes()) # revealed: bool
```
## Test Plan
* Many new Markdown tests for the new type variant
* Added enum literal types to property tests, ran property tests
## Ecosystem analysis
Summary:
Lots of false positives removed. All of the new diagnostics are
either new true positives (the majority) or known problems. Click for
detailed analysis</summary>
Details:
```diff
AutoSplit (https://github.com/Toufool/AutoSplit)
+ error[call-non-callable] src/capture_method/__init__.py:137:9: Method `__getitem__` of type `bound method CaptureMethodDict.__getitem__(key: Never, /) -> type[CaptureMethodBase]` is not callable on object of type `CaptureMethodDict`
+ error[call-non-callable] src/capture_method/__init__.py:147:9: Method `__getitem__` of type `bound method CaptureMethodDict.__getitem__(key: Never, /) -> type[CaptureMethodBase]` is not callable on object of type `CaptureMethodDict`
+ error[call-non-callable] src/capture_method/__init__.py:148:1: Method `__getitem__` of type `bound method CaptureMethodDict.__getitem__(key: Never, /) -> type[CaptureMethodBase]` is not callable on object of type `CaptureMethodDict`
```
New true positives. That `__getitem__` method is apparently annotated
with `Never` to prevent developers from using it.
```diff
dd-trace-py (https://github.com/DataDog/dd-trace-py)
+ error[invalid-assignment] ddtrace/vendor/psutil/_common.py:29:5: Object of type `None` is not assignable to `Literal[AddressFamily.AF_INET6]`
+ error[invalid-assignment] ddtrace/vendor/psutil/_common.py:33:5: Object of type `None` is not assignable to `Literal[AddressFamily.AF_UNIX]`
```
Arguably true positives:
|
||
|
Dhruv Manilawala
|
22177e6915
|
[ty] Surface matched overload diagnostic directly (#18452)
Some checks are pending
CI / cargo fuzz build (push) Blocked by required conditions
CI / Determine changes (push) Waiting to run
CI / cargo fmt (push) Waiting to run
CI / cargo clippy (push) Blocked by required conditions
CI / cargo test (linux) (push) Blocked by required conditions
CI / cargo test (linux, release) (push) Blocked by required conditions
CI / cargo test (windows) (push) Blocked by required conditions
CI / cargo test (wasm) (push) Blocked by required conditions
CI / cargo build (release) (push) Waiting to run
CI / cargo build (msrv) (push) Blocked by required conditions
CI / fuzz parser (push) Blocked by required conditions
CI / test scripts (push) Blocked by required conditions
CI / ecosystem (push) Blocked by required conditions
CI / Fuzz for new ty panics (push) Blocked by required conditions
CI / cargo shear (push) Blocked by required conditions
CI / python package (push) Waiting to run
CI / pre-commit (push) Waiting to run
CI / mkdocs (push) Waiting to run
CI / formatter instabilities and black similarity (push) Blocked by required conditions
CI / test ruff-lsp (push) Blocked by required conditions
CI / check playground (push) Blocked by required conditions
CI / benchmarks-instrumented (push) Blocked by required conditions
CI / benchmarks-walltime (push) Blocked by required conditions
[ty Playground] Release / publish (push) Waiting to run
## Summary
This PR resolves the way diagnostics are reported for an invalid call to
an overloaded function.
If any of the steps in the overload call evaluation algorithm yields a
matching overload but it's type checking that failed, the
`no-matching-overload` diagnostic is incorrect because there is a
matching overload, it's the arguments passed that are invalid as per the
signature. So, this PR improves that by surfacing the diagnostics on the
matching overload directly.
It also provides additional context, specifically the matching overload
where this error occurred and other non-matching overloads. Consider the
following example:
```py
from typing import overload
@overload
def f() -> None: ...
@overload
def f(x: int) -> int: ...
@overload
def f(x: int, y: int) -> int: ...
def f(x: int | None = None, y: int | None = None) -> int | None:
return None
f("a")
```
We get:
<img width="857" alt="Screenshot 2025-06-18 at 11 07 10"
src="https://github.com/user-attachments/assets/8dbcaf13-2a74-4661-aa94-1225c9402ea6"
/>
## Test Plan
Update test cases, resolve existing todos and validate the updated
snapshots.
|
||
|
Dhruv Manilawala
|
c7e020df6b
|
[ty] Filter overloads based on Any / Unknown (#18607)
## Summary Closes: astral-sh/ty#552 This PR adds support for step 5 of the overload call evaluation algorithm which specifies: > For all arguments, determine whether all possible materializations of the argument’s type are > assignable to the corresponding parameter type for each of the remaining overloads. If so, > eliminate all of the subsequent remaining overloads. The algorithm works in two parts: 1. Find out the participating parameter indexes. These are the parameters that aren't gradual equivalent to one or more parameter types at the same index in other overloads. 2. Loop over each overload and check whether that would be the _final_ overload for the argument types i.e., the remaining overloads will never be matched against these argument types For step 1, the participating parameter indexes are computed by just comparing whether all the parameter types at the corresponding index for all the overloads are **gradual equivalent**. The step 2 of the algorithm used is described in [this comment](https://github.com/astral-sh/ty/issues/552#issuecomment-2969165421). ## Test Plan Update the overload call tests. |
||
|
Dhruv Manilawala
|
7ea773daf2
|
[ty] Argument type expansion for overload call evaluation (#18382)
## Summary
Part of astral-sh/ty#104, closes: astral-sh/ty#468
This PR implements the argument type expansion which is step 3 of the
overload call evaluation algorithm.
Specifically, this step needs to be taken if type checking resolves to
no matching overload and there are argument types that can be expanded.
## Test Plan
Add new test cases.
## Ecosystem analysis
This PR removes 174 `no-matching-overload` false positives -- I looked
at a lot of them and they all are false positives.
One thing that I'm not able to understand is that in
|