diff --git a/crates/ty_python_semantic/resources/mdtest/call/callables_as_descriptors.md b/crates/ty_python_semantic/resources/mdtest/call/callables_as_descriptors.md new file mode 100644 index 0000000000..ca780a7b1a --- /dev/null +++ b/crates/ty_python_semantic/resources/mdtest/call/callables_as_descriptors.md @@ -0,0 +1,185 @@ +# Callables as descriptors? + + + +```toml +[environment] +python-version = "3.14" +``` + +## Introduction + +Some common callable objects (functions, lambdas) are also bound-method descriptors. That is, they +have a `__get__` method which returns a bound-method object that binds the receiver instance to the +first argument (and thus the bound-method object has a different signature, lacking the first +argument): + +```py +from ty_extensions import CallableTypeOf +from typing import Callable + +class C1: + def method(self: C1, x: int) -> str: + return str(x) + +def _( + accessed_on_class: CallableTypeOf[C1.method], + accessed_on_instance: CallableTypeOf[C1().method], +): + reveal_type(accessed_on_class) # revealed: (self: C1, x: int) -> str + reveal_type(accessed_on_instance) # revealed: (x: int) -> str +``` + +Other callable objects (`staticmethod` objects, instances of classes with a `__call__` method but no +dedicated `__get__` method) are *not* bound-method descriptors. If accessed as class attributes via +an instance, they are simply themselves: + +```py +class NonDescriptorCallable2: + def __call__(self, c2: C2, x: int) -> str: + return str(x) + +class C2: + non_descriptor_callable: NonDescriptorCallable2 = NonDescriptorCallable2() + +def _( + accessed_on_class: CallableTypeOf[C2.non_descriptor_callable], + accessed_on_instance: CallableTypeOf[C2().non_descriptor_callable], +): + reveal_type(accessed_on_class) # revealed: (c2: C2, x: int) -> str + reveal_type(accessed_on_instance) # revealed: (c2: C2, x: int) -> str +``` + +Both kinds of objects can inhabit the same `Callable` type: + +```py +class NonDescriptorCallable3: + def __call__(self, c3: C3, x: int) -> str: + return str(x) + +class C3: + def method(self: C3, x: int) -> str: + return str(x) + + non_descriptor_callable: NonDescriptorCallable3 = NonDescriptorCallable3() + + callable_m: Callable[[C3, int], str] = method + callable_n: Callable[[C3, int], str] = non_descriptor_callable +``` + +However, when they are accessed on instances of `C3`, they have different signatures: + +```py +def _( + method_accessed_on_instance: CallableTypeOf[C3().method], + callable_accessed_on_instance: CallableTypeOf[C3().non_descriptor_callable], +): + reveal_type(method_accessed_on_instance) # revealed: (x: int) -> str + reveal_type(callable_accessed_on_instance) # revealed: (c3: C3, x: int) -> str +``` + +This leaves the question how the `callable_m` and `callable_n` attributes should be treated when +accessed on instances of `C3`. If we treat `Callable` as being equivalent to a protocol that defines +a `__call__` method (and no `__get__` method), then they should show no bound-method behavior. This +is what we currently do: + +```py +reveal_type(C3().callable_m) # revealed: (C3, int, /) -> str +reveal_type(C3().callable_n) # revealed: (C3, int, /) -> str +``` + +However, this leads to unsoundness: `C3().callable_m` is actually `C3.method` which *is* a +bound-method descriptor. We currently allow the following call, which will fail at runtime: + +```py +C3().callable_m(C3(), 1) # runtime error! ("takes 2 positional arguments but 3 were given") +``` + +If we were to treat `Callable`s as bound-method descriptors, then the signatures of `callable_m` and +`callable_n` when accessed on instances would bind the `self` argument: + +- `C3().callable_m`: `(x: int) -> str` +- `C3().callable_n`: `(x: int) -> str` + +This would be equally unsound, because now we would allow a call to `C3().callable_n(1)` which would +also fail at runtime. + +There is no perfect solution here, but we can use some heuristics to improve the situation for +certain use cases (at the cost of purity and simplicity). + +## Use case: Decorating a method with a `Callable`-typed decorator + +A commonly used pattern in the ecosystem is to use a `Callable`-typed decorator on a method with the +intention that it shouldn't influence the method's descriptor behavior. For example: + +```py +from typing import Callable + +# TODO: this could use a generic signature, but we don't support +# `ParamSpec` and solving of typevars inside `Callable` types. +def memoize(f: Callable[[C1, int], str]) -> Callable[[C1, int], str]: + raise NotImplementedError + +class C1: + def method(self, x: int) -> str: + return str(x) + + @memoize + def method_decorated(self, x: int) -> str: + return str(x) + +C1().method(1) + +C1().method_decorated(1) +``` + +This also works with an argumentless `Callable` annotation: + +```py +def memoize2(f: Callable) -> Callable: + raise NotImplementedError + +class C2: + @memoize2 + def method_decorated(self, x: int) -> str: + return str(x) + +C2().method_decorated(1) +``` + +Note that we currently only apply this heuristic when calling a function such as `memoize` via the +decorator syntax. This is inconsistent, of course, because the above *should* be equivalent to the +following, but here we emit errors: + +```py +def memoize3(f: Callable[[C3, int], str]) -> Callable[[C3, int], str]: + raise NotImplementedError + +class C3: + def method(self, x: int) -> str: + return str(x) + + method_decorated = memoize3(method) + +# error: [missing-argument] +# error: [invalid-argument-type] +C3().method_decorated(1) +``` + +The reason for this is that the heuristic is problematic. We don't *know* that the `Callable` in the +return type of `memoize` is actually related to the method that we pass in. But when `memoize` is +applied as a decorator, it is reasonable to assume so. In general, a function call might however +return a `Callable` that is unrelated to the argument passed in. And here, it seems more +reasonable/safe to treat the `Callable` as a non-descriptor: + +```py +def convert_int_function(c: Callable[[int], int]) -> Callable[[int], str]: + return lambda x: str(c(x)) + +class C4: + abs = convert_int_function(abs) + round = convert_int_function(round) + +reveal_type(C4().abs) # revealed: Unknown | ((int, /) -> str) +reveal_type(C4().round) # revealed: Unknown | ((int, /) -> str) +``` diff --git a/crates/ty_python_semantic/src/types.rs b/crates/ty_python_semantic/src/types.rs index 894ef87710..4f0ed60d05 100644 --- a/crates/ty_python_semantic/src/types.rs +++ b/crates/ty_python_semantic/src/types.rs @@ -1002,6 +1002,13 @@ impl<'db> Type<'db> { } } + pub(crate) const fn unwrap_as_callable_type(self) -> Option> { + match self { + Type::Callable(callable_type) => Some(callable_type), + _ => None, + } + } + pub(crate) const fn expect_dynamic(self) -> DynamicType<'db> { self.into_dynamic() .expect("Expected a Type::Dynamic variant") diff --git a/crates/ty_python_semantic/src/types/infer/builder.rs b/crates/ty_python_semantic/src/types/infer/builder.rs index 1e93f456ca..4161fa8e05 100644 --- a/crates/ty_python_semantic/src/types/infer/builder.rs +++ b/crates/ty_python_semantic/src/types/infer/builder.rs @@ -2175,7 +2175,26 @@ impl<'db, 'ast> TypeInferenceBuilder<'db, 'ast> { .try_call(self.db(), &CallArguments::positional([inferred_ty])) .map(|bindings| bindings.return_type(self.db())) { - Ok(return_ty) => return_ty, + Ok(return_ty) => { + let is_input_function_like = inferred_ty.is_function_literal() + || inferred_ty + .unwrap_as_callable_type() + .is_some_and(|callable| callable.is_function_like(self.db())); + if is_input_function_like + && let Some(callable_type) = return_ty.unwrap_as_callable_type() + { + // When a method on a class is decorated with a function that returns a `Callable`, assume that + // the returned callable is also function-like. See "Decorating a method with a `Callable`-typed + // decorator" in `callables_as_descriptors.md` for the extended explanation. + Type::Callable(CallableType::new( + self.db(), + callable_type.signatures(self.db()), + true, + )) + } else { + return_ty + } + } Err(CallError(_, bindings)) => { bindings.report_diagnostics(&self.context, (*decorator_node).into()); bindings.return_type(self.db()) diff --git a/crates/ty_python_semantic/src/types/signatures.rs b/crates/ty_python_semantic/src/types/signatures.rs index 039b89a6eb..20d70ee2c6 100644 --- a/crates/ty_python_semantic/src/types/signatures.rs +++ b/crates/ty_python_semantic/src/types/signatures.rs @@ -579,7 +579,13 @@ impl<'db> Signature<'db> { } pub(crate) fn bind_self(&self, db: &'db dyn Db, self_type: Option>) -> Self { - let mut parameters = Parameters::new(self.parameters().iter().skip(1).cloned()); + let mut parameters = self.parameters.iter().cloned().peekable(); + + if parameters.peek().is_some_and(Parameter::is_positional) { + parameters.next(); + } + + let mut parameters = Parameters::new(parameters); let mut return_ty = self.return_ty; if let Some(self_type) = self_type { parameters = parameters.apply_type_mapping_impl(