Merge f2e2b2f000 into 7064c38e53

crates/ty_python_semantic/resources/mdtest/call/callables_as_descriptors.md

@@ -0,0 +1,185 @@
+# Callables as descriptors?
+
+<!-- blacken-docs:off -->
+
+```toml
+[environment]
+python-version = "3.14"
+```
+
+## Introduction
+
+Some common callable objects (functions, lambdas) are also bound-method descriptors. That is, they
+have a `__get__` method which returns a bound-method object that binds the receiver instance to the
+first argument (and thus the bound-method object has a different signature, lacking the first
+argument):
+
+```py
+from ty_extensions import CallableTypeOf
+from typing import Callable
+
+class C1:
+    def method(self: C1, x: int) -> str:
+        return str(x)
+
+def _(
+    accessed_on_class: CallableTypeOf[C1.method],
+    accessed_on_instance: CallableTypeOf[C1().method],
+):
+    reveal_type(accessed_on_class)     # revealed: (self: C1, x: int) -> str
+    reveal_type(accessed_on_instance)  # revealed:           (x: int) -> str
+```
+
+Other callable objects (`staticmethod` objects, instances of classes with a `__call__` method but no
+dedicated `__get__` method) are *not* bound-method descriptors. If accessed as class attributes via
+an instance, they are simply themselves:
+
+```py
+class NonDescriptorCallable2:
+    def __call__(self, c2: C2, x: int) -> str:
+        return str(x)
+
+class C2:
+    non_descriptor_callable: NonDescriptorCallable2 = NonDescriptorCallable2()
+
+def _(
+    accessed_on_class: CallableTypeOf[C2.non_descriptor_callable],
+    accessed_on_instance: CallableTypeOf[C2().non_descriptor_callable],
+):
+    reveal_type(accessed_on_class)     # revealed: (c2: C2, x: int) -> str
+    reveal_type(accessed_on_instance)  # revealed: (c2: C2, x: int) -> str
+```
+
+Both kinds of objects can inhabit the same `Callable` type:
+
+```py
+class NonDescriptorCallable3:
+    def __call__(self, c3: C3, x: int) -> str:
+        return str(x)
+
+class C3:
+    def method(self: C3, x: int) -> str:
+        return str(x)
+
+    non_descriptor_callable: NonDescriptorCallable3 = NonDescriptorCallable3()
+
+    callable_m: Callable[[C3, int], str] = method
+    callable_n: Callable[[C3, int], str] = non_descriptor_callable
+```
+
+However, when they are accessed on instances of `C3`, they have different signatures:
+
+```py
+def _(
+    method_accessed_on_instance: CallableTypeOf[C3().method],
+    callable_accessed_on_instance: CallableTypeOf[C3().non_descriptor_callable],
+):
+    reveal_type(method_accessed_on_instance)    # revealed:         (x: int) -> str
+    reveal_type(callable_accessed_on_instance)  # revealed: (c3: C3, x: int) -> str
+```
+
+This leaves the question how the `callable_m` and `callable_n` attributes should be treated when
+accessed on instances of `C3`. If we treat `Callable` as being equivalent to a protocol that defines
+a `__call__` method (and no `__get__` method), then they should show no bound-method behavior. This
+is what we currently do:
+
+```py
+reveal_type(C3().callable_m)  # revealed: (C3, int, /) -> str
+reveal_type(C3().callable_n)  # revealed: (C3, int, /) -> str
+```
+
+However, this leads to unsoundness: `C3().callable_m` is actually `C3.method` which *is* a
+bound-method descriptor. We currently allow the following call, which will fail at runtime:
+
+```py
+C3().callable_m(C3(), 1)  # runtime error! ("takes 2 positional arguments but 3 were given")
+```
+
+If we were to treat `Callable`s as bound-method descriptors, then the signatures of `callable_m` and
+`callable_n` when accessed on instances would bind the `self` argument:
+
+- `C3().callable_m`: `(x: int) -> str`
+- `C3().callable_n`: `(x: int) -> str`
+
+This would be equally unsound, because now we would allow a call to `C3().callable_n(1)` which would
+also fail at runtime.
+
+There is no perfect solution here, but we can use some heuristics to improve the situation for
+certain use cases (at the cost of purity and simplicity).
+
+## Use case: Decorating a method with a `Callable`-typed decorator
+
+A commonly used pattern in the ecosystem is to use a `Callable`-typed decorator on a method with the
+intention that it shouldn't influence the method's descriptor behavior. For example:
+
+```py
+from typing import Callable
+
+# TODO: this could use a generic signature, but we don't support
+# `ParamSpec` and solving of typevars inside `Callable` types.
+def memoize(f: Callable[[C1, int], str]) -> Callable[[C1, int], str]:
+    raise NotImplementedError
+
+class C1:
+    def method(self, x: int) -> str:
+        return str(x)
+
+    @memoize
+    def method_decorated(self, x: int) -> str:
+        return str(x)
+
+C1().method(1)
+
+C1().method_decorated(1)
+```
+
+This also works with an argumentless `Callable` annotation:
+
+```py
+def memoize2(f: Callable) -> Callable:
+    raise NotImplementedError
+
+class C2:
+    @memoize2
+    def method_decorated(self, x: int) -> str:
+        return str(x)
+
+C2().method_decorated(1)
+```
+
+Note that we currently only apply this heuristic when calling a function such as `memoize` via the
+decorator syntax. This is inconsistent, of course, because the above *should* be equivalent to the
+following, but here we emit errors:
+
+```py
+def memoize3(f: Callable[[C3, int], str]) -> Callable[[C3, int], str]:
+    raise NotImplementedError
+
+class C3:
+    def method(self, x: int) -> str:
+        return str(x)
+
+    method_decorated = memoize3(method)
+
+# error: [missing-argument]
+# error: [invalid-argument-type]
+C3().method_decorated(1)
+```
+
+The reason for this is that the heuristic is problematic. We don't *know* that the `Callable` in the
+return type of `memoize` is actually related to the method that we pass in. But when `memoize` is
+applied as a decorator, it is reasonable to assume so. In general, a function call might however
+return a `Callable` that is unrelated to the argument passed in. And here, it seems more
+reasonable/safe to treat the `Callable` as a non-descriptor:
+
+```py
+def convert_int_function(c: Callable[[int], int]) -> Callable[[int], str]:
+    return lambda x: str(c(x))
+
+class C4:
+    abs = convert_int_function(abs)
+    round = convert_int_function(round)
+
+reveal_type(C4().abs)  # revealed: Unknown | ((int, /) -> str)
+reveal_type(C4().round)  # revealed: Unknown | ((int, /) -> str)
+```

crates/ty_python_semantic/src/types.rs

@@ -1002,6 +1002,13 @@ impl<'db> Type<'db> {
         }
     }
 
+    pub(crate) const fn unwrap_as_callable_type(self) -> Option<CallableType<'db>> {
+        match self {
+            Type::Callable(callable_type) => Some(callable_type),
+            _ => None,
+        }
+    }
+
     pub(crate) const fn expect_dynamic(self) -> DynamicType<'db> {
         self.into_dynamic()
             .expect("Expected a Type::Dynamic variant")

crates/ty_python_semantic/src/types/infer/builder.rs

@@ -2175,7 +2175,26 @@ impl<'db, 'ast> TypeInferenceBuilder<'db, 'ast> {
                 .try_call(self.db(), &CallArguments::positional([inferred_ty]))
                 .map(|bindings| bindings.return_type(self.db()))
             {
-                Ok(return_ty) => return_ty,
+                Ok(return_ty) => {
+                    let is_input_function_like = inferred_ty.is_function_literal()
+                        || inferred_ty
+                            .unwrap_as_callable_type()
+                            .is_some_and(|callable| callable.is_function_like(self.db()));
+                    if is_input_function_like
+                        && let Some(callable_type) = return_ty.unwrap_as_callable_type()
+                    {
+                        // When a method on a class is decorated with a function that returns a `Callable`, assume that
+                        // the returned callable is also function-like. See "Decorating a method with a `Callable`-typed
+                        // decorator" in `callables_as_descriptors.md` for the extended explanation.
+                        Type::Callable(CallableType::new(
+                            self.db(),
+                            callable_type.signatures(self.db()),
+                            true,
+                        ))
+                    } else {
+                        return_ty
+                    }
+                }
                 Err(CallError(_, bindings)) => {
                     bindings.report_diagnostics(&self.context, (*decorator_node).into());
                     bindings.return_type(self.db())

crates/ty_python_semantic/src/types/signatures.rs

@@ -579,7 +579,13 @@ impl<'db> Signature<'db> {
     }
 
     pub(crate) fn bind_self(&self, db: &'db dyn Db, self_type: Option<Type<'db>>) -> Self {
-        let mut parameters = Parameters::new(self.parameters().iter().skip(1).cloned());
+        let mut parameters = self.parameters.iter().cloned().peekable();
+
+        if parameters.peek().is_some_and(Parameter::is_positional) {
+            parameters.next();
+        }
+
+        let mut parameters = Parameters::new(parameters);
         let mut return_ty = self.return_ty;
         if let Some(self_type) = self_type {
             parameters = parameters.apply_type_mapping_impl(