[red-knot] Understand typing.Callable (#16493)

## Summary

Part of https://github.com/astral-sh/ruff/issues/15382

This PR implements a general callable type that wraps around a
`Signature` and it uses that new type to represent `typing.Callable`.

It also implements `Display` support for `Callable`. The format is as:
```
([<arg name>][: <arg type>][ = <default type>], ...) -> <return type>
```

The `/` and `*` separators are added at the correct boundary for
positional-only and keyword-only parameters. Now, as `typing.Callable`
only has positional-only parameters, the rendered signature would be:

```py
Callable[[int, str], None]
# (int, str, /) -> None
```

The `/` separator represents that all the arguments are positional-only.

The relationship methods that check assignability, subtype relationship,
etc. are not yet implemented and will be done so as a follow-up.

## Test Plan

Add test cases for display support for `Signature` and various mdtest
for `typing.Callable`.

crates/red_knot_python_semantic/resources/mdtest/annotations/callable.md

@@ -0,0 +1,195 @@
+# Callable
+
+References:
+
+- <https://typing.readthedocs.io/en/latest/spec/callables.html#callable>
+
+TODO: Use `collections.abc` as importing from `typing` is deprecated but this requires support for
+`*` imports. See: <https://docs.python.org/3/library/typing.html#deprecated-aliases>.
+
+## Invalid forms
+
+The `Callable` special form requires _exactly_ two arguments where the first argument is either a
+parameter type list, parameter specification, `typing.Concatenate`, or `...` and the second argument
+is the return type. Here, we explore various invalid forms.
+
+### Empty
+
+A bare `Callable` without any type arguments:
+
+```py
+from typing import Callable
+
+def _(c: Callable):
+    reveal_type(c)  # revealed: (...) -> Unknown
+```
+
+### Invalid parameter type argument
+
+When it's not a list:
+
+```py
+from typing import Callable
+
+# error: [invalid-type-form] "The first argument to `Callable` must be either a list of types, ParamSpec, Concatenate, or `...`"
+def _(c: Callable[int, str]):
+    reveal_type(c)  # revealed: (...) -> Unknown
+```
+
+Or, when it's a literal type:
+
+```py
+# error: [invalid-type-form] "The first argument to `Callable` must be either a list of types, ParamSpec, Concatenate, or `...`"
+def _(c: Callable[42, str]):
+    reveal_type(c)  # revealed: (...) -> Unknown
+```
+
+Or, when one of the parameter type is invalid in the list:
+
+```py
+def _(c: Callable[[int, 42, str, False], None]):
+    # revealed: (int, @Todo(number literal in type expression), str, @Todo(boolean literal in type expression), /) -> None
+    reveal_type(c)
+```
+
+### Missing return type
+
+Using a parameter list:
+
+```py
+from typing import Callable
+
+# error: [invalid-type-form] "Special form `typing.Callable` expected exactly two arguments (parameter types and return type)"
+def _(c: Callable[[int, str]]):
+    reveal_type(c)  # revealed: (int, str, /) -> Unknown
+```
+
+Or, an ellipsis:
+
+```py
+# error: [invalid-type-form] "Special form `typing.Callable` expected exactly two arguments (parameter types and return type)"
+def _(c: Callable[...]):
+    reveal_type(c)  # revealed: (...) -> Unknown
+```
+
+### More than two arguments
+
+We can't reliably infer the callable type if there are more then 2 arguments because we don't know
+which argument corresponds to either the parameters or the return type.
+
+```py
+from typing import Callable
+
+# error: [invalid-type-form] "Special form `typing.Callable` expected exactly two arguments (parameter types and return type)"
+def _(c: Callable[[int], str, str]):
+    reveal_type(c)  # revealed: (...) -> Unknown
+```
+
+## Simple
+
+A simple `Callable` with multiple parameters and a return type:
+
+```py
+from typing import Callable
+
+def _(c: Callable[[int, str], int]):
+    reveal_type(c)  # revealed: (int, str, /) -> int
+```
+
+## Nested
+
+A nested `Callable` as one of the parameter types:
+
+```py
+from typing import Callable
+
+def _(c: Callable[[Callable[[int], str]], int]):
+    reveal_type(c)  # revealed: ((int, /) -> str, /) -> int
+```
+
+And, as the return type:
+
+```py
+def _(c: Callable[[int, str], Callable[[int], int]]):
+    reveal_type(c)  # revealed: (int, str, /) -> (int, /) -> int
+```
+
+## Gradual form
+
+The `Callable` special form supports the use of `...` in place of the list of parameter types. This
+is a [gradual form] indicating that the type is consistent with any input signature:
+
+```py
+from typing import Callable
+
+def gradual_form(c: Callable[..., str]):
+    reveal_type(c)  # revealed: (...) -> str
+```
+
+## Using `typing.Concatenate`
+
+Using `Concatenate` as the first argument to `Callable`:
+
+```py
+from typing_extensions import Callable, Concatenate
+
+def _(c: Callable[Concatenate[int, str, ...], int]):
+    reveal_type(c)  # revealed: (*args: @Todo(todo signature *args), **kwargs: @Todo(todo signature **kwargs)) -> int
+```
+
+And, as one of the parameter types:
+
+```py
+def _(c: Callable[[Concatenate[int, str, ...], int], int]):
+    reveal_type(c)  # revealed: (*args: @Todo(todo signature *args), **kwargs: @Todo(todo signature **kwargs)) -> int
+```
+
+## Using `typing.ParamSpec`
+
+Using a `ParamSpec` in a `Callable` annotation:
+
+```py
+from typing_extensions import Callable
+
+# TODO: Not an error; remove once `ParamSpec` is supported
+# error: [invalid-type-form]
+def _[**P1](c: Callable[P1, int]):
+    reveal_type(c)  # revealed: (...) -> Unknown
+```
+
+And, using the legacy syntax:
+
+```py
+from typing_extensions import ParamSpec
+
+P2 = ParamSpec("P2")
+
+# TODO: Not an error; remove once `ParamSpec` is supported
+# error: [invalid-type-form]
+def _(c: Callable[P2, int]):
+    reveal_type(c)  # revealed: (...) -> Unknown
+```
+
+## Using `typing.Unpack`
+
+Using the unpack operator (`*`):
+
+```py
+from typing_extensions import Callable, TypeVarTuple
+
+Ts = TypeVarTuple("Ts")
+
+def _(c: Callable[[int, *Ts], int]):
+    reveal_type(c)  # revealed: (*args: @Todo(todo signature *args), **kwargs: @Todo(todo signature **kwargs)) -> int
+```
+
+And, using the legacy syntax using `Unpack`:
+
+```py
+from typing_extensions import Unpack
+
+def _(c: Callable[[int, Unpack[Ts]], int]):
+    reveal_type(c)  # revealed: (*args: @Todo(todo signature *args), **kwargs: @Todo(todo signature **kwargs)) -> int
+```
+
+[gradual form]: https://typing.readthedocs.io/en/latest/spec/glossary.html#term-gradual-form

crates/red_knot_python_semantic/resources/mdtest/annotations/unsupported_special_forms.md

@@ -29,6 +29,8 @@ def i(callback: Callable[Concatenate[int, P], R_co], *args: P.args, **kwargs: P.
     # TODO: should understand the annotation
     reveal_type(kwargs)  # revealed: dict
 
+    # TODO: not an error; remove once `call` is implemented for `Callable`
+    # error: [call-non-callable]
     return callback(42, *args, **kwargs)
 
 class Foo:

crates/red_knot_python_semantic/resources/mdtest/invalid_syntax.md

@@ -91,3 +91,16 @@ match while:
 for x in foo.pass:
     pass
 ```
+
+## Invalid annotation
+
+### `typing.Callable`
+
+```py
+from typing import Callable
+
+# error: [invalid-syntax] "Expected index or slice expression"
+# error: [invalid-type-form] "Special form `typing.Callable` expected exactly two arguments (parameter types and return type)"
+def _(c: Callable[]):
+    reveal_type(c)  # revealed: (...) -> Unknown
+```

crates/red_knot_python_semantic/resources/mdtest/type_properties/is_single_valued.md

@@ -3,7 +3,7 @@
 A type is single-valued iff it is not empty and all inhabitants of it compare equal.
 
 ```py
-from typing_extensions import Any, Literal, LiteralString, Never
+from typing_extensions import Any, Literal, LiteralString, Never, Callable
 from knot_extensions import is_single_valued, static_assert
 
 static_assert(is_single_valued(None))
@@ -22,4 +22,7 @@ static_assert(not is_single_valued(Any))
 static_assert(not is_single_valued(Literal[1, 2]))
 
 static_assert(not is_single_valued(tuple[None, int]))
+
+static_assert(not is_single_valued(Callable[..., None]))
+static_assert(not is_single_valued(Callable[[int, str], None]))
 ```

crates/red_knot_python_semantic/resources/mdtest/type_properties/is_singleton.md

@@ -5,7 +5,7 @@ A type is a singleton type iff it has exactly one inhabitant.
 ## Basic
 
 ```py
-from typing_extensions import Literal, Never
+from typing_extensions import Literal, Never, Callable
 from knot_extensions import is_singleton, static_assert
 
 static_assert(is_singleton(None))
@@ -23,6 +23,9 @@ static_assert(not is_singleton(Literal[1, 2]))
 static_assert(not is_singleton(tuple[()]))
 static_assert(not is_singleton(tuple[None]))
 static_assert(not is_singleton(tuple[None, Literal[True]]))
+
+static_assert(not is_singleton(Callable[..., None]))
+static_assert(not is_singleton(Callable[[int, str], None]))
 ```
 
 ## `NoDefault`

crates/red_knot_python_semantic/src/types.rs

@@ -678,6 +678,11 @@ impl<'db> Type<'db> {
                     .is_subtype_of(db, target)
             }
 
+            (Type::Callable(CallableType::General(_)), _) => {
+                // TODO: Implement subtyping for general callable types
+                false
+            }
+
             // A fully static heterogenous tuple type `A` is a subtype of a fully static heterogeneous tuple type `B`
             // iff the two tuple types have the same number of elements and each element-type in `A` is a subtype
             // of the element-type at the same index in `B`. (Now say that 5 times fast.)
@@ -1248,6 +1253,12 @@ impl<'db> Type<'db> {
                 // TODO: add checks for the above cases once we support them
                 instance.is_disjoint_from(db, KnownClass::Tuple.to_instance(db))
             }
+
+            (Type::Callable(CallableType::General(_)), _)
+            | (_, Type::Callable(CallableType::General(_))) => {
+                // TODO: Implement disjointness for general callable types
+                false
+            }
         }
     }
 
@@ -1293,14 +1304,19 @@ impl<'db> Type<'db> {
             }
             Type::Union(union) => union.is_fully_static(db),
             Type::Intersection(intersection) => intersection.is_fully_static(db),
+            // TODO: Once we support them, make sure that we return `false` for other types
+            // containing gradual forms such as `tuple[Any, ...]`.
+            // Conversely, make sure to return `true` for homogeneous tuples such as
+            // `tuple[int, ...]`, once we add support for them.
             Type::Tuple(tuple) => tuple
                 .elements(db)
                 .iter()
                 .all(|elem| elem.is_fully_static(db)),
-            // TODO: Once we support them, make sure that we return `false` for other types
-            // containing gradual forms such as `tuple[Any, ...]` or `Callable[..., str]`.
-            // Conversely, make sure to return `true` for homogeneous tuples such as
-            // `tuple[int, ...]`, once we add support for them.
+            Type::Callable(CallableType::General(_)) => {
+                // TODO: `Callable` is not fully static when the parameter argument is `...` or
+                // when any parameter type or return type is not fully static.
+                false
+            }
         }
     }
 
@@ -1334,6 +1350,12 @@ impl<'db> Type<'db> {
             | Type::ClassLiteral(..)
             | Type::ModuleLiteral(..)
             | Type::KnownInstance(..) => true,
+            Type::Callable(CallableType::General(_)) => {
+                // A general callable type is never a singleton because for any given signature,
+                // there could be any number of distinct objects that are all callable with that
+                // signature.
+                false
+            }
             Type::Instance(InstanceType { class }) => {
                 class.known(db).is_some_and(KnownClass::is_singleton)
             }
@@ -1403,7 +1425,8 @@ impl<'db> Type<'db> {
             | Type::Intersection(..)
             | Type::LiteralString
             | Type::AlwaysTruthy
-            | Type::AlwaysFalsy => false,
+            | Type::AlwaysFalsy
+            | Type::Callable(CallableType::General(_)) => false,
         }
     }
 
@@ -1582,6 +1605,10 @@ impl<'db> Type<'db> {
                     .to_instance(db)
                     .instance_member(db, name)
             }
+            Type::Callable(CallableType::General(_)) => {
+                // TODO: Implement static member lookup for general callable types
+                Symbol::todo("static member lookup on general callable type").into()
+            }
 
             Type::IntLiteral(_) => KnownClass::Int.to_instance(db).instance_member(db, name),
             Type::BooleanLiteral(_) => KnownClass::Bool.to_instance(db).instance_member(db, name),
@@ -1926,6 +1953,10 @@ impl<'db> Type<'db> {
                     .to_instance(db)
                     .member(db, &name)
             }
+            Type::Callable(CallableType::General(_)) => {
+                // TODO
+                Symbol::todo("member lookup on general callable type").into()
+            }
 
             Type::Instance(InstanceType { class })
                 if class.is_known(db, KnownClass::VersionInfo) && name == "major" =>
@@ -3053,6 +3084,12 @@ impl<'db> Type<'db> {
             Type::KnownInstance(KnownInstanceType::Unknown) => Ok(Type::unknown()),
             Type::KnownInstance(KnownInstanceType::AlwaysTruthy) => Ok(Type::AlwaysTruthy),
             Type::KnownInstance(KnownInstanceType::AlwaysFalsy) => Ok(Type::AlwaysFalsy),
+            Type::KnownInstance(KnownInstanceType::Callable) => {
+                // TODO: Use an opt-in rule for a bare `Callable`
+                Ok(Type::Callable(CallableType::General(
+                    GeneralCallableType::unknown(db),
+                )))
+            }
             Type::KnownInstance(_) => Ok(todo_type!(
                 "Invalid or unsupported `KnownInstanceType` in `Type::to_type_expression`"
             )),
@@ -3125,6 +3162,10 @@ impl<'db> Type<'db> {
             Type::Callable(CallableType::WrapperDescriptorDunderGet) => {
                 KnownClass::WrapperDescriptorType.to_class_literal(db)
             }
+            Type::Callable(CallableType::General(_)) => {
+                // TODO: Get the meta type
+                todo_type!(".to_meta_type() for general callable type")
+            }
             Type::ModuleLiteral(_) => KnownClass::ModuleType.to_class_literal(db),
             Type::Tuple(_) => KnownClass::Tuple.to_class_literal(db),
             Type::ClassLiteral(ClassLiteralType { class }) => class.metaclass(db),
@@ -4313,9 +4354,30 @@ pub struct BoundMethodType<'db> {
     self_instance: Type<'db>,
 }
 
+/// This type represents a general callable type that are used to represent `typing.Callable`
+/// and `lambda` expressions.
+#[salsa::interned]
+pub struct GeneralCallableType<'db> {
+    #[return_ref]
+    signature: Signature<'db>,
+}
+
+impl<'db> GeneralCallableType<'db> {
+    /// Create a general callable type which accepts any parameters and returns an `Unknown` type.
+    pub(crate) fn unknown(db: &'db dyn Db) -> Self {
+        GeneralCallableType::new(
+            db,
+            Signature::new(Parameters::unknown(), Some(Type::unknown())),
+        )
+    }
+}
+
 /// A type that represents callable objects.
 #[derive(Debug, Copy, Clone, Hash, PartialEq, Eq, salsa::Update)]
 pub enum CallableType<'db> {
+    /// Represents a general callable type.
+    General(GeneralCallableType<'db>),
+
     /// Represents a callable `instance.method` where `instance` is an instance of a class
     /// and `method` is a method (of that class).
     ///

crates/red_knot_python_semantic/src/types/diagnostic.rs

@@ -1151,3 +1151,18 @@ pub(crate) fn report_invalid_arguments_to_annotated<'db>(
         ),
     );
 }
+
+pub(crate) fn report_invalid_arguments_to_callable<'db>(
+    db: &'db dyn Db,
+    context: &InferContext<'db>,
+    subscript: &ast::ExprSubscript,
+) {
+    context.report_lint(
+        &INVALID_TYPE_FORM,
+        subscript,
+        format_args!(
+            "Special form `{}` expected exactly two arguments (parameter types and return type)",
+            KnownInstanceType::Callable.repr(db)
+        ),
+    );
+}

crates/red_knot_python_semantic/src/types/display.rs

@@ -7,6 +7,7 @@ use ruff_python_ast::str::{Quote, TripleQuotes};
 use ruff_python_literal::escape::AsciiEscape;
 
 use crate::types::class_base::ClassBase;
+use crate::types::signatures::{Parameter, Parameters, Signature};
 use crate::types::{
     CallableType, ClassLiteralType, InstanceType, IntersectionType, KnownClass, StringLiteralType,
     Type, UnionType,
@@ -88,6 +89,9 @@ impl Display for DisplayRepresentation<'_> {
             },
             Type::KnownInstance(known_instance) => f.write_str(known_instance.repr(self.db)),
             Type::FunctionLiteral(function) => f.write_str(function.name(self.db)),
+            Type::Callable(CallableType::General(callable)) => {
+                callable.signature(self.db).display(self.db).fmt(f)
+            }
             Type::Callable(CallableType::BoundMethod(bound_method)) => {
                 write!(
                     f,
@@ -156,6 +160,99 @@ impl Display for DisplayRepresentation<'_> {
     }
 }
 
+impl<'db> Signature<'db> {
+    fn display(&'db self, db: &'db dyn Db) -> DisplaySignature<'db> {
+        DisplaySignature {
+            parameters: self.parameters(),
+            return_ty: self.return_ty.as_ref(),
+            db,
+        }
+    }
+}
+
+struct DisplaySignature<'db> {
+    parameters: &'db Parameters<'db>,
+    return_ty: Option<&'db Type<'db>>,
+    db: &'db dyn Db,
+}
+
+impl Display for DisplaySignature<'_> {
+    fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
+        f.write_char('(')?;
+
+        if self.parameters.is_gradual() {
+            // We represent gradual form as `...` in the signature, internally the parameters still
+            // contain `(*args, **kwargs)` parameters.
+            f.write_str("...")?;
+        } else {
+            let mut star_added = false;
+            let mut needs_slash = false;
+            let mut join = f.join(", ");
+
+            for parameter in self.parameters.as_slice() {
+                if !star_added && parameter.is_keyword_only() {
+                    join.entry(&'*');
+                    star_added = true;
+                }
+                if parameter.is_positional_only() {
+                    needs_slash = true;
+                } else if needs_slash {
+                    join.entry(&'/');
+                    needs_slash = false;
+                }
+                join.entry(&parameter.display(self.db));
+            }
+            if needs_slash {
+                join.entry(&'/');
+            }
+            join.finish()?;
+        }
+
+        write!(
+            f,
+            ") -> {}",
+            self.return_ty.unwrap_or(&Type::unknown()).display(self.db)
+        )?;
+
+        Ok(())
+    }
+}
+
+impl<'db> Parameter<'db> {
+    fn display(&'db self, db: &'db dyn Db) -> DisplayParameter<'db> {
+        DisplayParameter { param: self, db }
+    }
+}
+
+struct DisplayParameter<'db> {
+    param: &'db Parameter<'db>,
+    db: &'db dyn Db,
+}
+
+impl Display for DisplayParameter<'_> {
+    fn fmt(&self, f: &mut Formatter<'_>) -> fmt::Result {
+        if let Some(name) = self.param.display_name() {
+            write!(f, "{name}")?;
+            if let Some(annotated_type) = self.param.annotated_type() {
+                write!(f, ": {}", annotated_type.display(self.db))?;
+            }
+            // Default value can only be specified if `name` is given.
+            if let Some(default_ty) = self.param.default_type() {
+                if self.param.annotated_type().is_some() {
+                    write!(f, " = {}", default_ty.display(self.db))?;
+                } else {
+                    write!(f, "={}", default_ty.display(self.db))?;
+                }
+            }
+        } else if let Some(ty) = self.param.annotated_type() {
+            // This case is specifically for the `Callable` signature where name and default value
+            // cannot be provided.
+            ty.display(self.db).fmt(f)?;
+        }
+        Ok(())
+    }
+}
+
 impl<'db> UnionType<'db> {
     fn display(&'db self, db: &'db dyn Db) -> DisplayUnionType<'db> {
         DisplayUnionType { db, ty: self }
@@ -375,8 +472,14 @@ impl Display for DisplayStringLiteralType<'_> {
 
 #[cfg(test)]
 mod tests {
+    use ruff_python_ast::name::Name;
+
     use crate::db::tests::setup_db;
-    use crate::types::{SliceLiteralType, StringLiteralType, Type};
+    use crate::types::{
+        KnownClass, Parameter, ParameterKind, Parameters, Signature, SliceLiteralType,
+        StringLiteralType, Type,
+    };
+    use crate::Db;
 
     #[test]
     fn test_slice_literal_display() {
@@ -443,4 +546,226 @@ mod tests {
             r#"Literal["\""]"#
         );
     }
+
+    fn display_signature<'db>(
+        db: &dyn Db,
+        parameters: impl IntoIterator<Item = Parameter<'db>>,
+        return_ty: Option<Type<'db>>,
+    ) -> String {
+        Signature::new(Parameters::new(parameters), return_ty)
+            .display(db)
+            .to_string()
+    }
+
+    #[test]
+    fn signature_display() {
+        let db = setup_db();
+
+        // Empty parameters with no return type.
+        assert_eq!(display_signature(&db, [], None), "() -> Unknown");
+
+        // Empty parameters with a return type.
+        assert_eq!(
+            display_signature(&db, [], Some(Type::none(&db))),
+            "() -> None"
+        );
+
+        // Single parameter type (no name) with a return type.
+        assert_eq!(
+            display_signature(
+                &db,
+                [Parameter::new(
+                    None,
+                    Some(Type::none(&db)),
+                    ParameterKind::PositionalOrKeyword { default_ty: None }
+                )],
+                Some(Type::none(&db))
+            ),
+            "(None) -> None"
+        );
+
+        // Two parameters where one has annotation and the other doesn't.
+        assert_eq!(
+            display_signature(
+                &db,
+                [
+                    Parameter::new(
+                        Some(Name::new_static("x")),
+                        None,
+                        ParameterKind::PositionalOrKeyword {
+                            default_ty: Some(KnownClass::Int.to_instance(&db))
+                        }
+                    ),
+                    Parameter::new(
+                        Some(Name::new_static("y")),
+                        Some(KnownClass::Str.to_instance(&db)),
+                        ParameterKind::PositionalOrKeyword {
+                            default_ty: Some(KnownClass::Str.to_instance(&db))
+                        }
+                    )
+                ],
+                Some(Type::none(&db))
+            ),
+            "(x=int, y: str = str) -> None"
+        );
+
+        // All positional only parameters.
+        assert_eq!(
+            display_signature(
+                &db,
+                [
+                    Parameter::new(
+                        Some(Name::new_static("x")),
+                        None,
+                        ParameterKind::PositionalOnly { default_ty: None }
+                    ),
+                    Parameter::new(
+                        Some(Name::new_static("y")),
+                        None,
+                        ParameterKind::PositionalOnly { default_ty: None }
+                    )
+                ],
+                Some(Type::none(&db))
+            ),
+            "(x, y, /) -> None"
+        );
+
+        // Positional-only parameters mixed with non-positional-only parameters.
+        assert_eq!(
+            display_signature(
+                &db,
+                [
+                    Parameter::new(
+                        Some(Name::new_static("x")),
+                        None,
+                        ParameterKind::PositionalOnly { default_ty: None }
+                    ),
+                    Parameter::new(
+                        Some(Name::new_static("y")),
+                        None,
+                        ParameterKind::PositionalOrKeyword { default_ty: None }
+                    )
+                ],
+                Some(Type::none(&db))
+            ),
+            "(x, /, y) -> None"
+        );
+
+        // All keyword-only parameters.
+        assert_eq!(
+            display_signature(
+                &db,
+                [
+                    Parameter::new(
+                        Some(Name::new_static("x")),
+                        None,
+                        ParameterKind::KeywordOnly { default_ty: None }
+                    ),
+                    Parameter::new(
+                        Some(Name::new_static("y")),
+                        None,
+                        ParameterKind::KeywordOnly { default_ty: None }
+                    )
+                ],
+                Some(Type::none(&db))
+            ),
+            "(*, x, y) -> None"
+        );
+
+        // Keyword-only parameters mixed with non-keyword-only parameters.
+        assert_eq!(
+            display_signature(
+                &db,
+                [
+                    Parameter::new(
+                        Some(Name::new_static("x")),
+                        None,
+                        ParameterKind::PositionalOrKeyword { default_ty: None }
+                    ),
+                    Parameter::new(
+                        Some(Name::new_static("y")),
+                        None,
+                        ParameterKind::KeywordOnly { default_ty: None }
+                    )
+                ],
+                Some(Type::none(&db))
+            ),
+            "(x, *, y) -> None"
+        );
+
+        // A mix of all parameter kinds.
+        assert_eq!(
+            display_signature(
+                &db,
+                [
+                    Parameter::new(
+                        Some(Name::new_static("a")),
+                        None,
+                        ParameterKind::PositionalOnly { default_ty: None },
+                    ),
+                    Parameter::new(
+                        Some(Name::new_static("b")),
+                        Some(KnownClass::Int.to_instance(&db)),
+                        ParameterKind::PositionalOnly { default_ty: None },
+                    ),
+                    Parameter::new(
+                        Some(Name::new_static("c")),
+                        None,
+                        ParameterKind::PositionalOnly {
+                            default_ty: Some(Type::IntLiteral(1)),
+                        },
+                    ),
+                    Parameter::new(
+                        Some(Name::new_static("d")),
+                        Some(KnownClass::Int.to_instance(&db)),
+                        ParameterKind::PositionalOnly {
+                            default_ty: Some(Type::IntLiteral(2)),
+                        },
+                    ),
+                    Parameter::new(
+                        Some(Name::new_static("e")),
+                        None,
+                        ParameterKind::PositionalOrKeyword {
+                            default_ty: Some(Type::IntLiteral(3)),
+                        },
+                    ),
+                    Parameter::new(
+                        Some(Name::new_static("f")),
+                        Some(KnownClass::Int.to_instance(&db)),
+                        ParameterKind::PositionalOrKeyword {
+                            default_ty: Some(Type::IntLiteral(4)),
+                        },
+                    ),
+                    Parameter::new(
+                        Some(Name::new_static("args")),
+                        Some(Type::object(&db)),
+                        ParameterKind::Variadic,
+                    ),
+                    Parameter::new(
+                        Some(Name::new_static("g")),
+                        None,
+                        ParameterKind::KeywordOnly {
+                            default_ty: Some(Type::IntLiteral(5)),
+                        },
+                    ),
+                    Parameter::new(
+                        Some(Name::new_static("h")),
+                        Some(KnownClass::Int.to_instance(&db)),
+                        ParameterKind::KeywordOnly {
+                            default_ty: Some(Type::IntLiteral(6)),
+                        },
+                    ),
+                    Parameter::new(
+                        Some(Name::new_static("kwargs")),
+                        Some(KnownClass::Str.to_instance(&db)),
+                        ParameterKind::KeywordVariadic,
+                    ),
+                ],
+                Some(KnownClass::Bytes.to_instance(&db))
+            ),
+            "(a, b: int, c=Literal[1], d: int = Literal[2], \
+                /, e=Literal[3], f: int = Literal[4], *args: object, \
+                *, g=Literal[5], h: int = Literal[6], **kwargs: str) -> bytes"
+        );
+    }
 }

crates/red_knot_python_semantic/src/types/infer.rs

@@ -56,11 +56,11 @@ use crate::symbol::{
 };
 use crate::types::call::{Argument, CallArguments, UnionCallError};
 use crate::types::diagnostic::{
-    report_invalid_arguments_to_annotated, report_invalid_assignment,
-    report_invalid_attribute_assignment, report_unresolved_module, TypeCheckDiagnostics,
-    CALL_NON_CALLABLE, CALL_POSSIBLY_UNBOUND_METHOD, CONFLICTING_DECLARATIONS,
-    CONFLICTING_METACLASS, CYCLIC_CLASS_DEFINITION, DIVISION_BY_ZERO, DUPLICATE_BASE,
-    INCONSISTENT_MRO, INVALID_ATTRIBUTE_ACCESS, INVALID_BASE, INVALID_DECLARATION,
+    report_invalid_arguments_to_annotated, report_invalid_arguments_to_callable,
+    report_invalid_assignment, report_invalid_attribute_assignment, report_unresolved_module,
+    TypeCheckDiagnostics, CALL_NON_CALLABLE, CALL_POSSIBLY_UNBOUND_METHOD,
+    CONFLICTING_DECLARATIONS, CONFLICTING_METACLASS, CYCLIC_CLASS_DEFINITION, DIVISION_BY_ZERO,
+    DUPLICATE_BASE, INCONSISTENT_MRO, INVALID_ATTRIBUTE_ACCESS, INVALID_BASE, INVALID_DECLARATION,
     INVALID_PARAMETER_DEFAULT, INVALID_TYPE_FORM, INVALID_TYPE_VARIABLE_CONSTRAINTS,
     POSSIBLY_UNBOUND_ATTRIBUTE, POSSIBLY_UNBOUND_IMPORT, UNDEFINED_REVEAL, UNRESOLVED_ATTRIBUTE,
     UNRESOLVED_IMPORT, UNSUPPORTED_OPERATOR,
@@ -70,10 +70,12 @@ use crate::types::unpacker::{UnpackResult, Unpacker};
 use crate::types::{
     class::MetaclassErrorKind, todo_type, Class, DynamicType, FunctionType, InstanceType,
     IntersectionBuilder, IntersectionType, KnownClass, KnownFunction, KnownInstanceType,
-    MetaclassCandidate, SliceLiteralType, SubclassOfType, Symbol, SymbolAndQualifiers, Truthiness,
-    TupleType, Type, TypeAliasType, TypeAndQualifiers, TypeArrayDisplay, TypeQualifiers,
-    TypeVarBoundOrConstraints, TypeVarInstance, UnionBuilder, UnionType,
+    MetaclassCandidate, Parameter, Parameters, SliceLiteralType, SubclassOfType, Symbol,
+    SymbolAndQualifiers, Truthiness, TupleType, Type, TypeAliasType, TypeAndQualifiers,
+    TypeArrayDisplay, TypeQualifiers, TypeVarBoundOrConstraints, TypeVarInstance, UnionBuilder,
+    UnionType,
 };
+use crate::types::{CallableType, GeneralCallableType, ParameterKind, Signature};
 use crate::unpack::Unpack;
 use crate::util::subscript::{PyIndex, PySlice};
 use crate::Db;
@@ -5593,14 +5595,16 @@ impl<'db> TypeInferenceBuilder<'db> {
 
             // TODO: an Ellipsis literal *on its own* does not have any meaning in annotation
             // expressions, but is meaningful in the context of a number of special forms.
-            ast::Expr::EllipsisLiteral(_literal) => todo_type!(),
+            ast::Expr::EllipsisLiteral(_literal) => {
+                todo_type!("ellipsis literal in type expression")
+            }
 
             // Other literals do not have meaningful values in the annotation expression context.
             // However, we will we want to handle these differently when working with special forms,
             // since (e.g.) `123` is not valid in an annotation expression but `Literal[123]` is.
-            ast::Expr::BytesLiteral(_literal) => todo_type!(),
-            ast::Expr::NumberLiteral(_literal) => todo_type!(),
-            ast::Expr::BooleanLiteral(_literal) => todo_type!(),
+            ast::Expr::BytesLiteral(_literal) => todo_type!("bytes literal in type expression"),
+            ast::Expr::NumberLiteral(_literal) => todo_type!("number literal in type expression"),
+            ast::Expr::BooleanLiteral(_literal) => todo_type!("boolean literal in type expression"),
 
             ast::Expr::Subscript(subscript) => {
                 let ast::ExprSubscript {
@@ -6000,8 +6004,61 @@ impl<'db> TypeInferenceBuilder<'db> {
                 todo_type!("Generic PEP-695 type alias")
             }
             KnownInstanceType::Callable => {
-                self.infer_type_expression(arguments_slice);
-                todo_type!("Callable types")
+                let ast::Expr::Tuple(ast::ExprTuple {
+                    elts: arguments, ..
+                }) = arguments_slice
+                else {
+                    report_invalid_arguments_to_callable(self.db(), &self.context, subscript);
+
+                    // If it's not a tuple, defer it to inferring the parameter types which could
+                    // return an `Err` if the expression is invalid in that position. In which
+                    // case, we'll fallback to using an unknown list of parameters.
+                    let parameters = self
+                        .infer_callable_parameter_types(arguments_slice)
+                        .unwrap_or_else(|()| Parameters::unknown());
+
+                    let callable_type =
+                        Type::Callable(CallableType::General(GeneralCallableType::new(
+                            self.db(),
+                            Signature::new(parameters, Some(Type::unknown())),
+                        )));
+
+                    // `Parameters` is not a `Type` variant, so we're storing the outer callable
+                    // type on the arguments slice instead.
+                    self.store_expression_type(arguments_slice, callable_type);
+
+                    return callable_type;
+                };
+
+                let [first_argument, second_argument] = arguments.as_slice() else {
+                    report_invalid_arguments_to_callable(self.db(), &self.context, subscript);
+                    self.infer_type_expression(arguments_slice);
+                    return Type::Callable(CallableType::General(GeneralCallableType::unknown(
+                        self.db(),
+                    )));
+                };
+
+                let Ok(parameters) = self.infer_callable_parameter_types(first_argument) else {
+                    self.infer_type_expression(arguments_slice);
+                    return Type::Callable(CallableType::General(GeneralCallableType::unknown(
+                        self.db(),
+                    )));
+                };
+
+                let return_type = self.infer_type_expression(second_argument);
+
+                let callable_type =
+                    Type::Callable(CallableType::General(GeneralCallableType::new(
+                        self.db(),
+                        Signature::new(parameters, Some(return_type)),
+                    )));
+
+                // `Signature` / `Parameters` are not a `Type` variant, so we're storing the outer
+                // callable type on the these expressions instead.
+                self.store_expression_type(arguments_slice, callable_type);
+                self.store_expression_type(first_argument, callable_type);
+
+                callable_type
             }
 
             // Type API special forms
@@ -6257,6 +6314,73 @@ impl<'db> TypeInferenceBuilder<'db> {
             }
         })
     }
+
+    /// Infer the first argument to a `typing.Callable` type expression and returns the
+    /// corresponding [`Parameters`].
+    ///
+    /// It returns an [`Err`] if the argument is invalid i.e., not a list of types, parameter
+    /// specification, `typing.Concatenate`, or `...`.
+    fn infer_callable_parameter_types(
+        &mut self,
+        parameters: &ast::Expr,
+    ) -> Result<Parameters<'db>, ()> {
+        Ok(match parameters {
+            ast::Expr::EllipsisLiteral(ast::ExprEllipsisLiteral { .. }) => {
+                Parameters::gradual_form()
+            }
+            ast::Expr::List(ast::ExprList { elts: params, .. }) => {
+                let mut parameter_types = Vec::with_capacity(params.len());
+
+                // Whether to infer `Todo` for the parameters
+                let mut return_todo = false;
+
+                for param in params {
+                    let param_type = self.infer_type_expression(param);
+                    // This is similar to what we currently do for inferring tuple type expression.
+                    // We currently infer `Todo` for the parameters to avoid invalid diagnostics
+                    // when trying to check for assignability or any other relation. For example,
+                    // `*tuple[int, str]`, `Unpack[]`, etc. are not yet supported.
+                    return_todo |= param_type.is_todo()
+                        && matches!(param, ast::Expr::Starred(_) | ast::Expr::Subscript(_));
+                    parameter_types.push(param_type);
+                }
+
+                if return_todo {
+                    // TODO: `Unpack`
+                    Parameters::todo()
+                } else {
+                    Parameters::new(parameter_types.iter().map(|param_type| {
+                        Parameter::new(
+                            None,
+                            Some(*param_type),
+                            ParameterKind::PositionalOnly { default_ty: None },
+                        )
+                    }))
+                }
+            }
+            ast::Expr::Subscript(_) => {
+                // TODO: Support `Concatenate[...]`
+                Parameters::todo()
+            }
+            ast::Expr::Name(name) if name.is_invalid() => {
+                // This is a special case to avoid raising the error suggesting what the first
+                // argument should be. This only happens when there's already a syntax error like
+                // `Callable[]`.
+                return Err(());
+            }
+            _ => {
+                // TODO: Check whether `Expr::Name` is a ParamSpec
+                self.context.report_lint(
+                    &INVALID_TYPE_FORM,
+                    parameters,
+                    format_args!(
+                        "The first argument to `Callable` must be either a list of types, ParamSpec, Concatenate, or `...`",
+                    ),
+                );
+                return Err(());
+            }
+        })
+    }
 }
 
 /// The deferred state of a specific expression in an inference region.

crates/red_knot_python_semantic/src/types/signatures.rs

@@ -1,11 +1,11 @@
-use super::{definition_expression_type, Type};
+use super::{definition_expression_type, DynamicType, Type};
 use crate::Db;
 use crate::{semantic_index::definition::Definition, types::todo_type};
 use ruff_python_ast::{self as ast, name::Name};
 
 /// A typed callable signature.
-#[derive(Clone, Debug, PartialEq, Eq, salsa::Update)]
-pub(crate) struct Signature<'db> {
+#[derive(Clone, Debug, PartialEq, Eq, Hash, salsa::Update)]
+pub struct Signature<'db> {
     /// Parameters, in source order.
     ///
     /// The ordering of parameters in a valid signature must be: first positional-only parameters,
@@ -67,29 +67,105 @@ impl<'db> Signature<'db> {
     }
 }
 
-// TODO: use SmallVec here once invariance bug is fixed
-#[derive(Clone, Debug, PartialEq, Eq, salsa::Update)]
-pub(crate) struct Parameters<'db>(Vec<Parameter<'db>>);
+#[derive(Clone, Debug, PartialEq, Eq, Hash, salsa::Update)]
+pub(crate) struct Parameters<'db> {
+    // TODO: use SmallVec here once invariance bug is fixed
+    value: Vec<Parameter<'db>>,
+
+    /// Whether this parameter list represents a gradual form using `...` as the only parameter.
+    ///
+    /// If this is `true`, the `value` will still contain the variadic and keyword-variadic
+    /// parameters. This flag is used to distinguish between an explicit `...` in the callable type
+    /// as in `Callable[..., int]` and the variadic arguments in `lambda` expression as in
+    /// `lambda *args, **kwargs: None`.
+    ///
+    /// The display implementation utilizes this flag to use `...` instead of displaying the
+    /// individual variadic and keyword-variadic parameters.
+    ///
+    /// Note: This flag is also used to indicate invalid forms of `Callable` annotations.
+    is_gradual: bool,
+}
 
 impl<'db> Parameters<'db> {
     pub(crate) fn new(parameters: impl IntoIterator<Item = Parameter<'db>>) -> Self {
-        Self(parameters.into_iter().collect())
+        Self {
+            value: parameters.into_iter().collect(),
+            is_gradual: false,
+        }
+    }
+
+    pub(crate) fn as_slice(&self) -> &[Parameter<'db>] {
+        self.value.as_slice()
+    }
+
+    pub(crate) const fn is_gradual(&self) -> bool {
+        self.is_gradual
     }
 
     /// Return todo parameters: (*args: Todo, **kwargs: Todo)
-    fn todo() -> Self {
-        Self(vec![
-            Parameter {
-                name: Some(Name::new_static("args")),
-                annotated_ty: Some(todo_type!("todo signature *args")),
-                kind: ParameterKind::Variadic,
-            },
-            Parameter {
-                name: Some(Name::new_static("kwargs")),
-                annotated_ty: Some(todo_type!("todo signature **kwargs")),
-                kind: ParameterKind::KeywordVariadic,
-            },
-        ])
+    pub(crate) fn todo() -> Self {
+        Self {
+            value: vec![
+                Parameter {
+                    name: Some(Name::new_static("args")),
+                    annotated_ty: Some(todo_type!("todo signature *args")),
+                    kind: ParameterKind::Variadic,
+                },
+                Parameter {
+                    name: Some(Name::new_static("kwargs")),
+                    annotated_ty: Some(todo_type!("todo signature **kwargs")),
+                    kind: ParameterKind::KeywordVariadic,
+                },
+            ],
+            is_gradual: false,
+        }
+    }
+
+    /// Return parameters that represents a gradual form using `...` as the only parameter.
+    ///
+    /// Internally, this is represented as `(*Any, **Any)` that accepts parameters of type [`Any`].
+    ///
+    /// [`Any`]: crate::types::DynamicType::Any
+    pub(crate) fn gradual_form() -> Self {
+        Self {
+            value: vec![
+                Parameter {
+                    name: None,
+                    annotated_ty: Some(Type::Dynamic(DynamicType::Any)),
+                    kind: ParameterKind::Variadic,
+                },
+                Parameter {
+                    name: None,
+                    annotated_ty: Some(Type::Dynamic(DynamicType::Any)),
+                    kind: ParameterKind::KeywordVariadic,
+                },
+            ],
+            is_gradual: true,
+        }
+    }
+
+    /// Return parameters that represents an unknown list of parameters.
+    ///
+    /// Internally, this is represented as `(*Unknown, **Unknown)` that accepts parameters of type
+    /// [`Unknown`].
+    ///
+    /// [`Unknown`]: crate::types::DynamicType::Unknown
+    pub(crate) fn unknown() -> Self {
+        Self {
+            value: vec![
+                Parameter {
+                    name: None,
+                    annotated_ty: Some(Type::Dynamic(DynamicType::Unknown)),
+                    kind: ParameterKind::Variadic,
+                },
+                Parameter {
+                    name: None,
+                    annotated_ty: Some(Type::Dynamic(DynamicType::Unknown)),
+                    kind: ParameterKind::KeywordVariadic,
+                },
+            ],
+            is_gradual: true,
+        }
     }
 
     fn from_parameters(
@@ -146,22 +222,21 @@ impl<'db> Parameters<'db> {
         let keywords = kwarg.as_ref().map(|arg| {
             Parameter::from_node_and_kind(db, definition, arg, ParameterKind::KeywordVariadic)
         });
-        Self(
+        Self::new(
             positional_only
                 .chain(positional_or_keyword)
                 .chain(variadic)
                 .chain(keyword_only)
-                .chain(keywords)
-                .collect(),
+                .chain(keywords),
         )
     }
 
     pub(crate) fn len(&self) -> usize {
-        self.0.len()
+        self.value.len()
     }
 
     pub(crate) fn iter(&self) -> std::slice::Iter<Parameter<'db>> {
-        self.0.iter()
+        self.value.iter()
     }
 
     /// Iterate initial positional parameters, not including variadic parameter, if any.
@@ -175,7 +250,7 @@ impl<'db> Parameters<'db> {
 
     /// Return parameter at given index, or `None` if index is out-of-range.
     pub(crate) fn get(&self, index: usize) -> Option<&Parameter<'db>> {
-        self.0.get(index)
+        self.value.get(index)
     }
 
     /// Return positional parameter at given index, or `None` if `index` is out of range.
@@ -218,7 +293,7 @@ impl<'db, 'a> IntoIterator for &'a Parameters<'db> {
     type IntoIter = std::slice::Iter<'a, Parameter<'db>>;
 
     fn into_iter(self) -> Self::IntoIter {
-        self.0.iter()
+        self.value.iter()
     }
 }
 
@@ -226,11 +301,11 @@ impl<'db> std::ops::Index<usize> for Parameters<'db> {
     type Output = Parameter<'db>;
 
     fn index(&self, index: usize) -> &Self::Output {
-        &self.0[index]
+        &self.value[index]
     }
 }
 
-#[derive(Clone, Debug, PartialEq, Eq, salsa::Update)]
+#[derive(Clone, Debug, PartialEq, Eq, Hash, salsa::Update)]
 pub(crate) struct Parameter<'db> {
     /// Parameter name.
     ///
@@ -272,6 +347,14 @@ impl<'db> Parameter<'db> {
         }
     }
 
+    pub(crate) fn is_keyword_only(&self) -> bool {
+        matches!(self.kind, ParameterKind::KeywordOnly { .. })
+    }
+
+    pub(crate) fn is_positional_only(&self) -> bool {
+        matches!(self.kind, ParameterKind::PositionalOnly { .. })
+    }
+
     pub(crate) fn is_variadic(&self) -> bool {
         matches!(self.kind, ParameterKind::Variadic)
     }
@@ -328,7 +411,7 @@ impl<'db> Parameter<'db> {
     }
 }
 
-#[derive(Clone, Debug, PartialEq, Eq, salsa::Update)]
+#[derive(Clone, Debug, PartialEq, Eq, Hash, salsa::Update)]
 pub(crate) enum ParameterKind<'db> {
     /// Positional-only parameter, e.g. `def f(x, /): ...`
     PositionalOnly { default_ty: Option<Type<'db>> },
@@ -361,7 +444,7 @@ mod tests {
 
     #[track_caller]
     fn assert_params<'db>(signature: &Signature<'db>, expected: &[Parameter<'db>]) {
-        assert_eq!(signature.parameters.0.as_slice(), expected);
+        assert_eq!(signature.parameters.value.as_slice(), expected);
     }
 
     #[test]
@@ -490,7 +573,7 @@ mod tests {
             name: Some(name),
             annotated_ty,
             kind: ParameterKind::PositionalOrKeyword { .. },
-        }] = &sig.parameters.0[..]
+        }] = &sig.parameters.value[..]
         else {
             panic!("expected one positional-or-keyword parameter");
         };
@@ -524,7 +607,7 @@ mod tests {
             name: Some(name),
             annotated_ty,
             kind: ParameterKind::PositionalOrKeyword { .. },
-        }] = &sig.parameters.0[..]
+        }] = &sig.parameters.value[..]
         else {
             panic!("expected one positional-or-keyword parameter");
         };
@@ -562,7 +645,7 @@ mod tests {
             name: Some(b_name),
             annotated_ty: b_annotated_ty,
             kind: ParameterKind::PositionalOrKeyword { .. },
-        }] = &sig.parameters.0[..]
+        }] = &sig.parameters.value[..]
         else {
             panic!("expected two positional-or-keyword parameters");
         };
@@ -605,7 +688,7 @@ mod tests {
             name: Some(b_name),
             annotated_ty: b_annotated_ty,
             kind: ParameterKind::PositionalOrKeyword { .. },
-        }] = &sig.parameters.0[..]
+        }] = &sig.parameters.value[..]
         else {
             panic!("expected two positional-or-keyword parameters");
         };

crates/red_knot_python_semantic/src/types/type_ordering.rs

@@ -77,6 +77,12 @@ pub(super) fn union_elements_ordering<'db>(left: &Type<'db>, right: &Type<'db>)
         (Type::Callable(CallableType::WrapperDescriptorDunderGet), _) => Ordering::Less,
         (_, Type::Callable(CallableType::WrapperDescriptorDunderGet)) => Ordering::Greater,
 
+        (Type::Callable(CallableType::General(_)), Type::Callable(CallableType::General(_))) => {
+            Ordering::Equal
+        }
+        (Type::Callable(CallableType::General(_)), _) => Ordering::Less,
+        (_, Type::Callable(CallableType::General(_))) => Ordering::Greater,
+
         (Type::Tuple(left), Type::Tuple(right)) => left.cmp(right),
         (Type::Tuple(_), _) => Ordering::Less,
         (_, Type::Tuple(_)) => Ordering::Greater,

crates/ruff_python_ast/src/nodes.rs

@@ -2179,6 +2179,13 @@ impl ExprName {
     pub fn id(&self) -> &Name {
         &self.id
     }
+
+    /// Returns `true` if this node represents an invalid name i.e., the `ctx` is [`Invalid`].
+    ///
+    /// [`Invalid`]: ExprContext::Invalid
+    pub const fn is_invalid(&self) -> bool {
+        matches!(self.ctx, ExprContext::Invalid)
+    }
 }
 
 impl ExprList {