mirror of https://github.com/astral-sh/ruff.git synced 2025-09-29 05:15:12 +00:00

Dhruv Manilawala 44ad201262

[red-knot] Add support for overloaded functions (#17366 )

## Summary

Part of #15383, this PR adds support for overloaded callables.

Typing spec: https://typing.python.org/en/latest/spec/overload.html

Specifically, it does the following:
1. Update the `FunctionType::signature` method to return signatures from
a possibly overloaded callable using a new `FunctionSignature` enum
2. Update `CallableType` to accommodate overloaded callable by updating
the inner type to `Box<[Signature]>`
3. Update the relation methods on `CallableType` with logic specific to
overloads
4. Update the display of callable type to display a list of signatures
enclosed by parenthesis
5. Update `CallableTypeOf` special form to recognize overloaded callable
6. Update subtyping, assignability and fully static check to account for
callables (equivalence is planned to be done as a follow-up)

For (2), it is required to be done in this PR because otherwise I'd need
to add some workaround for `into_callable_type` and I though it would be
best to include it in here.

For (2), another possible design would be convert `CallableType` in an
enum with two variants `CallableType::Single` and
`CallableType::Overload` but I decided to go with `Box<[Signature]>` for
now to (a) mirror it to be equivalent to `overload` field on
`CallableSignature` and (b) to avoid any refactor in this PR. This could
be done in a follow-up to better split the two kind of callables.

### Design

There were two main candidates on how to represent the overloaded
definition:
1. To include it in the existing infrastructure which is what this PR is
doing by recognizing all the signatures within the
`FunctionType::signature` method
2. To create a new `Overload` type variant

<details><summary>For context, this is what I had in mind with the new
type variant:</summary>
<p>

```rs
pub enum Type {
	FunctionLiteral(FunctionType),
    Overload(OverloadType),
    BoundMethod(BoundMethodType),
    ...
}

pub struct OverloadType {
	// FunctionLiteral or BoundMethod
    overloads: Box<[Type]>,
	// FunctionLiteral or BoundMethod
    implementation: Option<Type>
}

pub struct BoundMethodType {
    kind: BoundMethodKind,
    self_instance: Type,
}

pub enum BoundMethodKind {
    Function(FunctionType),
    Overload(OverloadType),
}
```

</p>
</details> 

The main reasons to choose (1) are the simplicity in the implementation,
reusing the existing infrastructure, avoiding any complications that the
new type variant has specifically around the different variants between
function and methods which would require the overload type to use `Type`
instead.

### Implementation

The core logic is how to collect all the overloaded functions. The way
this is done in this PR is by recording a **use** on the `Identifier`
node that represents the function name in the use-def map. This is then
used to fetch the previous symbol using the same name. This way the
signatures are going to be propagated from top to bottom (from first
overload to the final overload or the implementation) with each function
/ method. For example:

```py
from typing import overload

@overload
def foo(x: int) -> int: ...
@overload
def foo(x: str) -> str: ...
def foo(x: int | str) -> int | str:
	return x
```

Here, each definition of `foo` knows about all the signatures that comes
before itself. So, the first overload would only see itself, the second
would see the first and itself and so on until the implementation or the
final overload.

This approach required some updates specifically recognizing
`Identifier` node to record the function use because it doesn't use
`ExprName`.

## Test Plan

Update existing test cases which were limited by the overload support
and add test cases for the following cases:
* Valid overloads as functions, methods, generics, version specific
* Invalid overloads as stated in
https://typing.python.org/en/latest/spec/overload.html#invalid-overload-definitions
(implementation will be done in a follow-up)
* Various relation: fully static, subtyping, and assignability (others
in a follow-up)

## Ecosystem changes

_WIP_

After going through the ecosystem changes (there are a lot!), here's
what I've found:

We need assignability check between a callable type and a class literal
because a lot of builtins are defined as classes in typeshed whose
constructor method is overloaded e.g., `map`, `sorted`, `list.sort`,
`max`, `min` with the `key` parameter, `collections.abc.defaultdict`,
etc. (https://github.com/astral-sh/ruff/issues/17343). This makes up
most of the ecosystem diff **roughly 70 diagnostics**. For example:

```py
from collections import defaultdict

# red-knot: No overload of bound method `__init__` matches arguments [lint:no-matching-overload]
defaultdict(int)
# red-knot: No overload of bound method `__init__` matches arguments [lint:no-matching-overload]
defaultdict(list)

class Foo:
    def __init__(self, x: int):
        self.x = x

# red-knot: No overload of function `__new__` matches arguments [lint:no-matching-overload]
map(Foo, ["a", "b", "c"])
```

Duplicate diagnostics in unpacking
(https://github.com/astral-sh/ruff/issues/16514) has **~16
diagnostics**.

Support for the `callable` builtin which requires `TypeIs` support. This
is **5 diagnostics**. For example:
```py
from typing import Any

def _(x: Any | None) -> None:
    if callable(x):
        # red-knot: `Any | None`
        # Pyright: `(...) -> object`
        # mypy: `Any`
        # pyrefly: `(...) -> object`
        reveal_type(x)
```

Narrowing on `assert` which has **11 diagnostics**. This is being worked
on in https://github.com/astral-sh/ruff/pull/17345. For example:
```py
import re

match = re.search("", "")
assert match
match.group()  # error: [possibly-unbound-attribute]
```

Others:
* `Self`: 2
* Type aliases: 6
* Generics: 3
* Protocols: 13
* Unpacking in comprehension: 1
(https://github.com/astral-sh/ruff/pull/17396)

## Performance

Refer to
https://github.com/astral-sh/ruff/pull/17366#issuecomment-2814053046.

2025-04-18 09:57:40 +05:30

14 KiB

Raw Blame History

Overloads

Reference: https://typing.python.org/en/latest/spec/overload.html

`typing.overload`

The definition of typing.overload in typeshed is an identity function.

from typing import overload

def foo(x: int) -> int:
    return x

reveal_type(foo)  # revealed: def foo(x: int) -> int
bar = overload(foo)
reveal_type(bar)  # revealed: def foo(x: int) -> int

Functions

from typing import overload

@overload
def add() -> None: ...
@overload
def add(x: int) -> int: ...
@overload
def add(x: int, y: int) -> int: ...
def add(x: int | None = None, y: int | None = None) -> int | None:
    return (x or 0) + (y or 0)

reveal_type(add)  # revealed: Overload[() -> None, (x: int) -> int, (x: int, y: int) -> int]
reveal_type(add())  # revealed: None
reveal_type(add(1))  # revealed: int
reveal_type(add(1, 2))  # revealed: int

Overriding

These scenarios are to verify that the overloaded and non-overloaded definitions are correctly overridden by each other.

An overloaded function is overriding another overloaded function:

from typing import overload

@overload
def foo() -> None: ...
@overload
def foo(x: int) -> int: ...
def foo(x: int | None = None) -> int | None:
    return x

reveal_type(foo)  # revealed: Overload[() -> None, (x: int) -> int]
reveal_type(foo())  # revealed: None
reveal_type(foo(1))  # revealed: int

@overload
def foo() -> None: ...
@overload
def foo(x: str) -> str: ...
def foo(x: str | None = None) -> str | None:
    return x

reveal_type(foo)  # revealed: Overload[() -> None, (x: str) -> str]
reveal_type(foo())  # revealed: None
reveal_type(foo(""))  # revealed: str

A non-overloaded function is overriding an overloaded function:

def foo(x: int) -> int:
    return x

reveal_type(foo)  # revealed: def foo(x: int) -> int

An overloaded function is overriding a non-overloaded function:

reveal_type(foo)  # revealed: def foo(x: int) -> int

@overload
def foo() -> None: ...
@overload
def foo(x: bytes) -> bytes: ...
def foo(x: bytes | None = None) -> bytes | None:
    return x

reveal_type(foo)  # revealed: Overload[() -> None, (x: bytes) -> bytes]
reveal_type(foo())  # revealed: None
reveal_type(foo(b""))  # revealed: bytes

Methods

from typing import overload

class Foo1:
    @overload
    def method(self) -> None: ...
    @overload
    def method(self, x: int) -> int: ...
    def method(self, x: int | None = None) -> int | None:
        return x

foo1 = Foo1()
reveal_type(foo1.method)  # revealed: Overload[() -> None, (x: int) -> int]
reveal_type(foo1.method())  # revealed: None
reveal_type(foo1.method(1))  # revealed: int

class Foo2:
    @overload
    def method(self) -> None: ...
    @overload
    def method(self, x: str) -> str: ...
    def method(self, x: str | None = None) -> str | None:
        return x

foo2 = Foo2()
reveal_type(foo2.method)  # revealed: Overload[() -> None, (x: str) -> str]
reveal_type(foo2.method())  # revealed: None
reveal_type(foo2.method(""))  # revealed: str

Constructor

from typing import overload

class Foo:
    @overload
    def __init__(self) -> None: ...
    @overload
    def __init__(self, x: int) -> None: ...
    def __init__(self, x: int | None = None) -> None:
        self.x = x

foo = Foo()
reveal_type(foo)  # revealed: Foo
reveal_type(foo.x)  # revealed: Unknown | int | None

foo1 = Foo(1)
reveal_type(foo1)  # revealed: Foo
reveal_type(foo1.x)  # revealed: Unknown | int | None

Version specific

Function definitions can vary between multiple Python versions.

Overload and non-overload (3.9)

Here, the same function is overloaded in one version and not in another.

[environment]
python-version = "3.9"

import sys
from typing import overload

if sys.version_info < (3, 10):
    def func(x: int) -> int:
        return x

elif sys.version_info <= (3, 12):
    @overload
    def func() -> None: ...
    @overload
    def func(x: int) -> int: ...
    def func(x: int | None = None) -> int | None:
        return x

reveal_type(func)  # revealed: def func(x: int) -> int
func()  # error: [missing-argument]

Overload and non-overload (3.10)

[environment]
python-version = "3.10"

import sys
from typing import overload

if sys.version_info < (3, 10):
    def func(x: int) -> int:
        return x

elif sys.version_info <= (3, 12):
    @overload
    def func() -> None: ...
    @overload
    def func(x: int) -> int: ...
    def func(x: int | None = None) -> int | None:
        return x

reveal_type(func)  # revealed: Overload[() -> None, (x: int) -> int]
reveal_type(func())  # revealed: None
reveal_type(func(1))  # revealed: int

Some overloads are version specific (3.9)

[environment]
python-version = "3.9"

overloaded.pyi:

import sys
from typing import overload

if sys.version_info >= (3, 10):
    @overload
    def func() -> None: ...

@overload
def func(x: int) -> int: ...
@overload
def func(x: str) -> str: ...

main.py:

from overloaded import func

reveal_type(func)  # revealed: Overload[(x: int) -> int, (x: str) -> str]
func()  # error: [no-matching-overload]
reveal_type(func(1))  # revealed: int
reveal_type(func(""))  # revealed: str

Some overloads are version specific (3.10)

[environment]
python-version = "3.10"

overloaded.pyi:

import sys
from typing import overload

@overload
def func() -> None: ...

if sys.version_info >= (3, 10):
    @overload
    def func(x: int) -> int: ...

@overload
def func(x: str) -> str: ...

main.py:

from overloaded import func

reveal_type(func)  # revealed: Overload[() -> None, (x: int) -> int, (x: str) -> str]
reveal_type(func())  # revealed: None
reveal_type(func(1))  # revealed: int
reveal_type(func(""))  # revealed: str

Generic

[environment]
python-version = "3.12"

For an overloaded generic function, it's not necessary for all overloads to be generic.

from typing import overload

@overload
def func() -> None: ...
@overload
def func[T](x: T) -> T: ...
def func[T](x: T | None = None) -> T | None:
    return x

reveal_type(func)  # revealed: Overload[() -> None, (x: T) -> T]
reveal_type(func())  # revealed: None
reveal_type(func(1))  # revealed: Literal[1]
reveal_type(func(""))  # revealed: Literal[""]

Invalid

At least two overloads

At least two @overload-decorated definitions must be present.

from typing import overload

# TODO: error
@overload
def func(x: int) -> int: ...
def func(x: int | str) -> int | str:
    return x

Overload without an implementation

Regular modules

In regular modules, a series of @overload-decorated definitions must be followed by exactly one non-@overload-decorated definition (for the same function/method).

from typing import overload

# TODO: error because implementation does not exists
@overload
def func(x: int) -> int: ...
@overload
def func(x: str) -> str: ...

class Foo:
    # TODO: error because implementation does not exists
    @overload
    def method(self, x: int) -> int: ...
    @overload
    def method(self, x: str) -> str: ...

Stub files

Overload definitions within stub files are exempt from this check.

from typing import overload

@overload
def func(x: int) -> int: ...
@overload
def func(x: str) -> str: ...

Protocols

Overload definitions within protocols are exempt from this check.

from typing import Protocol, overload

class Foo(Protocol):
    @overload
    def f(self, x: int) -> int: ...
    @overload
    def f(self, x: str) -> str: ...

Abstract methods

Overload definitions within abstract base classes are exempt from this check.

from abc import ABC, abstractmethod
from typing import overload

class AbstractFoo(ABC):
    @overload
    @abstractmethod
    def f(self, x: int) -> int: ...
    @overload
    @abstractmethod
    def f(self, x: str) -> str: ...

Using the @abstractmethod decorator requires that the class's metaclass is ABCMeta or is derived from it.

class Foo:
    # TODO: Error because implementation does not exists
    @overload
    @abstractmethod
    def f(self, x: int) -> int: ...
    @overload
    @abstractmethod
    def f(self, x: str) -> str: ...

And, the @abstractmethod decorator must be present on all the @overload-ed methods.

class PartialFoo1(ABC):
    @overload
    @abstractmethod
    def f(self, x: int) -> int: ...
    @overload
    def f(self, x: str) -> str: ...

class PartialFoo(ABC):
    @overload
    def f(self, x: int) -> int: ...
    @overload
    @abstractmethod
    def f(self, x: str) -> str: ...

Inconsistent decorators

`@staticmethod` / `@classmethod`

If one overload signature is decorated with @staticmethod or @classmethod, all overload signatures must be similarly decorated. The implementation, if present, must also have a consistent decorator.

from __future__ import annotations

from typing import overload

class CheckStaticMethod:
    # TODO: error because `@staticmethod` does not exist on all overloads
    @overload
    def method1(x: int) -> int: ...
    @overload
    def method1(x: str) -> str: ...
    @staticmethod
    def method1(x: int | str) -> int | str:
        return x
    # TODO: error because `@staticmethod` does not exist on all overloads
    @overload
    def method2(x: int) -> int: ...
    @overload
    @staticmethod
    def method2(x: str) -> str: ...
    @staticmethod
    def method2(x: int | str) -> int | str:
        return x
    # TODO: error because `@staticmethod` does not exist on the implementation
    @overload
    @staticmethod
    def method3(x: int) -> int: ...
    @overload
    @staticmethod
    def method3(x: str) -> str: ...
    def method3(x: int | str) -> int | str:
        return x

    @overload
    @staticmethod
    def method4(x: int) -> int: ...
    @overload
    @staticmethod
    def method4(x: str) -> str: ...
    @staticmethod
    def method4(x: int | str) -> int | str:
        return x

class CheckClassMethod:
    def __init__(self, x: int) -> None:
        self.x = x
    # TODO: error because `@classmethod` does not exist on all overloads
    @overload
    @classmethod
    def try_from1(cls, x: int) -> CheckClassMethod: ...
    @overload
    def try_from1(cls, x: str) -> None: ...
    @classmethod
    def try_from1(cls, x: int | str) -> CheckClassMethod | None:
        if isinstance(x, int):
            return cls(x)
        return None
    # TODO: error because `@classmethod` does not exist on all overloads
    @overload
    def try_from2(cls, x: int) -> CheckClassMethod: ...
    @overload
    @classmethod
    def try_from2(cls, x: str) -> None: ...
    @classmethod
    def try_from2(cls, x: int | str) -> CheckClassMethod | None:
        if isinstance(x, int):
            return cls(x)
        return None
    # TODO: error because `@classmethod` does not exist on the implementation
    @overload
    @classmethod
    def try_from3(cls, x: int) -> CheckClassMethod: ...
    @overload
    @classmethod
    def try_from3(cls, x: str) -> None: ...
    def try_from3(cls, x: int | str) -> CheckClassMethod | None:
        if isinstance(x, int):
            return cls(x)
        return None

    @overload
    @classmethod
    def try_from4(cls, x: int) -> CheckClassMethod: ...
    @overload
    @classmethod
    def try_from4(cls, x: str) -> None: ...
    @classmethod
    def try_from4(cls, x: int | str) -> CheckClassMethod | None:
        if isinstance(x, int):
            return cls(x)
        return None

`@final` / `@override`

If a @final or @override decorator is supplied for a function with overloads, the decorator should be applied only to the overload implementation if it is present.

from typing_extensions import final, overload, override

class Foo:
    @overload
    def method1(self, x: int) -> int: ...
    @overload
    def method1(self, x: str) -> str: ...
    @final
    def method1(self, x: int | str) -> int | str:
        return x
    # TODO: error because `@final` is not on the implementation
    @overload
    @final
    def method2(self, x: int) -> int: ...
    @overload
    def method2(self, x: str) -> str: ...
    def method2(self, x: int | str) -> int | str:
        return x
    # TODO: error because `@final` is not on the implementation
    @overload
    def method3(self, x: int) -> int: ...
    @overload
    @final
    def method3(self, x: str) -> str: ...
    def method3(self, x: int | str) -> int | str:
        return x

class Base:
    @overload
    def method(self, x: int) -> int: ...
    @overload
    def method(self, x: str) -> str: ...
    def method(self, x: int | str) -> int | str:
        return x

class Sub1(Base):
    @overload
    def method(self, x: int) -> int: ...
    @overload
    def method(self, x: str) -> str: ...
    @override
    def method(self, x: int | str) -> int | str:
        return x

class Sub2(Base):
    # TODO: error because `@override` is not on the implementation
    @overload
    def method(self, x: int) -> int: ...
    @overload
    @override
    def method(self, x: str) -> str: ...
    def method(self, x: int | str) -> int | str:
        return x

class Sub3(Base):
    # TODO: error because `@override` is not on the implementation
    @overload
    @override
    def method(self, x: int) -> int: ...
    @overload
    def method(self, x: str) -> str: ...
    def method(self, x: int | str) -> int | str:
        return x

`@final` / `@override` in stub files

If an overload implementation isn’t present (for example, in a stub file), the @final or @override decorator should be applied only to the first overload.

from typing_extensions import final, overload, override

class Foo:
    @overload
    @final
    def method1(self, x: int) -> int: ...
    @overload
    def method1(self, x: str) -> str: ...

    # TODO: error because `@final` is not on the first overload
    @overload
    def method2(self, x: int) -> int: ...
    @final
    @overload
    def method2(self, x: str) -> str: ...

class Base:
    @overload
    def method(self, x: int) -> int: ...
    @overload
    def method(self, x: str) -> str: ...

class Sub1(Base):
    @overload
    @override
    def method(self, x: int) -> int: ...
    @overload
    def method(self, x: str) -> str: ...

class Sub2(Base):
    # TODO: error because `@override` is not on the first overload
    @overload
    def method(self, x: int) -> int: ...
    @overload
    @override
    def method(self, x: str) -> str: ...

14 KiB Raw Blame History Unescape Escape