language-servers/ruff
1
0
Fork 0
mirror of https://github.com/astral-sh/ruff.git synced 2025-11-03 13:23:10 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

[ty] Infer nonlocal types as unions of all reachable bindings (#18750)

Browse source 
## Summary

This PR includes a behavioral change to how we infer types for public
uses of symbols within a module. Where we would previously use the type
that a use at the end of the scope would see, we now consider all
reachable bindings and union the results:

```py
x = None

def f():
    reveal_type(x)  # previously `Unknown | Literal[1]`, now `Unknown | None | Literal[1]`

f()

x = 1

f()
```

This helps especially in cases where the the end of the scope is not
reachable:

```py
def outer(x: int):
    def inner():
        reveal_type(x)  # previously `Unknown`, now `int`

    raise ValueError
```

This PR also proposes to skip the boundness analysis of public uses.
This is consistent with the "all reachable bindings" strategy, because
the implicit `x = <unbound>` binding is also always reachable, and we
would have to emit "possibly-unresolved" diagnostics for every public
use otherwise. Changing this behavior allows common use-cases like the
following to type check without any errors:

```py
def outer(flag: bool):
    if flag:
        x = 1

        def inner():
            print(x)  # previously: possibly-unresolved-reference, now: no error
```

closes https://github.com/astral-sh/ty/issues/210
closes https://github.com/astral-sh/ty/issues/607
closes https://github.com/astral-sh/ty/issues/699

## Follow up

It is now possible to resolve the following TODO, but I would like to do
that as a follow-up, because it requires some changes to how we treat
implicit attribute assignments, which could result in ecosystem changes
that I'd like to see separately.


315fb0f3da/crates/ty_python_semantic/src/semantic_index/builder.rs (L1095-L1117)

## Ecosystem analysis

[**Full report**](https://shark.fish/diff-public-types.html)

* This change obviously removes a lot of `possibly-unresolved-reference`
diagnostics (7818) because we do not analyze boundness for public uses
of symbols inside modules anymore.
* As the primary goal here, this change also removes a lot of
false-positive `unresolved-reference` diagnostics (231) in scenarios
like this:
    ```py
    def _(flag: bool):
        if flag:
            x = 1
    
            def inner():
                x
    
            raise
    ```
* This change also introduces some new false positives for cases like:
    ```py
    def _():
        x = None
    
        x = "test"
    
        def inner():
x.upper() # Attribute `upper` on type `Unknown | None | Literal["test"]`
is possibly unbound
    ```
We have test cases for these situations and it's plausible that we can
improve this in a follow-up.


## Test Plan

New Markdown tests
This commit is contained in:
David Peter 2025-06-26 12:24:40 +02:00 • committed by GitHub
parent 2362263d5e
commit b01003f81d
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
17 changed files with 983 additions and 171 deletions
Download patch file Download diff file Expand all files Collapse all files

121
crates/ty_python_semantic/src/semantic_index/use_def.rs
View file

@ -237,6 +237,7 @@ use self::place_state::{
LiveDeclarationsIterator, PlaceState, ScopedDefinitionId,
};
use crate::node_key::NodeKey;
use crate::place::BoundnessAnalysis;
use crate::semantic_index::ast_ids::ScopedUseId;
use crate::semantic_index::definition::{Definition, DefinitionState};
use crate::semantic_index::narrowing_constraints::{
@ -251,6 +252,7 @@ use crate::semantic_index::predicate::{
use crate::semantic_index::reachability_constraints::{
ReachabilityConstraints, ReachabilityConstraintsBuilder, ScopedReachabilityConstraintId,
};
use crate::semantic_index::use_def::place_state::PreviousDefinitions;
use crate::semantic_index::{EagerSnapshotResult, SemanticIndex};
use crate::types::{IntersectionBuilder, Truthiness, Type, infer_narrowing_constraint};
@ -296,7 +298,10 @@ pub(crate) struct UseDefMap<'db> {
bindings_by_declaration: FxHashMap<Definition<'db>, Bindings>,
/// [`PlaceState`] visible at end of scope for each place.
public_places: IndexVec<ScopedPlaceId, PlaceState>,
end_of_scope_places: IndexVec<ScopedPlaceId, PlaceState>,
/// All potentially reachable bindings and declarations, for each place.
reachable_definitions: IndexVec<ScopedPlaceId, ReachableDefinitions>,
/// Snapshot of bindings in this scope that can be used to resolve a reference in a nested
/// eager scope.
@ -332,7 +337,10 @@ impl<'db> UseDefMap<'db> {
&self,
use_id: ScopedUseId,
) -> BindingWithConstraintsIterator<'_, 'db> {
self.bindings_iterator(&self.bindings_by_use[use_id])
self.bindings_iterator(
&self.bindings_by_use[use_id],
BoundnessAnalysis::BasedOnUnboundVisibility,
)
}
pub(crate) fn applicable_constraints(
@ -394,11 +402,24 @@ impl<'db> UseDefMap<'db> {
.may_be_true()
}
pub(crate) fn public_bindings(
pub(crate) fn end_of_scope_bindings(
&self,
place: ScopedPlaceId,
) -> BindingWithConstraintsIterator<'_, 'db> {
self.bindings_iterator(self.public_places[place].bindings())
self.bindings_iterator(
self.end_of_scope_places[place].bindings(),
BoundnessAnalysis::BasedOnUnboundVisibility,
)
}
pub(crate) fn all_reachable_bindings(
&self,
place: ScopedPlaceId,
) -> BindingWithConstraintsIterator<'_, 'db> {
self.bindings_iterator(
&self.reachable_definitions[place].bindings,
BoundnessAnalysis::AssumeBound,
)
}
pub(crate) fn eager_snapshot(
@ -409,9 +430,9 @@ impl<'db> UseDefMap<'db> {
Some(EagerSnapshot::Constraint(constraint)) => {
EagerSnapshotResult::FoundConstraint(*constraint)
}
Some(EagerSnapshot::Bindings(bindings)) => {
EagerSnapshotResult::FoundBindings(self.bindings_iterator(bindings))
}
Some(EagerSnapshot::Bindings(bindings)) => EagerSnapshotResult::FoundBindings(
self.bindings_iterator(bindings, BoundnessAnalysis::BasedOnUnboundVisibility),
),
None => EagerSnapshotResult::NotFound,
}
}
@ -420,39 +441,53 @@ impl<'db> UseDefMap<'db> {
&self,
declaration: Definition<'db>,
) -> BindingWithConstraintsIterator<'_, 'db> {
self.bindings_iterator(&self.bindings_by_declaration[&declaration])
self.bindings_iterator(
&self.bindings_by_declaration[&declaration],
BoundnessAnalysis::BasedOnUnboundVisibility,
)
}
pub(crate) fn declarations_at_binding(
&self,
binding: Definition<'db>,
) -> DeclarationsIterator<'_, 'db> {
self.declarations_iterator(&self.declarations_by_binding[&binding])
self.declarations_iterator(
&self.declarations_by_binding[&binding],
BoundnessAnalysis::BasedOnUnboundVisibility,
)
}
pub(crate) fn public_declarations<'map>(
pub(crate) fn end_of_scope_declarations<'map>(
&'map self,
place: ScopedPlaceId,
) -> DeclarationsIterator<'map, 'db> {
let declarations = self.public_places[place].declarations();
self.declarations_iterator(declarations)
let declarations = self.end_of_scope_places[place].declarations();
self.declarations_iterator(declarations, BoundnessAnalysis::BasedOnUnboundVisibility)
}
pub(crate) fn all_public_declarations<'map>(
pub(crate) fn all_reachable_declarations(
&self,
place: ScopedPlaceId,
) -> DeclarationsIterator<'_, 'db> {
let declarations = &self.reachable_definitions[place].declarations;
self.declarations_iterator(declarations, BoundnessAnalysis::AssumeBound)
}
pub(crate) fn all_end_of_scope_declarations<'map>(
&'map self,
) -> impl Iterator<Item = (ScopedPlaceId, DeclarationsIterator<'map, 'db>)> + 'map {
(0..self.public_places.len())
(0..self.end_of_scope_places.len())
.map(ScopedPlaceId::from_usize)
.map(|place_id| (place_id, self.public_declarations(place_id)))
.map(|place_id| (place_id, self.end_of_scope_declarations(place_id)))
}
pub(crate) fn all_public_bindings<'map>(
pub(crate) fn all_end_of_scope_bindings<'map>(
&'map self,
) -> impl Iterator<Item = (ScopedPlaceId, BindingWithConstraintsIterator<'map, 'db>)> + 'map
{
(0..self.public_places.len())
(0..self.end_of_scope_places.len())
.map(ScopedPlaceId::from_usize)
.map(|place_id| (place_id, self.public_bindings(place_id)))
.map(|place_id| (place_id, self.end_of_scope_bindings(place_id)))
}
/// This function is intended to be called only once inside `TypeInferenceBuilder::infer_function_body`.
@ -478,12 +513,14 @@ impl<'db> UseDefMap<'db> {
fn bindings_iterator<'map>(
&'map self,
bindings: &'map Bindings,
boundness_analysis: BoundnessAnalysis,
) -> BindingWithConstraintsIterator<'map, 'db> {
BindingWithConstraintsIterator {
all_definitions: &self.all_definitions,
predicates: &self.predicates,
narrowing_constraints: &self.narrowing_constraints,
reachability_constraints: &self.reachability_constraints,
boundness_analysis,
inner: bindings.iter(),
}
}
@ -491,11 +528,13 @@ impl<'db> UseDefMap<'db> {
fn declarations_iterator<'map>(
&'map self,
declarations: &'map Declarations,
boundness_analysis: BoundnessAnalysis,
) -> DeclarationsIterator<'map, 'db> {
DeclarationsIterator {
all_definitions: &self.all_definitions,
predicates: &self.predicates,
reachability_constraints: &self.reachability_constraints,
boundness_analysis,
inner: declarations.iter(),
}
}
@ -531,6 +570,7 @@ pub(crate) struct BindingWithConstraintsIterator<'map, 'db> {
pub(crate) predicates: &'map Predicates<'db>,
pub(crate) narrowing_constraints: &'map NarrowingConstraints,
pub(crate) reachability_constraints: &'map ReachabilityConstraints,
pub(crate) boundness_analysis: BoundnessAnalysis,
inner: LiveBindingsIterator<'map>,
}
@ -611,6 +651,7 @@ pub(crate) struct DeclarationsIterator<'map, 'db> {
all_definitions: &'map IndexVec<ScopedDefinitionId, DefinitionState<'db>>,
pub(crate) predicates: &'map Predicates<'db>,
pub(crate) reachability_constraints: &'map ReachabilityConstraints,
pub(crate) boundness_analysis: BoundnessAnalysis,
inner: LiveDeclarationsIterator<'map>,
}
@ -639,6 +680,12 @@ impl<'db> Iterator for DeclarationsIterator<'_, 'db> {
impl std::iter::FusedIterator for DeclarationsIterator<'_, '_> {}
#[derive(Debug, PartialEq, Eq, salsa::Update)]
struct ReachableDefinitions {
bindings: Bindings,
declarations: Declarations,
}
/// A snapshot of the definitions and constraints state at a particular point in control flow.
#[derive(Clone, Debug)]
pub(super) struct FlowSnapshot {
@ -648,7 +695,7 @@ pub(super) struct FlowSnapshot {
#[derive(Debug)]
pub(super) struct UseDefMapBuilder<'db> {
/// Append-only array of [`Definition`].
/// Append-only array of [`DefinitionState`].
all_definitions: IndexVec<ScopedDefinitionId, DefinitionState<'db>>,
/// Builder of predicates.
@ -679,6 +726,9 @@ pub(super) struct UseDefMapBuilder<'db> {
/// Currently live bindings and declarations for each place.
place_states: IndexVec<ScopedPlaceId, PlaceState>,
/// All potentially reachable bindings and declarations, for each place.
reachable_definitions: IndexVec<ScopedPlaceId, ReachableDefinitions>,
/// Snapshots of place states in this scope that can be used to resolve a reference in a
/// nested eager scope.
eager_snapshots: EagerSnapshots,
@ -700,6 +750,7 @@ impl<'db> UseDefMapBuilder<'db> {
declarations_by_binding: FxHashMap::default(),
bindings_by_declaration: FxHashMap::default(),
place_states: IndexVec::new(),
reachable_definitions: IndexVec::new(),
eager_snapshots: EagerSnapshots::default(),
is_class_scope,
}
@ -720,6 +771,11 @@ impl<'db> UseDefMapBuilder<'db> {
.place_states
.push(PlaceState::undefined(self.reachability));
debug_assert_eq!(place, new_place);
let new_place = self.reachable_definitions.push(ReachableDefinitions {
bindings: Bindings::unbound(self.reachability),
declarations: Declarations::undeclared(self.reachability),
});
debug_assert_eq!(place, new_place);
}
pub(super) fn record_binding(
@ -738,6 +794,14 @@ impl<'db> UseDefMapBuilder<'db> {
self.is_class_scope,
is_place_name,
);
self.reachable_definitions[place].bindings.record_binding(
def_id,
self.reachability,
self.is_class_scope,
is_place_name,
PreviousDefinitions::AreKept,
);
}
pub(super) fn add_predicate(&mut self, predicate: Predicate<'db>) -> ScopedPredicateId {
@ -845,6 +909,10 @@ impl<'db> UseDefMapBuilder<'db> {
self.bindings_by_declaration
.insert(declaration, place_state.bindings().clone());
place_state.record_declaration(def_id, self.reachability);
self.reachable_definitions[place]
.declarations
.record_declaration(def_id, self.reachability, PreviousDefinitions::AreKept);
}
pub(super) fn record_declaration_and_binding(
@ -866,6 +934,17 @@ impl<'db> UseDefMapBuilder<'db> {
self.is_class_scope,
is_place_name,
);
self.reachable_definitions[place]
.declarations
.record_declaration(def_id, self.reachability, PreviousDefinitions::AreKept);
self.reachable_definitions[place].bindings.record_binding(
def_id,
self.reachability,
self.is_class_scope,
is_place_name,
PreviousDefinitions::AreKept,
);
}
pub(super) fn delete_binding(&mut self, place: ScopedPlaceId, is_place_name: bool) {
@ -1000,6 +1079,7 @@ impl<'db> UseDefMapBuilder<'db> {
pub(super) fn finish(mut self) -> UseDefMap<'db> {
self.all_definitions.shrink_to_fit();
self.place_states.shrink_to_fit();
self.reachable_definitions.shrink_to_fit();
self.bindings_by_use.shrink_to_fit();
self.node_reachability.shrink_to_fit();
self.declarations_by_binding.shrink_to_fit();
@ -1013,7 +1093,8 @@ impl<'db> UseDefMapBuilder<'db> {
reachability_constraints: self.reachability_constraints.build(),
bindings_by_use: self.bindings_by_use,
node_reachability: self.node_reachability,
public_places: self.place_states,
end_of_scope_places: self.place_states,
reachable_definitions: self.reachable_definitions,
declarations_by_binding: self.declarations_by_binding,
bindings_by_declaration: self.bindings_by_declaration,
eager_snapshots: self.eager_snapshots,

38
crates/ty_python_semantic/src/semantic_index/use_def/place_state.rs
View file

@ -92,8 +92,20 @@ pub(super) struct LiveDeclaration {
pub(super) type LiveDeclarationsIterator<'a> = std::slice::Iter<'a, LiveDeclaration>;
#[derive(Clone, Copy, Debug)]
pub(super) enum PreviousDefinitions {
AreShadowed,
AreKept,
}
impl PreviousDefinitions {
pub(super) fn are_shadowed(self) -> bool {
matches!(self, PreviousDefinitions::AreShadowed)
}
}
impl Declarations {
fn undeclared(reachability_constraint: ScopedReachabilityConstraintId) -> Self {
pub(super) fn undeclared(reachability_constraint: ScopedReachabilityConstraintId) -> Self {
let initial_declaration = LiveDeclaration {
declaration: ScopedDefinitionId::UNBOUND,
reachability_constraint,
@ -104,13 +116,16 @@ impl Declarations {
}
/// Record a newly-encountered declaration for this place.
fn record_declaration(
pub(super) fn record_declaration(
&mut self,
declaration: ScopedDefinitionId,
reachability_constraint: ScopedReachabilityConstraintId,
previous_definitions: PreviousDefinitions,
) {
// The new declaration replaces all previous live declaration in this path.
self.live_declarations.clear();
if previous_definitions.are_shadowed() {
// The new declaration replaces all previous live declaration in this path.
self.live_declarations.clear();
}
self.live_declarations.push(LiveDeclaration {
declaration,
reachability_constraint,
@ -205,7 +220,7 @@ pub(super) struct LiveBinding {
pub(super) type LiveBindingsIterator<'a> = std::slice::Iter<'a, LiveBinding>;
impl Bindings {
fn unbound(reachability_constraint: ScopedReachabilityConstraintId) -> Self {
pub(super) fn unbound(reachability_constraint: ScopedReachabilityConstraintId) -> Self {
let initial_binding = LiveBinding {
binding: ScopedDefinitionId::UNBOUND,
narrowing_constraint: ScopedNarrowingConstraint::empty(),
@ -224,6 +239,7 @@ impl Bindings {
reachability_constraint: ScopedReachabilityConstraintId,
is_class_scope: bool,
is_place_name: bool,
previous_definitions: PreviousDefinitions,
) {
// If we are in a class scope, and the unbound name binding was previously visible, but we will
// now replace it, record the narrowing constraints on it:
@ -232,7 +248,9 @@ impl Bindings {
}
// The new binding replaces all previous live bindings in this path, and has no
// constraints.
self.live_bindings.clear();
if previous_definitions.are_shadowed() {
self.live_bindings.clear();
}
self.live_bindings.push(LiveBinding {
binding,
narrowing_constraint: ScopedNarrowingConstraint::empty(),
@ -349,6 +367,7 @@ impl PlaceState {
reachability_constraint,
is_class_scope,
is_place_name,
PreviousDefinitions::AreShadowed,
);
}
@ -380,8 +399,11 @@ impl PlaceState {
declaration_id: ScopedDefinitionId,
reachability_constraint: ScopedReachabilityConstraintId,
) {
self.declarations
.record_declaration(declaration_id, reachability_constraint);
self.declarations.record_declaration(
declaration_id,
reachability_constraint,
PreviousDefinitions::AreShadowed,
);
}
/// Merge another [`PlaceState`] into this one.