mirrors/ruff
1
0
Fork 0
mirror of https://github.com/astral-sh/ruff.git synced 2025-10-02 22:54:42 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

[ty] Reachability constraints (#18621)

Browse source 
## Summary



* Completely removes the concept of visibility constraints. Reachability
constraints are now used to model the static visibility of bindings and
declarations. Reachability constraints are *much* easier to reason about
/ work with, since they are applied at the beginning of a branch, and
not applied retroactively. Removing the duplication between visibility
and reachability constraints also leads to major code simplifications
[^1]. For an overview of how the new constraint system works, see the
updated doc comment in `reachability_constraints.rs`.
* Fixes a [control-flow modeling bug
(panic)](https://github.com/astral-sh/ty/issues/365) involving `break`
statements in loops
* Fixes a [bug where](https://github.com/astral-sh/ty/issues/624) where
`elif` branches would have wrong reachability constraints
* Fixes a [bug where](https://github.com/astral-sh/ty/issues/648) code
after infinite loops would not be considered unreachble
* Fixes a panic on the `pywin32` ecosystem project, which we should be
able to move to `good.txt` once this has been merged.
* Removes some false positives in unreachable code because we infer
`Never` more often, due to the fact that reachability constraints now
apply retroactively to *all* active bindings, not just to bindings
inside a branch.
* As one example, this removes the `division-by-zero` diagnostic from
https://github.com/astral-sh/ty/issues/443 because we now infer `Never`
for the divisor.
* Supersedes and includes similar test changes as
https://github.com/astral-sh/ruff/pull/18392


closes https://github.com/astral-sh/ty/issues/365
closes https://github.com/astral-sh/ty/issues/624
closes https://github.com/astral-sh/ty/issues/642
closes https://github.com/astral-sh/ty/issues/648

## Benchmarks

Benchmarks on black, pandas, and sympy showed that this is neither a
performance improvement, nor a regression.

## Test Plan

Regression tests for:
- [x] https://github.com/astral-sh/ty/issues/365
- [x] https://github.com/astral-sh/ty/issues/624
- [x] https://github.com/astral-sh/ty/issues/642
- [x] https://github.com/astral-sh/ty/issues/648

[^1]: I'm afraid this is something that @carljm advocated for since the
beginning, and I'm not sure anymore why we have never seriously tried
this before. So I suggest we do *not* attempt to do a historical deep
dive to find out exactly why this ever became so complicated, and just
enjoy the fact that we eventually arrived here.

---------

Co-authored-by: Carl Meyer <carl@astral.sh>
This commit is contained in:
David Peter 2025-06-17 09:24:28 +02:00 • committed by GitHub
parent c22f809049
commit 3a77768f79
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
18 changed files with 683 additions and 806 deletions
Download patch file Download diff file Expand all files Collapse all files

17
crates/ty_ide/src/completion.rs
View file

@ -1202,7 +1202,7 @@ print(f\"{some<CURSOR>
}
#[test]
fn statically_invisible_symbols() {
fn statically_unreachable_symbols() {
let test = cursor_test(
"\
if 1 + 2 != 3:
@ -1850,6 +1850,21 @@ def test_point(p2: Point):
test.assert_completions_include("orthogonal_direction");
}
#[test]
fn regression_test_issue_642() {
// Regression test for https://github.com/astral-sh/ty/issues/642
let test = cursor_test(
r#"
match 0:
case 1 i<CURSOR>:
pass
"#,
);
assert_snapshot!(test.completions(), @r"<No completions found>");
}
impl CursorTest {
fn completions(&self) -> String {
self.completions_if(|_| true)

4
crates/ty_python_semantic/resources/mdtest/import/star.md
View file

@ -827,8 +827,8 @@ if sys.version_info >= (3, 12):
from exporter import *
# it's correct to have no diagnostics here as this branch is unreachable
reveal_type(A) # revealed: Unknown
reveal_type(B) # revealed: bool
reveal_type(A) # revealed: Never
reveal_type(B) # revealed: Never
else:
from exporter import *

4
crates/ty_python_semantic/resources/mdtest/narrow/conditionals/boolean.md
View file

@ -52,7 +52,7 @@ def _(x: A | B):
if False and isinstance(x, A):
# TODO: should emit an `unreachable code` diagnostic
reveal_type(x) # revealed: A
reveal_type(x) # revealed: Never
else:
reveal_type(x) # revealed: A | B
@ -65,7 +65,7 @@ def _(x: A | B):
reveal_type(x) # revealed: A | B
else:
# TODO: should emit an `unreachable code` diagnostic
reveal_type(x) # revealed: B & ~A
reveal_type(x) # revealed: Never
reveal_type(x) # revealed: A | B
```

2
crates/ty_python_semantic/resources/mdtest/narrow/truthiness.md
View file

@ -199,7 +199,7 @@ def f(x: Literal[0, 1], y: Literal["", "hello"]):
reveal_type(y) # revealed: Literal["", "hello"]
if (x or not x) and (y and not y):
reveal_type(x) # revealed: Literal[0, 1]
reveal_type(x) # revealed: Never
reveal_type(y) # revealed: Never
else:
# ~(x or not x) or ~(y and not y)

3
crates/ty_python_semantic/resources/mdtest/narrow/type.md
View file

@ -127,7 +127,8 @@ class B: ...
def _[T](x: A | B):
if type(x) is A[str]:
reveal_type(x) # revealed: (A[int] & A[Unknown]) | (B & A[Unknown])
# `type()` never returns a generic alias, so `type(x)` cannot be `A[str]`
reveal_type(x) # revealed: Never
else:
reveal_type(x) # revealed: A[int] | B
```

33
crates/ty_python_semantic/resources/mdtest/statically_known_branches.md
View file

@ -994,6 +994,39 @@ else:
reveal_type(x) # revealed: Literal[1]
```
#### `if` nested inside `while True`
These are regression test for <https://github.com/astral-sh/ty/issues/365>. First, make sure that we
do not panic in the original scenario:
```py
def flag() -> bool:
return True
while True:
if flag():
break
else:
c = 1
break
c # error: [possibly-unresolved-reference]
```
And also check that we understand control flow correctly:
```py
c = 1
while True:
if False:
c = 2
break
break
reveal_type(c) # revealed: Literal[1]
```
## `match` statements
```toml

64
crates/ty_python_semantic/resources/mdtest/unreachable.md
View file

@ -72,6 +72,17 @@ def f2():
# TODO: we should mark this as unreachable
print("unreachable")
def f3():
if False:
return
elif True:
return
else:
pass
# TODO: we should mark this as unreachable
print("unreachable")
```
### `Never` / `NoReturn`
@ -211,9 +222,8 @@ reachable or not. Some developers like to use things like early `return` stateme
and for this use case, it is helpful to still see some diagnostics in unreachable sections.
We currently follow the second approach, but we do not attempt to provide the full set of
diagnostics in unreachable sections. In fact, we silence a certain category of diagnostics
(`unresolved-reference`, `unresolved-attribute`, …), in order to avoid *incorrect* diagnostics. In
the future, we may revisit this decision.
diagnostics in unreachable sections. In fact, a large number of diagnostics are suppressed in
unreachable code, simply due to the fact that we infer `Never` for most of the symbols.
### Use of variables in unreachable code
@ -301,7 +311,8 @@ elif sys.version_info >= (3, 11):
elif sys.version_info >= (3, 10):
pass
else:
pass
# This branch is also unreachable, because the previous `elif` branch is always true
ExceptionGroup # no error here
```
And for nested `if` statements:
@ -378,6 +389,18 @@ while sys.version_info >= (3, 11):
ExceptionGroup
```
### Infinite loops
We also do not emit diagnostics in unreachable code after an infinite loop:
```py
def f():
while True:
pass
ExceptionGroup # no error here
```
### Silencing errors for actually unknown symbols
We currently also silence diagnostics for symbols that are not actually defined anywhere. It is
@ -500,33 +523,26 @@ def f():
1 / 0 # error: [division-by-zero]
```
## Limitations of the current approach
### Conflicting type information
The current approach of silencing only a subset of diagnostics in unreachable code leads to some
problems, and we may want to re-evaluate this decision in the future. To illustrate, consider the
following example:
We also support cases where type information for symbols conflicts between mutually exclusive
branches:
```py
if False:
import sys
if sys.version_info >= (3, 11):
x: int = 1
else:
x: str = "a"
if False:
# TODO We currently emit a false positive here:
# error: [invalid-assignment] "Object of type `Literal["a"]` is not assignable to `int`"
if sys.version_info >= (3, 11):
other: int = x
else:
other: str = x
```
The problem here originates from the fact that the type of `x` in the `False` branch conflicts with
the visible type of `x` in the `True` branch. When we type-check the lower `False` branch, we only
see the visible definition of `x`, which has a type of `str`.
In principle, this means that all diagnostics that depend on type information from "outside" the
unreachable section should be silenced. Similar problems to the one above can occur for other rule
types as well:
This is also supported for function calls, attribute accesses, etc.:
```py
from typing import Literal
@ -554,24 +570,14 @@ else:
number: Literal[0] = 0
if False:
# TODO
# error: [invalid-argument-type]
f(2)
# TODO
# error: [unknown-argument]
g(a=2, b=3)
# TODO
# error: [invalid-assignment]
C.x = 2
d: D = D()
# TODO
# error: [call-non-callable]
d()
# TODO
# error: [division-by-zero]
1 / number
```

116
crates/ty_python_semantic/src/place.rs
View file

@ -755,49 +755,46 @@ fn place_from_bindings_impl<'db>(
requires_explicit_reexport: RequiresExplicitReExport,
) -> Place<'db> {
let predicates = bindings_with_constraints.predicates;
let visibility_constraints = bindings_with_constraints.visibility_constraints;
let reachability_constraints = bindings_with_constraints.reachability_constraints;
let mut bindings_with_constraints = bindings_with_constraints.peekable();
let is_non_exported = |binding: Definition<'db>| {
requires_explicit_reexport.is_yes() && !is_reexported(db, binding)
};
let unbound_visibility_constraint = match bindings_with_constraints.peek() {
let unbound_reachability_constraint = match bindings_with_constraints.peek() {
Some(BindingWithConstraints {
binding,
visibility_constraint,
reachability_constraint,
narrowing_constraint: _,
}) if binding.is_undefined_or(is_non_exported) => Some(*visibility_constraint),
}) if binding.is_undefined_or(is_non_exported) => Some(*reachability_constraint),
_ => None,
};
let mut deleted_visibility = Truthiness::AlwaysFalse;
let mut deleted_reachability = Truthiness::AlwaysFalse;
// Evaluate this lazily because we don't always need it (for example, if there are no visible
// bindings at all, we don't need it), and it can cause us to evaluate visibility constraint
// bindings at all, we don't need it), and it can cause us to evaluate reachability constraint
// expressions, which is extra work and can lead to cycles.
let unbound_visibility = || {
unbound_visibility_constraint
.map(|visibility_constraint| {
visibility_constraints.evaluate(db, predicates, visibility_constraint)
let unbound_reachability = || {
unbound_reachability_constraint.map(|reachability_constraint| {
reachability_constraints.evaluate(db, predicates, reachability_constraint)
})
.unwrap_or(Truthiness::AlwaysFalse)
};
let mut types = bindings_with_constraints.filter_map(
|BindingWithConstraints {
binding,
narrowing_constraint,
visibility_constraint,
reachability_constraint,
}| {
let binding =
match binding {
let binding = match binding {
DefinitionState::Defined(binding) => binding,
DefinitionState::Undefined => {
return None;
}
DefinitionState::Deleted => {
deleted_visibility = deleted_visibility.or(
visibility_constraints.evaluate(db, predicates, visibility_constraint)
deleted_reachability = deleted_reachability.or(
reachability_constraints.evaluate(db, predicates, reachability_constraint)
);
return None;
}
@ -807,13 +804,14 @@ fn place_from_bindings_impl<'db>(
return None;
}
let static_visibility =
visibility_constraints.evaluate(db, predicates, visibility_constraint);
let static_reachability =
reachability_constraints.evaluate(db, predicates, reachability_constraint);
if static_visibility.is_always_false() {
// We found a binding that we have statically determined to not be visible from
// the use of the place that we are investigating. There are three interesting
// cases to consider:
if static_reachability.is_always_false() {
// If the static reachability evaluates to false, the binding is either not reachable
// from the start of the scope, or there is no control flow path from that binding to
// the use of the place that we are investigating. There are three interesting cases
// to consider:
//
// ```py
// def f1():
@ -827,35 +825,37 @@ fn place_from_bindings_impl<'db>(
// use(y)
//
// def f3(flag: bool):
// z = 1
// if flag:
// z = 1
// else:
// z = 2
// return
// use(z)
// ```
//
// In the first case, there is a single binding for `x`, and due to the statically
// known `False` condition, it is not visible at the use of `x`. However, we *can*
// see/reach the start of the scope from `use(x)`. This means that `x` is unbound
// and we should return `None`.
// In the first case, there is a single binding for `x`, but it is not reachable from
// the start of the scope. However, the use of `x` is reachable (`unbound_reachability`
// is not always-false). This means that `x` is unbound and we should return `None`.
//
// In the second case, `y` is also not visible at the use of `y`, but here, we can
// not see/reach the start of the scope. There is only one path of control flow,
// and it passes through that binding of `y` (which we can not see). This implies
// that we are in an unreachable section of code. We return `Never` in order to
// silence the `unresolve-reference` diagnostic that would otherwise be emitted at
// the use of `y`.
// In the second case, the binding of `y` is reachable, but there is no control flow
// path from the beginning of the scope, through that binding, to the use of `y` that
// we are investigating. There is also no control flow path from the start of the
// scope, through the implicit `y = <unbound>` binding, to the use of `y`. This means
// that `unbound_reachability` is always false. Since there are no other bindings, no
// control flow path can reach this use of `y`, implying that we are in unreachable
// section of code. We return `Never` in order to silence the `unresolve-reference`
// diagnostic that would otherwise be emitted at the use of `y`.
//
// In the third case, we have two bindings for `z`. The first one is visible, so we
// consider the case that we now encounter the second binding `z = 2`, which is not
// visible due to the early return. We *also* can not see the start of the scope
// from `use(z)` because both paths of control flow pass through a binding of `z`.
// The `z = 1` binding is visible, and so we are *not* in an unreachable section of
// code. However, it is still okay to return `Never` in this case, because we will
// union the types of all bindings, and `Never` will be eliminated automatically.
// In the third case, we have two bindings for `z`. The first one is visible (there
// is a path of control flow from the start of the scope, through that binding, to
// the use of `z`). So we consider the case that we now encounter the second binding
// `z = 2`, which is not visible due to the early return. The `z = <unbound>` binding
// is not live (shadowed by the other bindings), so `unbound_reachability` is `None`.
// Here, we are *not* in an unreachable section of code. However, it is still okay to
// return `Never` in this case, because we will union the types of all bindings, and
// `Never` will be eliminated automatically.
if unbound_visibility().is_always_false() {
// The scope-start is not visible
if unbound_reachability().is_none_or(Truthiness::is_always_false) {
return Some(Type::Never);
}
return None;
@ -867,14 +867,14 @@ fn place_from_bindings_impl<'db>(
);
if let Some(first) = types.next() {
let boundness = match unbound_visibility() {
Truthiness::AlwaysTrue => {
let boundness = match unbound_reachability() {
Some(Truthiness::AlwaysTrue) => {
unreachable!(
"If we have at least one binding, the scope-start should not be definitely visible"
"If we have at least one binding, the implicit `unbound` binding should not be definitely visible"
)
}
Truthiness::AlwaysFalse => Boundness::Bound,
Truthiness::Ambiguous => Boundness::PossiblyUnbound,
Some(Truthiness::AlwaysFalse) | None => Boundness::Bound,
Some(Truthiness::Ambiguous) => Boundness::PossiblyUnbound,
};
let ty = if let Some(second) = types.next() {
@ -882,7 +882,7 @@ fn place_from_bindings_impl<'db>(
} else {
first
};
match deleted_visibility {
match deleted_reachability {
Truthiness::AlwaysFalse => Place::Type(ty, boundness),
Truthiness::AlwaysTrue => Place::Unbound,
Truthiness::Ambiguous => Place::Type(ty, Boundness::PossiblyUnbound),
@ -903,19 +903,19 @@ fn place_from_declarations_impl<'db>(
requires_explicit_reexport: RequiresExplicitReExport,
) -> PlaceFromDeclarationsResult<'db> {
let predicates = declarations.predicates;
let visibility_constraints = declarations.visibility_constraints;
let reachability_constraints = declarations.reachability_constraints;
let mut declarations = declarations.peekable();
let is_non_exported = |declaration: Definition<'db>| {
requires_explicit_reexport.is_yes() && !is_reexported(db, declaration)
};
let undeclared_visibility = match declarations.peek() {
let undeclared_reachability = match declarations.peek() {
Some(DeclarationWithConstraint {
declaration,
visibility_constraint,
reachability_constraint,
}) if declaration.is_undefined_or(is_non_exported) => {
visibility_constraints.evaluate(db, predicates, *visibility_constraint)
reachability_constraints.evaluate(db, predicates, *reachability_constraint)
}
_ => Truthiness::AlwaysFalse,
};
@ -923,7 +923,7 @@ fn place_from_declarations_impl<'db>(
let mut types = declarations.filter_map(
|DeclarationWithConstraint {
declaration,
visibility_constraint,
reachability_constraint,
}| {
let DefinitionState::Defined(declaration) = declaration else {
return None;
@ -933,10 +933,10 @@ fn place_from_declarations_impl<'db>(
return None;
}
let static_visibility =
visibility_constraints.evaluate(db, predicates, visibility_constraint);
let static_reachability =
reachability_constraints.evaluate(db, predicates, reachability_constraint);
if static_visibility.is_always_false() {
if static_reachability.is_always_false() {
None
} else {
Some(declaration_type(db, declaration))
@ -964,10 +964,10 @@ fn place_from_declarations_impl<'db>(
first
};
if conflicting.is_empty() {
let boundness = match undeclared_visibility {
let boundness = match undeclared_reachability {
Truthiness::AlwaysTrue => {
unreachable!(
"If we have at least one declaration, the scope-start should not be definitely visible"
"If we have at least one declaration, the implicit `unbound` binding should not be definitely visible"
)
}
Truthiness::AlwaysFalse => Boundness::Bound,

2
crates/ty_python_semantic/src/semantic_index.rs
View file

@ -33,8 +33,8 @@ pub(crate) mod narrowing_constraints;
pub mod place;
pub(crate) mod predicate;
mod re_exports;
mod reachability_constraints;
mod use_def;
mod visibility_constraints;
pub(crate) use self::use_def::{
BindingWithConstraints, BindingWithConstraintsIterator, DeclarationWithConstraint,

222
crates/ty_python_semantic/src/semantic_index/builder.rs
View file

@ -40,12 +40,12 @@ use crate::semantic_index::predicate::{
StarImportPlaceholderPredicate,
};
use crate::semantic_index::re_exports::exported_names;
use crate::semantic_index::reachability_constraints::{
ReachabilityConstraintsBuilder, ScopedReachabilityConstraintId,
};
use crate::semantic_index::use_def::{
EagerSnapshotKey, FlowSnapshot, ScopedEagerSnapshotId, UseDefMapBuilder,
};
use crate::semantic_index::visibility_constraints::{
ScopedVisibilityConstraintId, VisibilityConstraintsBuilder,
};
use crate::unpack::{Unpack, UnpackKind, UnpackPosition, UnpackValue};
use crate::{Db, Program};
@ -157,7 +157,7 @@ impl<'db, 'ast> SemanticIndexBuilder<'db, 'ast> {
builder.push_scope_with_parent(
NodeWithScopeRef::Module,
None,
ScopedVisibilityConstraintId::ALWAYS_TRUE,
ScopedReachabilityConstraintId::ALWAYS_TRUE,
);
builder
@ -237,7 +237,7 @@ impl<'db, 'ast> SemanticIndexBuilder<'db, 'ast> {
&mut self,
node: NodeWithScopeRef,
parent: Option<FileScopeId>,
reachability: ScopedVisibilityConstraintId,
reachability: ScopedReachabilityConstraintId,
) {
let children_start = self.scopes.next_index() + 1;
@ -354,9 +354,9 @@ impl<'db, 'ast> SemanticIndexBuilder<'db, 'ast> {
&self.use_def_maps[scope_id]
}
fn current_visibility_constraints_mut(&mut self) -> &mut VisibilityConstraintsBuilder {
fn current_reachability_constraints_mut(&mut self) -> &mut ReachabilityConstraintsBuilder {
let scope_id = self.current_scope();
&mut self.use_def_maps[scope_id].visibility_constraints
&mut self.use_def_maps[scope_id].reachability_constraints
}
fn current_ast_ids(&mut self) -> &mut AstIdsBuilder {
@ -576,55 +576,15 @@ impl<'db, 'ast> SemanticIndexBuilder<'db, 'ast> {
id
}
/// Records a previously added visibility constraint by applying it to all live bindings
/// and declarations.
fn record_visibility_constraint_id(&mut self, constraint: ScopedVisibilityConstraintId) {
self.current_use_def_map_mut()
.record_visibility_constraint(constraint);
}
/// Negates the given visibility constraint and then adds it to all live bindings and declarations.
fn record_negated_visibility_constraint(
&mut self,
constraint: ScopedVisibilityConstraintId,
) -> ScopedVisibilityConstraintId {
let id = self
.current_visibility_constraints_mut()
.add_not_constraint(constraint);
self.record_visibility_constraint_id(id);
id
}
/// Records a visibility constraint by applying it to all live bindings and declarations.
fn record_visibility_constraint(
&mut self,
predicate: Predicate<'db>,
) -> ScopedVisibilityConstraintId {
let predicate_id = self.current_use_def_map_mut().add_predicate(predicate);
let id = self
.current_visibility_constraints_mut()
.add_atom(predicate_id);
self.record_visibility_constraint_id(id);
id
}
/// Records that all remaining statements in the current block are unreachable, and therefore
/// not visible.
/// Records that all remaining statements in the current block are unreachable.
fn mark_unreachable(&mut self) {
self.current_use_def_map_mut().mark_unreachable();
}
/// Records a visibility constraint that always evaluates to "ambiguous".
fn record_ambiguous_visibility(&mut self) {
/// Records a reachability constraint that always evaluates to "ambiguous".
fn record_ambiguous_reachability(&mut self) {
self.current_use_def_map_mut()
.record_visibility_constraint(ScopedVisibilityConstraintId::AMBIGUOUS);
}
/// Simplifies (resets) visibility constraints on all live bindings and declarations that did
/// not see any new definitions since the given snapshot.
fn simplify_visibility_constraints(&mut self, snapshot: FlowSnapshot) {
self.current_use_def_map_mut()
.simplify_visibility_constraints(snapshot);
.record_reachability_constraint(ScopedReachabilityConstraintId::AMBIGUOUS);
}
/// Record a constraint that affects the reachability of the current position in the semantic
@ -634,7 +594,7 @@ impl<'db, 'ast> SemanticIndexBuilder<'db, 'ast> {
fn record_reachability_constraint(
&mut self,
predicate: Predicate<'db>,
) -> ScopedVisibilityConstraintId {
) -> ScopedReachabilityConstraintId {
let predicate_id = self.add_predicate(predicate);
self.record_reachability_constraint_id(predicate_id)
}
@ -643,22 +603,22 @@ impl<'db, 'ast> SemanticIndexBuilder<'db, 'ast> {
fn record_reachability_constraint_id(
&mut self,
predicate_id: ScopedPredicateId,
) -> ScopedVisibilityConstraintId {
let visibility_constraint = self
.current_visibility_constraints_mut()
) -> ScopedReachabilityConstraintId {
let reachability_constraint = self
.current_reachability_constraints_mut()
.add_atom(predicate_id);
self.current_use_def_map_mut()
.record_reachability_constraint(visibility_constraint);
visibility_constraint
.record_reachability_constraint(reachability_constraint);
reachability_constraint
}
/// Record the negation of a given reachability/visibility constraint.
/// Record the negation of a given reachability constraint.
fn record_negated_reachability_constraint(
&mut self,
reachability_constraint: ScopedVisibilityConstraintId,
reachability_constraint: ScopedReachabilityConstraintId,
) {
let negated_constraint = self
.current_visibility_constraints_mut()
.current_reachability_constraints_mut()
.add_not_constraint(reachability_constraint);
self.current_use_def_map_mut()
.record_reachability_constraint(negated_constraint);
@ -1139,12 +1099,10 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
let pre_return_state = matches!(last_stmt, ast::Stmt::Return(_))
.then(|| builder.flow_snapshot());
builder.visit_stmt(last_stmt);
let scope_start_visibility =
builder.current_use_def_map().scope_start_visibility;
let reachability = builder.current_use_def_map().reachability;
if let Some(pre_return_state) = pre_return_state {
builder.flow_restore(pre_return_state);
builder.current_use_def_map_mut().scope_start_visibility =
scope_start_visibility;
builder.current_use_def_map_mut().reachability = reachability;
}
}
@ -1297,11 +1255,11 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
continue;
};
// In order to understand the visibility of definitions created by a `*` import,
// we need to know the visibility of the global-scope definitions in the
// In order to understand the reachability of definitions created by a `*` import,
// we need to know the reachability of the global-scope definitions in the
// `referenced_module` the symbols imported from. Much like predicates for `if`
// statements can only have their visibility constraints resolved at type-inference
// time, the visibility of these global-scope definitions in the external module
// statements can only have their reachability constraints resolved at type-inference
// time, the reachability of these global-scope definitions in the external module
// cannot be resolved at this point. As such, we essentially model each definition
// stemming from a `from exporter *` import as something like:
//
@ -1328,7 +1286,7 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
self.current_use_def_map().single_place_snapshot(symbol_id);
self.push_additional_definition(symbol_id, node_ref);
self.current_use_def_map_mut()
.record_and_negate_star_import_visibility_constraint(
.record_and_negate_star_import_reachability_constraint(
star_import,
symbol_id,
pre_definition,
@ -1387,7 +1345,7 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
// reachability constraint on the `msg` expression.
//
// The other important part is the `<halt>`. This lets us skip the usual merging of
// flow states and simplification of visibility constraints, since there is no way
// flow states and simplification of reachability constraints, since there is no way
// of getting out of that `msg` branch. We simply restore to the post-test state.
self.visit_expr(test);
@ -1399,12 +1357,10 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
self.record_narrowing_constraint(negated_predicate);
self.record_reachability_constraint(negated_predicate);
self.visit_expr(msg);
self.record_visibility_constraint(negated_predicate);
self.flow_restore(post_test);
}
self.record_narrowing_constraint(predicate);
self.record_visibility_constraint(predicate);
self.record_reachability_constraint(predicate);
}
@ -1479,13 +1435,10 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
self.visit_expr(&node.test);
let mut no_branch_taken = self.flow_snapshot();
let mut last_predicate = self.record_expression_narrowing_constraint(&node.test);
let mut reachability_constraint =
let mut last_reachability_constraint =
self.record_reachability_constraint(last_predicate);
self.visit_body(&node.body);
let visibility_constraint_id = self.record_visibility_constraint(last_predicate);
let mut vis_constraints = vec![visibility_constraint_id];
let mut post_clauses: Vec<FlowSnapshot> = vec![];
let elif_else_clauses = node
.elif_else_clauses
@ -1509,38 +1462,27 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
// we can only take an elif/else branch if none of the previous ones were
// taken
self.flow_restore(no_branch_taken.clone());
self.record_negated_narrowing_constraint(last_predicate);
self.record_negated_reachability_constraint(reachability_constraint);
let elif_predicate = if let Some(elif_test) = clause_test {
self.record_negated_narrowing_constraint(last_predicate);
self.record_negated_reachability_constraint(last_reachability_constraint);
if let Some(elif_test) = clause_test {
self.visit_expr(elif_test);
// A test expression is evaluated whether the branch is taken or not
no_branch_taken = self.flow_snapshot();
reachability_constraint =
last_predicate = self.record_expression_narrowing_constraint(elif_test);
last_reachability_constraint =
self.record_reachability_constraint(last_predicate);
let predicate = self.record_expression_narrowing_constraint(elif_test);
Some(predicate)
} else {
None
};
}
self.visit_body(clause_body);
for id in &vis_constraints {
self.record_negated_visibility_constraint(*id);
}
if let Some(elif_predicate) = elif_predicate {
last_predicate = elif_predicate;
let id = self.record_visibility_constraint(elif_predicate);
vis_constraints.push(id);
}
}
for post_clause_state in post_clauses {
self.flow_merge(post_clause_state);
}
self.simplify_visibility_constraints(no_branch_taken);
}
ast::Stmt::While(ast::StmtWhile {
test,
@ -1555,57 +1497,49 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
let predicate = self.record_expression_narrowing_constraint(test);
self.record_reachability_constraint(predicate);
// We need multiple copies of the visibility constraint for the while condition,
// We need multiple copies of the reachability constraint for the while condition,
// since we need to model situations where the first evaluation of the condition
// returns True, but a later evaluation returns False.
let first_predicate_id = self.current_use_def_map_mut().add_predicate(predicate);
let later_predicate_id = self.current_use_def_map_mut().add_predicate(predicate);
let first_vis_constraint_id = self
.current_visibility_constraints_mut()
.add_atom(first_predicate_id);
let later_vis_constraint_id = self
.current_visibility_constraints_mut()
.current_reachability_constraints_mut()
.add_atom(later_predicate_id);
// If the body is executed, we know that we've evaluated the condition at least
// once, and that the first evaluation was True. We might not have evaluated the
// condition more than once, so we can't assume that later evaluations were True.
// So the body's full reachability constraint is `first`.
self.record_reachability_constraint_id(first_predicate_id);
let outer_loop = self.push_loop();
self.visit_body(body);
let this_loop = self.pop_loop(outer_loop);
// If the body is executed, we know that we've evaluated the condition at least
// once, and that the first evaluation was True. We might not have evaluated the
// condition more than once, so we can't assume that later evaluations were True.
// So the body's full visibility constraint is `first`.
let body_vis_constraint_id = first_vis_constraint_id;
self.record_visibility_constraint_id(body_vis_constraint_id);
// We execute the `else` once the condition evaluates to false. This could happen
// without ever executing the body, if the condition is false the first time it's
// tested. So the starting flow state of the `else` clause is the union of:
// - the pre-loop state with a visibility constraint that the first evaluation of
// - the pre-loop state with a reachability constraint that the first evaluation of
// the while condition was false,
// - the post-body state (which already has a visibility constraint that the
// first evaluation was true) with a visibility constraint that a _later_
// - the post-body state (which already has a reachability constraint that the
// first evaluation was true) with a reachability constraint that a _later_
// evaluation of the while condition was false.
// To model this correctly, we need two copies of the while condition constraint,
// since the first and later evaluations might produce different results.
let post_body = self.flow_snapshot();
self.flow_restore(pre_loop.clone());
self.record_negated_visibility_constraint(first_vis_constraint_id);
self.flow_restore(pre_loop);
self.flow_merge(post_body);
self.record_negated_narrowing_constraint(predicate);
self.record_negated_reachability_constraint(later_vis_constraint_id);
self.visit_body(orelse);
self.record_negated_visibility_constraint(later_vis_constraint_id);
// Breaking out of a while loop bypasses the `else` clause, so merge in the break
// states after visiting `else`.
for break_state in this_loop.break_states {
let snapshot = self.flow_snapshot();
self.flow_restore(break_state);
self.record_visibility_constraint_id(body_vis_constraint_id);
self.flow_merge(snapshot);
}
self.simplify_visibility_constraints(pre_loop);
}
ast::Stmt::With(ast::StmtWith {
items,
@ -1652,7 +1586,7 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
let iter_expr = self.add_standalone_expression(iter);
self.visit_expr(iter);
self.record_ambiguous_visibility();
self.record_ambiguous_reachability();
let pre_loop = self.flow_snapshot();
@ -1709,10 +1643,15 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
&case.pattern,
case.guard.as_deref(),
);
let vis_constraint_id = self.record_reachability_constraint(match_predicate);
let reachability_constraint =
self.record_reachability_constraint(match_predicate);
let match_success_guard_failure = case.guard.as_ref().map(|guard| {
let guard_expr = self.add_standalone_expression(guard);
// We could also add the guard expression as a reachability constraint, but
// it seems unlikely that both the case predicate as well as the guard are
// statically known conditions, so we currently don't model that.
self.record_ambiguous_reachability();
self.visit_expr(guard);
let post_guard_eval = self.flow_snapshot();
let predicate = Predicate {
@ -1726,8 +1665,6 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
match_success_guard_failure
});
self.record_visibility_constraint_id(vis_constraint_id);
self.visit_body(&case.body);
post_case_snapshots.push(self.flow_snapshot());
@ -1738,6 +1675,7 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
// snapshots into.
self.flow_restore(no_case_matched.clone());
self.record_negated_narrowing_constraint(match_predicate);
self.record_negated_reachability_constraint(reachability_constraint);
if let Some(match_success_guard_failure) = match_success_guard_failure {
self.flow_merge(match_success_guard_failure);
} else {
@ -1748,15 +1686,12 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
debug_assert!(case.guard.is_none());
}
self.record_negated_visibility_constraint(vis_constraint_id);
no_case_matched = self.flow_snapshot();
}
for post_clause_state in post_case_snapshots {
self.flow_merge(post_clause_state);
}
self.simplify_visibility_constraints(no_case_matched);
}
ast::Stmt::Try(ast::StmtTry {
body,
@ -1767,7 +1702,7 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
range: _,
node_index: _,
}) => {
self.record_ambiguous_visibility();
self.record_ambiguous_reachability();
// Save the state prior to visiting any of the `try` block.
//
@ -2136,16 +2071,13 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
let predicate = self.record_expression_narrowing_constraint(test);
let reachability_constraint = self.record_reachability_constraint(predicate);
self.visit_expr(body);
let visibility_constraint = self.record_visibility_constraint(predicate);
let post_body = self.flow_snapshot();
self.flow_restore(pre_if.clone());
self.flow_restore(pre_if);
self.record_negated_narrowing_constraint(predicate);
self.record_negated_reachability_constraint(reachability_constraint);
self.visit_expr(orelse);
self.record_negated_visibility_constraint(visibility_constraint);
self.flow_merge(post_body);
self.simplify_visibility_constraints(pre_if);
}
ast::Expr::ListComp(
list_comprehension @ ast::ExprListComp {
@ -2203,18 +2135,17 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
node_index: _,
op,
}) => {
let pre_op = self.flow_snapshot();
let mut snapshots = vec![];
let mut visibility_constraints = vec![];
let mut reachability_constraints = vec![];
for (index, value) in values.iter().enumerate() {
self.visit_expr(value);
for vid in &visibility_constraints {
self.record_visibility_constraint_id(*vid);
for id in &reachability_constraints {
self.current_use_def_map_mut()
.record_reachability_constraint(*id); // TODO: nicer API
}
self.visit_expr(value);
// For the last value, we don't need to model control flow. There is no short-circuiting
// anymore.
if index < values.len() - 1 {
@ -2223,37 +2154,32 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
ast::BoolOp::And => self.add_predicate(predicate),
ast::BoolOp::Or => self.add_negated_predicate(predicate),
};
let visibility_constraint = self
.current_visibility_constraints_mut()
let reachability_constraint = self
.current_reachability_constraints_mut()
.add_atom(predicate_id);
let after_expr = self.flow_snapshot();
// We first model the short-circuiting behavior. We take the short-circuit
// path here if all of the previous short-circuit paths were not taken, so
// we record all previously existing visibility constraints, and negate the
// we record all previously existing reachability constraints, and negate the
// one for the current expression.
for vid in &visibility_constraints {
self.record_visibility_constraint_id(*vid);
}
self.record_negated_visibility_constraint(visibility_constraint);
self.record_negated_reachability_constraint(reachability_constraint);
snapshots.push(self.flow_snapshot());
// Then we model the non-short-circuiting behavior. Here, we need to delay
// the application of the visibility constraint until after the expression
// the application of the reachability constraint until after the expression
// has been evaluated, so we only push it onto the stack here.
self.flow_restore(after_expr);
self.record_narrowing_constraint_id(predicate_id);
self.record_reachability_constraint_id(predicate_id);
visibility_constraints.push(visibility_constraint);
reachability_constraints.push(reachability_constraint);
}
}
for snapshot in snapshots {
self.flow_merge(snapshot);
}
self.simplify_visibility_constraints(pre_op);
}
ast::Expr::StringLiteral(_) => {
// Track reachability of string literals, as they could be a stringified annotation

8
crates/ty_python_semantic/src/semantic_index/place.rs
View file

@ -15,7 +15,7 @@ use smallvec::{SmallVec, smallvec};
use crate::Db;
use crate::ast_node_ref::AstNodeRef;
use crate::node_key::NodeKey;
use crate::semantic_index::visibility_constraints::ScopedVisibilityConstraintId;
use crate::semantic_index::reachability_constraints::ScopedReachabilityConstraintId;
use crate::semantic_index::{PlaceSet, SemanticIndex, semantic_index};
#[derive(Debug, Clone, PartialEq, Eq, Hash, salsa::Update)]
@ -480,7 +480,7 @@ pub struct Scope {
parent: Option<FileScopeId>,
node: NodeWithScopeKind,
descendants: Range<FileScopeId>,
reachability: ScopedVisibilityConstraintId,
reachability: ScopedReachabilityConstraintId,
}
impl Scope {
@ -488,7 +488,7 @@ impl Scope {
parent: Option<FileScopeId>,
node: NodeWithScopeKind,
descendants: Range<FileScopeId>,
reachability: ScopedVisibilityConstraintId,
reachability: ScopedReachabilityConstraintId,
) -> Self {
Scope {
parent,
@ -522,7 +522,7 @@ impl Scope {
self.kind().is_eager()
}
pub(crate) fn reachability(&self) -> ScopedVisibilityConstraintId {
pub(crate) fn reachability(&self) -> ScopedReachabilityConstraintId {
self.reachability
}
}

8
crates/ty_python_semantic/src/semantic_index/predicate.rs
View file

@ -4,8 +4,8 @@
//!
//! - [_Narrowing constraints_][crate::semantic_index::narrowing_constraints] constrain the type of
//! a binding that is visible at a particular use.
//! - [_Visibility constraints_][crate::semantic_index::visibility_constraints] determine the
//! static visibility of a binding, and the reachability of a statement.
//! - [_Reachability constraints_][crate::semantic_index::reachability_constraints] determine the
//! static reachability of a binding, and the reachability of a statement or expression.
use ruff_db::files::File;
use ruff_index::{IndexVec, newtype_index};
@ -65,7 +65,7 @@ pub(crate) enum PredicateNode<'db> {
StarImportPlaceholder(StarImportPlaceholderPredicate<'db>),
}
/// Pattern kinds for which we support type narrowing and/or static visibility analysis.
/// Pattern kinds for which we support type narrowing and/or static reachability analysis.
#[derive(Debug, Clone, Hash, PartialEq, salsa::Update)]
pub(crate) enum PatternPredicateKind<'db> {
Singleton(Singleton),
@ -99,7 +99,7 @@ impl<'db> PatternPredicate<'db> {
/// A "placeholder predicate" that is used to model the fact that the boundness of a
/// (possible) definition or declaration caused by a `*` import cannot be fully determined
/// until type-inference time. This is essentially the same as a standard visibility constraint,
/// until type-inference time. This is essentially the same as a standard reachability constraint,
/// so we reuse the [`Predicate`] infrastructure to model it.
///
/// To illustrate, say we have a module `exporter.py` like so:

332
crates/ty_python_semantic/src/semantic_index/visibility_constraints.rs → crates/ty_python_semantic/src/semantic_index/reachability_constraints.rs
View file

@ -1,61 +1,37 @@
//! # Visibility constraints
//! # Reachability constraints
//!
//! During semantic index building, we collect visibility constraints for each binding and
//! declaration. These constraints are then used during type-checking to determine the static
//! visibility of a certain definition. This allows us to re-analyze control flow during type
//! checking, potentially "hiding" some branches that we can statically determine to never be
//! taken. Consider the following example first. We added implicit "unbound" definitions at the
//! start of the scope. Note how visibility constraints can apply to bindings outside of the
//! if-statement:
//! During semantic index building, we record so-called reachability constraints that keep track
//! of a set of conditions that need to apply in order for a certain statement or expression to
//! be reachable from the start of the scope. As an example, consider the following situation where
//! we have just processed two `if`-statements:
//! ```py
//! x = <unbound> # not a live binding for the use of x below, shadowed by `x = 1`
//! y = <unbound> # visibility constraint: ~test
//!
//! x = 1 # visibility constraint: ~test
//! if test:
//! x = 2 # visibility constraint: test
//!
//! y = 2 # visibility constraint: test
//!
//! use(x)
//! use(y)
//! <is this reachable?>
//! ```
//! The static truthiness of the `test` condition can either be always-false, ambiguous, or
//! always-true. Similarly, we have the same three options when evaluating a visibility constraint.
//! This outcome determines the visibility of a definition: always-true means that the definition
//! is definitely visible for a given use, always-false means that the definition is definitely
//! not visible, and ambiguous means that we might see this definition or not. In the latter case,
//! we need to consider both options during type inference and boundness analysis. For the example
//! above, these are the possible type inference / boundness results for the uses of `x` and `y`:
//! In this case, we would record a reachability constraint of `test`, which would later allow us
//! to re-analyze the control flow during type-checking, once we actually know the static truthiness
//! of `test`. When evaluating a constraint, there are three possible outcomes: always true, always
//! false, or ambiguous. For a simple constraint like this, always-true and always-false correspond
//! to the case in which we can infer that the type of `test` is `Literal[True]` or `Literal[False]`.
//! In any other case, like if the type of `test` is `bool` or `Unknown`, we can not statically
//! determine whether `test` is truthy or falsy, so the outcome would be "ambiguous".
//!
//! ```text
//! | `test` truthiness | `~test` truthiness | type of `x` | boundness of `y` |
//! |-------------------|--------------------|-----------------|------------------|
//! | always false | always true | `Literal[1]` | unbound |
//! | ambiguous | ambiguous | `Literal[1, 2]` | possibly unbound |
//! | always true | always false | `Literal[2]` | bound |
//! ```
//!
//! ### Sequential constraints (ternary AND)
//! ## Sequential constraints (ternary AND)
//!
//! As we have seen above, visibility constraints can apply outside of a control flow element.
//! So we need to consider the possibility that multiple constraints apply to the same binding.
//! Here, we consider what happens if multiple `if`-statements lead to a sequence of constraints.
//! Consider the following example:
//! Whenever control flow branches, we record reachability constraints. If we already have a
//! constraint, we create a new one using a ternary AND operation. Consider the following example:
//! ```py
//! x = 0
//!
//! if test1:
//! x = 1
//!
//! if test2:
//! x = 2
//! <is this reachable?>
//! ```
//! The binding `x = 2` is easy to analyze. Its visibility corresponds to the truthiness of `test2`.
//! For the `x = 1` binding, things are a bit more interesting. It is always visible if `test1` is
//! always-true *and* `test2` is always-false. It is never visible if `test1` is always-false *or*
//! `test2` is always-true. And it is ambiguous otherwise. This corresponds to a ternary *test1 AND
//! ~test2* operation in three-valued Kleene logic [Kleene]:
//! Here, we would accumulate a reachability constraint of `test1 AND test2`. We can statically
//! determine that this position is *always* reachable only if both `test1` and `test2` are
//! always true. On the other hand, we can statically determine that this position is *never*
//! reachable if *either* `test1` or `test2` is always false. In any other case, we can not
//! determine whether this position is reachable or not, so the outcome is "ambiguous". This
//! corresponds to a ternary *AND* operation in [Kleene] logic:
//!
//! ```text
//! | AND | always-false | ambiguous | always-true |
@ -65,38 +41,27 @@
//! | always true | always-false | ambiguous | always-true |
//! ```
//!
//! The `x = 0` binding can be handled similarly, with the difference that both `test1` and `test2`
//! are negated:
//!
//! ## Merged constraints (ternary OR)
//!
//! We also need to consider the case where control flow merges again. Consider a case like this:
//! ```py
//! x = 0 # ~test1 AND ~test2
//!
//! def _():
//! if test1:
//! x = 1 # test1 AND ~test2
//!
//! if test2:
//! x = 2 # test2
//! ```
//!
//! ### Merged constraints (ternary OR)
//!
//! Finally, we consider what happens in "parallel" control flow. Consider the following example
//! where we have omitted the test condition for the outer `if` for clarity:
//! ```py
//! x = 0
//!
//! if <…>:
//! if test1:
//! x = 1
//! pass
//! elif test2:
//! pass
//! else:
//! if test2:
//! x = 2
//! return
//!
//! use(x)
//! <is this reachable?>
//! ```
//! At the usage of `x`, i.e. after control flow has been merged again, the visibility of the `x =
//! 0` binding behaves as follows: the binding is always visible if `test1` is always-false *or*
//! `test2` is always-false; and it is never visible if `test1` is always-true *and* `test2` is
//! always-true. This corresponds to a ternary *OR* operation in Kleene logic:
//! Here, the first branch has a `test1` constraint, and the second branch has a `test2` constraint.
//! The third branch ends in a terminal statement [^1]. When we merge control flow, we need to consider
//! the reachability through either the first or the second branch. The current position is only
//! *definitely* unreachable if both `test1` and `test2` are always false. It is definitely
//! reachable if *either* `test1` or `test2` is always true. In any other case, we can not statically
//! determine whether it is reachable or not. This operation corresponds to a ternary *OR* operation:
//!
//! ```text
//! | OR | always-false | ambiguous | always-true |
@ -106,40 +71,95 @@
//! | always true | always-true | always-true | always-true |
//! ```
//!
//! Using this, we can annotate the visibility constraints for the example above:
//! ```py
//! x = 0 # ~test1 OR ~test2
//! [^1]: What's actually happening here is that we merge all three branches using a ternary OR. The
//! third branch has a reachability constraint of `always-false`, and `t OR always-false` is equal
//! to `t` (see first column in that table), so it was okay to omit the third branch in the discussion
//! above.
//!
//! if <…>:
//! if test1:
//! x = 1 # test1
//!
//! ## Negation
//!
//! Control flow elements like `if-elif-else` or `match` statements can also lead to negated
//! constraints. For example, we record a constraint of `~test` for the `else` branch here:
//! ```py
//! if test:
//! pass
//! else:
//! if test2:
//! x = 2 # test2
//! <is this reachable?>
//! ```
//!
//! ## Explicit ambiguity
//!
//! In some cases, we explicitly record an “ambiguous” constraint. We do this when branching on
//! something that we can not (or intentionally do not want to) analyze statically. `for` loops are
//! one example:
//! ```py
//! def _():
//! for _ in range(2):
//! return
//!
//! <is this reachable?>
//! ```
//! If we would not record any constraints at the branching point, we would have an `always-true`
//! reachability for the no-loop branch, and a `always-false` reachability for the branch which enters
//! the loop. Merging those would lead to a reachability of `always-true OR always-false = always-true`,
//! i.e. we would consider the end of the scope to be unconditionally reachable, which is not correct.
//!
//! Recording an ambiguous constraint at the branching point modifies the constraints in both branches to
//! `always-true AND ambiguous = ambiguous` and `always-false AND ambiguous = always-false`, respectively.
//! Merging these two using OR correctly leads to `ambiguous` for the end-of-scope reachability.
//!
//!
//! ## Reachability constraints and bindings
//!
//! To understand how reachability constraints apply to bindings in particular, consider the following
//! example:
//! ```py
//! x = <unbound> # not a live binding for the use of x below, shadowed by `x = 1`
//! y = <unbound> # reachability constraint: ~test
//!
//! x = 1 # reachability constraint: ~test
//! if test:
//! x = 2 # reachability constraint: test
//!
//! y = 2 # reachability constraint: test
//!
//! use(x)
//! use(y)
//! ```
//! Both the type and the boundness of `x` and `y` are affected by reachability constraints:
//!
//! ```text
//! | `test` truthiness | type of `x` | boundness of `y` |
//! |-------------------|-----------------|------------------|
//! | always false | `Literal[1]` | unbound |
//! | ambiguous | `Literal[1, 2]` | possibly unbound |
//! | always true | `Literal[2]` | bound |
//! ```
//!
//! ### Explicit ambiguity
//! To achieve this, we apply reachability constraints retroactively to bindings that came before
//! the branching point. In the example above, the `x = 1` binding has a `test` constraint in the
//! `if` branch, and a `~test` constraint in the implicit `else` branch. Since it is shadowed by
//! `x = 2` in the `if` branch, we are only left with the `~test` constraint after control flow
//! has merged again.
//!
//! In some cases, we explicitly add an “ambiguous” constraint to all bindings
//! in a certain control flow path. We do this when branching on something that we can not (or
//! intentionally do not want to) analyze statically. `for` loops are one example:
//! ```py
//! x = <unbound>
//! For live bindings, the reachability constraint therefore refers to the following question:
//! Is the binding reachable from the start of the scope, and is there a control flow path from
//! that binding to a use of that symbol at the current position?
//!
//! for _ in range(2):
//! x = 1
//! ```
//! Here, we report an ambiguous visibility constraint before branching off. If we don't do this,
//! the `x = <unbound>` binding would be considered unconditionally visible in the no-loop case.
//! And since the other branch does not have the live `x = <unbound>` binding, we would incorrectly
//! create a state where the `x = <unbound>` binding is always visible.
//! In the example above, `x = 1` is always reachable, but that binding can only reach the use of
//! `x` at the current position if `test` is falsy.
//!
//! To handle boundness correctly, we also add implicit `y = <unbound>` bindings at the start of
//! the scope. This allows us to determine whether a symbol is definitely bound (if that implicit
//! `y = <unbound>` binding is not visible), possibly unbound (if the reachability constraint
//! evaluates to `Ambiguous`), or definitely unbound (in case the `y = <unbound>` binding is
//! always visible).
//!
//!
//! ### Representing formulas
//!
//! Given everything above, we can represent a visibility constraint as a _ternary formula_. This
//! Given everything above, we can represent a reachability constraint as a _ternary formula_. This
//! is like a boolean formula (which maps several true/false variables to a single true/false
//! result), but which allows the third "ambiguous" value in addition to "true" and "false".
//!
@ -166,7 +186,7 @@
//! formulas (which have the same outputs for every combination of inputs) are represented by
//! exactly the same graph node. (Because of interning, this is not _equal_ nodes, but _identical_
//! ones.) That means that we can compare formulas for equivalence in constant time, and in
//! particular, can check whether a visibility constraint is statically always true or false,
//! particular, can check whether a reachability constraint is statically always true or false,
//! regardless of any Python program state, by seeing if the constraint's formula is the "true" or
//! "false" leaf node.
//!
@ -204,18 +224,18 @@ use crate::types::{Truthiness, Type, infer_expression_type};
/// for a particular [`Predicate`], if your formula needs to consider how a particular runtime
/// property might be different at different points in the execution of the program.
///
/// Visibility constraints are normalized, so equivalent constraints are guaranteed to have equal
/// reachability constraints are normalized, so equivalent constraints are guaranteed to have equal
/// IDs.
#[derive(Clone, Copy, Eq, Hash, PartialEq)]
pub(crate) struct ScopedVisibilityConstraintId(u32);
pub(crate) struct ScopedReachabilityConstraintId(u32);
impl std::fmt::Debug for ScopedVisibilityConstraintId {
impl std::fmt::Debug for ScopedReachabilityConstraintId {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
let mut f = f.debug_tuple("ScopedVisibilityConstraintId");
let mut f = f.debug_tuple("ScopedReachabilityConstraintId");
match *self {
// We use format_args instead of rendering the strings directly so that we don't get
// any quotes in the output: ScopedVisibilityConstraintId(AlwaysTrue) instead of
// ScopedVisibilityConstraintId("AlwaysTrue").
// any quotes in the output: ScopedReachabilityConstraintId(AlwaysTrue) instead of
// ScopedReachabilityConstraintId("AlwaysTrue").
ALWAYS_TRUE => f.field(&format_args!("AlwaysTrue")),
AMBIGUOUS => f.field(&format_args!("Ambiguous")),
ALWAYS_FALSE => f.field(&format_args!("AlwaysFalse")),
@ -237,34 +257,34 @@ impl std::fmt::Debug for ScopedVisibilityConstraintId {
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
struct InteriorNode {
/// A "variable" that is evaluated as part of a TDD ternary function. For visibility
/// A "variable" that is evaluated as part of a TDD ternary function. For reachability
/// constraints, this is a `Predicate` that represents some runtime property of the Python
/// code that we are evaluating.
atom: ScopedPredicateId,
if_true: ScopedVisibilityConstraintId,
if_ambiguous: ScopedVisibilityConstraintId,
if_false: ScopedVisibilityConstraintId,
if_true: ScopedReachabilityConstraintId,
if_ambiguous: ScopedReachabilityConstraintId,
if_false: ScopedReachabilityConstraintId,
}
impl ScopedVisibilityConstraintId {
impl ScopedReachabilityConstraintId {
/// A special ID that is used for an "always true" / "always visible" constraint.
pub(crate) const ALWAYS_TRUE: ScopedVisibilityConstraintId =
ScopedVisibilityConstraintId(0xffff_ffff);
pub(crate) const ALWAYS_TRUE: ScopedReachabilityConstraintId =
ScopedReachabilityConstraintId(0xffff_ffff);
/// A special ID that is used for an ambiguous constraint.
pub(crate) const AMBIGUOUS: ScopedVisibilityConstraintId =
ScopedVisibilityConstraintId(0xffff_fffe);
pub(crate) const AMBIGUOUS: ScopedReachabilityConstraintId =
ScopedReachabilityConstraintId(0xffff_fffe);
/// A special ID that is used for an "always false" / "never visible" constraint.
pub(crate) const ALWAYS_FALSE: ScopedVisibilityConstraintId =
ScopedVisibilityConstraintId(0xffff_fffd);
pub(crate) const ALWAYS_FALSE: ScopedReachabilityConstraintId =
ScopedReachabilityConstraintId(0xffff_fffd);
fn is_terminal(self) -> bool {
self.0 >= SMALLEST_TERMINAL.0
}
}
impl Idx for ScopedVisibilityConstraintId {
impl Idx for ScopedReachabilityConstraintId {
#[inline]
fn new(value: usize) -> Self {
assert!(value <= (SMALLEST_TERMINAL.0 as usize));
@ -280,35 +300,41 @@ impl Idx for ScopedVisibilityConstraintId {
}
// Rebind some constants locally so that we don't need as many qualifiers below.
const ALWAYS_TRUE: ScopedVisibilityConstraintId = ScopedVisibilityConstraintId::ALWAYS_TRUE;
const AMBIGUOUS: ScopedVisibilityConstraintId = ScopedVisibilityConstraintId::AMBIGUOUS;
const ALWAYS_FALSE: ScopedVisibilityConstraintId = ScopedVisibilityConstraintId::ALWAYS_FALSE;
const SMALLEST_TERMINAL: ScopedVisibilityConstraintId = ALWAYS_FALSE;
const ALWAYS_TRUE: ScopedReachabilityConstraintId = ScopedReachabilityConstraintId::ALWAYS_TRUE;
const AMBIGUOUS: ScopedReachabilityConstraintId = ScopedReachabilityConstraintId::AMBIGUOUS;
const ALWAYS_FALSE: ScopedReachabilityConstraintId = ScopedReachabilityConstraintId::ALWAYS_FALSE;
const SMALLEST_TERMINAL: ScopedReachabilityConstraintId = ALWAYS_FALSE;
/// A collection of visibility constraints for a given scope.
/// A collection of reachability constraints for a given scope.
#[derive(Debug, PartialEq, Eq, salsa::Update)]
pub(crate) struct VisibilityConstraints {
interiors: IndexVec<ScopedVisibilityConstraintId, InteriorNode>,
pub(crate) struct ReachabilityConstraints {
interiors: IndexVec<ScopedReachabilityConstraintId, InteriorNode>,
}
#[derive(Debug, Default, PartialEq, Eq)]
pub(crate) struct VisibilityConstraintsBuilder {
interiors: IndexVec<ScopedVisibilityConstraintId, InteriorNode>,
interior_cache: FxHashMap<InteriorNode, ScopedVisibilityConstraintId>,
not_cache: FxHashMap<ScopedVisibilityConstraintId, ScopedVisibilityConstraintId>,
pub(crate) struct ReachabilityConstraintsBuilder {
interiors: IndexVec<ScopedReachabilityConstraintId, InteriorNode>,
interior_cache: FxHashMap<InteriorNode, ScopedReachabilityConstraintId>,
not_cache: FxHashMap<ScopedReachabilityConstraintId, ScopedReachabilityConstraintId>,
and_cache: FxHashMap<
(ScopedVisibilityConstraintId, ScopedVisibilityConstraintId),
ScopedVisibilityConstraintId,
(
ScopedReachabilityConstraintId,
ScopedReachabilityConstraintId,
),
ScopedReachabilityConstraintId,
>,
or_cache: FxHashMap<
(ScopedVisibilityConstraintId, ScopedVisibilityConstraintId),
ScopedVisibilityConstraintId,
(
ScopedReachabilityConstraintId,
ScopedReachabilityConstraintId,
),
ScopedReachabilityConstraintId,
>,
}
impl VisibilityConstraintsBuilder {
pub(crate) fn build(self) -> VisibilityConstraints {
VisibilityConstraints {
impl ReachabilityConstraintsBuilder {
pub(crate) fn build(self) -> ReachabilityConstraints {
ReachabilityConstraints {
interiors: self.interiors,
}
}
@ -318,8 +344,8 @@ impl VisibilityConstraintsBuilder {
/// any internal node, since they are leaf nodes.
fn cmp_atoms(
&self,
a: ScopedVisibilityConstraintId,
b: ScopedVisibilityConstraintId,
a: ScopedReachabilityConstraintId,
b: ScopedReachabilityConstraintId,
) -> Ordering {
if a == b || (a.is_terminal() && b.is_terminal()) {
Ordering::Equal
@ -332,9 +358,9 @@ impl VisibilityConstraintsBuilder {
}
}
/// Adds an interior node, ensuring that we always use the same visibility constraint ID for
/// Adds an interior node, ensuring that we always use the same reachability constraint ID for
/// equal nodes.
fn add_interior(&mut self, node: InteriorNode) -> ScopedVisibilityConstraintId {
fn add_interior(&mut self, node: InteriorNode) -> ScopedReachabilityConstraintId {
// If the true and false branches lead to the same node, we can override the ambiguous
// branch to go there too. And this node is then redundant and can be reduced.
if node.if_true == node.if_false {
@ -347,7 +373,7 @@ impl VisibilityConstraintsBuilder {
.or_insert_with(|| self.interiors.push(node))
}
/// Adds a new visibility constraint that checks a single [`Predicate`].
/// Adds a new reachability constraint that checks a single [`Predicate`].
///
/// [`ScopedPredicateId`]s are the “variables” that are evaluated by a TDD. A TDD variable has
/// the same value no matter how many times it appears in the ternary formula that the TDD
@ -361,7 +387,7 @@ impl VisibilityConstraintsBuilder {
pub(crate) fn add_atom(
&mut self,
predicate: ScopedPredicateId,
) -> ScopedVisibilityConstraintId {
) -> ScopedReachabilityConstraintId {
self.add_interior(InteriorNode {
atom: predicate,
if_true: ALWAYS_TRUE,
@ -370,11 +396,11 @@ impl VisibilityConstraintsBuilder {
})
}
/// Adds a new visibility constraint that is the ternary NOT of an existing one.
/// Adds a new reachability constraint that is the ternary NOT of an existing one.
pub(crate) fn add_not_constraint(
&mut self,
a: ScopedVisibilityConstraintId,
) -> ScopedVisibilityConstraintId {
a: ScopedReachabilityConstraintId,
) -> ScopedReachabilityConstraintId {
if a == ALWAYS_TRUE {
return ALWAYS_FALSE;
} else if a == AMBIGUOUS {
@ -400,12 +426,12 @@ impl VisibilityConstraintsBuilder {
result
}
/// Adds a new visibility constraint that is the ternary OR of two existing ones.
/// Adds a new reachability constraint that is the ternary OR of two existing ones.
pub(crate) fn add_or_constraint(
&mut self,
a: ScopedVisibilityConstraintId,
b: ScopedVisibilityConstraintId,
) -> ScopedVisibilityConstraintId {
a: ScopedReachabilityConstraintId,
b: ScopedReachabilityConstraintId,
) -> ScopedReachabilityConstraintId {
match (a, b) {
(ALWAYS_TRUE, _) | (_, ALWAYS_TRUE) => return ALWAYS_TRUE,
(ALWAYS_FALSE, other) | (other, ALWAYS_FALSE) => return other,
@ -466,12 +492,12 @@ impl VisibilityConstraintsBuilder {
result
}
/// Adds a new visibility constraint that is the ternary AND of two existing ones.
/// Adds a new reachability constraint that is the ternary AND of two existing ones.
pub(crate) fn add_and_constraint(
&mut self,
a: ScopedVisibilityConstraintId,
b: ScopedVisibilityConstraintId,
) -> ScopedVisibilityConstraintId {
a: ScopedReachabilityConstraintId,
b: ScopedReachabilityConstraintId,
) -> ScopedReachabilityConstraintId {
match (a, b) {
(ALWAYS_FALSE, _) | (_, ALWAYS_FALSE) => return ALWAYS_FALSE,
(ALWAYS_TRUE, other) | (other, ALWAYS_TRUE) => return other,
@ -533,13 +559,13 @@ impl VisibilityConstraintsBuilder {
}
}
impl VisibilityConstraints {
/// Analyze the statically known visibility for a given visibility constraint.
impl ReachabilityConstraints {
/// Analyze the statically known reachability for a given constraint.
pub(crate) fn evaluate<'db>(
&self,
db: &'db dyn Db,
predicates: &Predicates<'db>,
mut id: ScopedVisibilityConstraintId,
mut id: ScopedReachabilityConstraintId,
) -> Truthiness {
loop {
let node = match id {

332
crates/ty_python_semantic/src/semantic_index/use_def.rs
View file

@ -221,43 +221,8 @@
//! snapshot (which has `x = 3` as the only live binding). The result of this merge is that we now
//! have two live bindings of `x`: `x = 3` and `x = 4`.
//!
//! Another piece of information that the `UseDefMap` needs to provide are visibility constraints.
//! These are similar to the narrowing constraints, but apply to bindings and declarations within a
//! control flow path. Consider the following example:
//! ```py
//! x = 1
//! if test:
//! x = 2
//! y = "y"
//! ```
//! In principle, there are two possible control flow paths here. However, if we can statically
//! infer `test` to be always truthy or always falsy (that is, `__bool__` of `test` is of type
//! `Literal[True]` or `Literal[False]`), we can rule out one of the possible paths. To support
//! this feature, we record a visibility constraint of `test` to all live bindings and declarations
//! *after* visiting the body of the `if` statement. And we record a negative visibility constraint
//! `~test` to all live bindings/declarations in the (implicit) `else` branch. For the example
//! above, we would record the following visibility constraints (adding the implicit "unbound"
//! definitions for clarity):
//! ```py
//! x = <unbound> # not live, shadowed by `x = 1`
//! y = <unbound> # visibility constraint: ~test
//!
//! x = 1 # visibility constraint: ~test
//! if test:
//! x = 2 # visibility constraint: test
//! y = "y" # visibility constraint: test
//! ```
//! When we encounter a use of `x` after this `if` statement, we would record two live bindings: `x
//! = 1` with a constraint of `~test`, and `x = 2` with a constraint of `test`. In type inference,
//! when we iterate over all live bindings, we can evaluate these constraints to determine if a
//! particular binding is actually visible. For example, if `test` is always truthy, we only see
//! the `x = 2` binding. If `test` is always falsy, we only see the `x = 1` binding. And if the
//! `__bool__` method of `test` returns type `bool`, we can see both bindings.
//!
//! Note that we also record visibility constraints for the start of the scope. This is important
//! to determine if a place is definitely bound, possibly unbound, or definitely unbound. In the
//! example above, The `y = <unbound>` binding is constrained by `~test`, so `y` would only be
//! definitely-bound if `test` is always truthy.
//! Another piece of information that the `UseDefMap` needs to provide are reachability constraints.
//! See [`reachability_constraints.rs`] for more details, in particular how they apply to bindings.
//!
//! The [`UseDefMapBuilder`] itself just exposes methods for taking a snapshot, resetting to a
//! snapshot, and merging a snapshot into the current state. The logic using these methods lives in
@ -282,8 +247,8 @@ use crate::semantic_index::place::{FileScopeId, PlaceExpr, ScopeKind, ScopedPlac
use crate::semantic_index::predicate::{
Predicate, Predicates, PredicatesBuilder, ScopedPredicateId, StarImportPlaceholderPredicate,
};
use crate::semantic_index::visibility_constraints::{
ScopedVisibilityConstraintId, VisibilityConstraints, VisibilityConstraintsBuilder,
use crate::semantic_index::reachability_constraints::{
ReachabilityConstraints, ReachabilityConstraintsBuilder, ScopedReachabilityConstraintId,
};
use crate::types::{IntersectionBuilder, Truthiness, Type, infer_narrowing_constraint};
@ -302,14 +267,14 @@ pub(crate) struct UseDefMap<'db> {
/// Array of narrowing constraints in this scope.
narrowing_constraints: NarrowingConstraints,
/// Array of visibility constraints in this scope.
visibility_constraints: VisibilityConstraints,
/// Array of reachability constraints in this scope.
reachability_constraints: ReachabilityConstraints,
/// [`Bindings`] reaching a [`ScopedUseId`].
bindings_by_use: IndexVec<ScopedUseId, Bindings>,
/// Tracks whether or not a given AST node is reachable from the start of the scope.
node_reachability: FxHashMap<NodeKey, ScopedVisibilityConstraintId>,
node_reachability: FxHashMap<NodeKey, ScopedReachabilityConstraintId>,
/// If the definition is a binding (only) -- `x = 1` for example -- then we need
/// [`Declarations`] to know whether this binding is permitted by the live declarations.
@ -335,20 +300,24 @@ pub(crate) struct UseDefMap<'db> {
/// eager scope.
eager_snapshots: EagerSnapshots,
/// Whether or not the start of the scope is visible.
/// Whether or not the end of the scope is reachable.
///
/// This is used to check if the function can implicitly return `None`.
/// For example:
///
/// ```python
/// def f(cond: bool) -> int:
/// ```py
/// def f(cond: bool) -> int | None:
/// if cond:
/// return 1
///
/// def g() -> int:
/// if True:
/// return 1
/// ```
///
/// In this case, the function may implicitly return `None`.
/// Function `f` may implicitly return `None`, but `g` cannot.
///
/// This is used by `UseDefMap::can_implicit_return`.
scope_start_visibility: ScopedVisibilityConstraintId,
/// This is used by [`UseDefMap::can_implicitly_return_none`].
end_of_scope_reachability: ScopedReachabilityConstraintId,
}
impl<'db> UseDefMap<'db> {
@ -378,12 +347,11 @@ impl<'db> UseDefMap<'db> {
pub(super) fn is_reachable(
&self,
db: &dyn crate::Db,
reachability: ScopedVisibilityConstraintId,
reachability: ScopedReachabilityConstraintId,
) -> bool {
!self
.visibility_constraints
self.reachability_constraints
.evaluate(db, &self.predicates, reachability)
.is_always_false()
.may_be_true()
}
/// Check whether or not a given expression is reachable from the start of the scope. This
@ -392,8 +360,8 @@ impl<'db> UseDefMap<'db> {
/// analysis.
#[track_caller]
pub(super) fn is_node_reachable(&self, db: &dyn crate::Db, node_key: NodeKey) -> bool {
!self
.visibility_constraints
self
.reachability_constraints
.evaluate(
db,
&self.predicates,
@ -402,7 +370,7 @@ impl<'db> UseDefMap<'db> {
.get(&node_key)
.expect("`is_node_reachable` should only be called on AST nodes with recorded reachability"),
)
.is_always_false()
.may_be_true()
}
pub(crate) fn public_bindings(
@ -467,20 +435,23 @@ impl<'db> UseDefMap<'db> {
}
/// This function is intended to be called only once inside `TypeInferenceBuilder::infer_function_body`.
pub(crate) fn can_implicit_return(&self, db: &dyn crate::Db) -> bool {
pub(crate) fn can_implicitly_return_none(&self, db: &dyn crate::Db) -> bool {
!self
.visibility_constraints
.evaluate(db, &self.predicates, self.scope_start_visibility)
.reachability_constraints
.evaluate(db, &self.predicates, self.end_of_scope_reachability)
.is_always_false()
}
pub(crate) fn is_binding_visible(
pub(crate) fn is_binding_reachable(
&self,
db: &dyn crate::Db,
binding: &BindingWithConstraints<'_, 'db>,
) -> Truthiness {
self.visibility_constraints
.evaluate(db, &self.predicates, binding.visibility_constraint)
self.reachability_constraints.evaluate(
db,
&self.predicates,
binding.reachability_constraint,
)
}
fn bindings_iterator<'map>(
@ -491,7 +462,7 @@ impl<'db> UseDefMap<'db> {
all_definitions: &self.all_definitions,
predicates: &self.predicates,
narrowing_constraints: &self.narrowing_constraints,
visibility_constraints: &self.visibility_constraints,
reachability_constraints: &self.reachability_constraints,
inner: bindings.iter(),
}
}
@ -503,7 +474,7 @@ impl<'db> UseDefMap<'db> {
DeclarationsIterator {
all_definitions: &self.all_definitions,
predicates: &self.predicates,
visibility_constraints: &self.visibility_constraints,
reachability_constraints: &self.reachability_constraints,
inner: declarations.iter(),
}
}
@ -538,7 +509,7 @@ pub(crate) struct BindingWithConstraintsIterator<'map, 'db> {
all_definitions: &'map IndexVec<ScopedDefinitionId, DefinitionState<'db>>,
pub(crate) predicates: &'map Predicates<'db>,
pub(crate) narrowing_constraints: &'map NarrowingConstraints,
pub(crate) visibility_constraints: &'map VisibilityConstraints,
pub(crate) reachability_constraints: &'map ReachabilityConstraints,
inner: LiveBindingsIterator<'map>,
}
@ -558,7 +529,7 @@ impl<'map, 'db> Iterator for BindingWithConstraintsIterator<'map, 'db> {
constraint_ids: narrowing_constraints
.iter_predicates(live_binding.narrowing_constraint),
},
visibility_constraint: live_binding.visibility_constraint,
reachability_constraint: live_binding.reachability_constraint,
})
}
}
@ -568,7 +539,7 @@ impl std::iter::FusedIterator for BindingWithConstraintsIterator<'_, '_> {}
pub(crate) struct BindingWithConstraints<'map, 'db> {
pub(crate) binding: DefinitionState<'db>,
pub(crate) narrowing_constraint: ConstraintsIterator<'map, 'db>,
pub(crate) visibility_constraint: ScopedVisibilityConstraintId,
pub(crate) reachability_constraint: ScopedReachabilityConstraintId,
}
pub(crate) struct ConstraintsIterator<'map, 'db> {
@ -618,13 +589,13 @@ impl<'db> ConstraintsIterator<'_, 'db> {
pub(crate) struct DeclarationsIterator<'map, 'db> {
all_definitions: &'map IndexVec<ScopedDefinitionId, DefinitionState<'db>>,
pub(crate) predicates: &'map Predicates<'db>,
pub(crate) visibility_constraints: &'map VisibilityConstraints,
pub(crate) reachability_constraints: &'map ReachabilityConstraints,
inner: LiveDeclarationsIterator<'map>,
}
pub(crate) struct DeclarationWithConstraint<'db> {
pub(crate) declaration: DefinitionState<'db>,
pub(crate) visibility_constraint: ScopedVisibilityConstraintId,
pub(crate) reachability_constraint: ScopedReachabilityConstraintId,
}
impl<'db> Iterator for DeclarationsIterator<'_, 'db> {
@ -634,11 +605,11 @@ impl<'db> Iterator for DeclarationsIterator<'_, 'db> {
self.inner.next().map(
|LiveDeclaration {
declaration,
visibility_constraint,
reachability_constraint,
}| {
DeclarationWithConstraint {
declaration: self.all_definitions[*declaration],
visibility_constraint: *visibility_constraint,
reachability_constraint: *reachability_constraint,
}
},
)
@ -651,8 +622,7 @@ impl std::iter::FusedIterator for DeclarationsIterator<'_, '_> {}
#[derive(Clone, Debug)]
pub(super) struct FlowSnapshot {
place_states: IndexVec<ScopedPlaceId, PlaceState>,
scope_start_visibility: ScopedVisibilityConstraintId,
reachability: ScopedVisibilityConstraintId,
reachability: ScopedReachabilityConstraintId,
}
#[derive(Debug)]
@ -666,60 +636,18 @@ pub(super) struct UseDefMapBuilder<'db> {
/// Builder of narrowing constraints.
pub(super) narrowing_constraints: NarrowingConstraintsBuilder,
/// Builder of visibility constraints.
pub(super) visibility_constraints: VisibilityConstraintsBuilder,
/// A constraint which describes the visibility of the unbound/undeclared state, i.e.
/// whether or not a use of a place at the current point in control flow would see
/// the fake `x = <unbound>` binding at the start of the scope. This is important for
/// cases like the following, where we need to hide the implicit unbound binding in
/// the "else" branch:
/// ```py
/// # x = <unbound>
///
/// if True:
/// x = 1
///
/// use(x) # the `x = <unbound>` binding is not visible here
/// ```
pub(super) scope_start_visibility: ScopedVisibilityConstraintId,
/// Builder of reachability constraints.
pub(super) reachability_constraints: ReachabilityConstraintsBuilder,
/// Live bindings at each so-far-recorded use.
bindings_by_use: IndexVec<ScopedUseId, Bindings>,
/// Tracks whether or not the scope start is visible at the current point in control flow.
/// This is subtly different from `scope_start_visibility`, as we apply these constraints
/// at the beginning of a branch. Visibility constraints, on the other hand, need to be
/// applied at the end of a branch, as we apply them retroactively to all live bindings:
/// ```py
/// y = 1
///
/// if test:
/// # we record a reachability constraint of [test] here,
/// # so that it can affect the use of `x`:
///
/// x # we store a reachability constraint of [test] for this use of `x`
///
/// y = 2
///
/// # we record a visibility constraint of [test] here, which retroactively affects
/// # the `y = 1` and the `y = 2` binding.
/// else:
/// # we record a reachability constraint of [~test] here.
///
/// pass
///
/// # we record a visibility constraint of [~test] here, which retroactively affects
/// # the `y = 1` binding.
///
/// use(y)
/// ```
/// Depending on the value of `test`, the `y = 1`, `y = 2`, or both bindings may be visible.
/// The use of `x` is recorded with a reachability constraint of `[test]`.
pub(super) reachability: ScopedVisibilityConstraintId,
/// Tracks whether or not the current point in control flow is reachable from the
/// start of the scope.
pub(super) reachability: ScopedReachabilityConstraintId,
/// Tracks whether or not a given AST node is reachable from the start of the scope.
node_reachability: FxHashMap<NodeKey, ScopedVisibilityConstraintId>,
node_reachability: FxHashMap<NodeKey, ScopedReachabilityConstraintId>,
/// Live declarations for each so-far-recorded binding.
declarations_by_binding: FxHashMap<Definition<'db>, Declarations>,
@ -744,10 +672,9 @@ impl<'db> UseDefMapBuilder<'db> {
all_definitions: IndexVec::from_iter([DefinitionState::Undefined]),
predicates: PredicatesBuilder::default(),
narrowing_constraints: NarrowingConstraintsBuilder::default(),
visibility_constraints: VisibilityConstraintsBuilder::default(),
scope_start_visibility: ScopedVisibilityConstraintId::ALWAYS_TRUE,
reachability_constraints: ReachabilityConstraintsBuilder::default(),
bindings_by_use: IndexVec::new(),
reachability: ScopedVisibilityConstraintId::ALWAYS_TRUE,
reachability: ScopedReachabilityConstraintId::ALWAYS_TRUE,
node_reachability: FxHashMap::default(),
declarations_by_binding: FxHashMap::default(),
bindings_by_declaration: FxHashMap::default(),
@ -757,14 +684,20 @@ impl<'db> UseDefMapBuilder<'db> {
}
}
pub(super) fn mark_unreachable(&mut self) {
self.record_visibility_constraint(ScopedVisibilityConstraintId::ALWAYS_FALSE);
self.reachability = ScopedVisibilityConstraintId::ALWAYS_FALSE;
self.reachability = ScopedReachabilityConstraintId::ALWAYS_FALSE;
for state in &mut self.place_states {
state.record_reachability_constraint(
&mut self.reachability_constraints,
ScopedReachabilityConstraintId::ALWAYS_FALSE,
);
}
}
pub(super) fn add_place(&mut self, place: ScopedPlaceId) {
let new_place = self
.place_states
.push(PlaceState::undefined(self.scope_start_visibility));
.push(PlaceState::undefined(self.reachability));
debug_assert_eq!(place, new_place);
}
@ -780,7 +713,7 @@ impl<'db> UseDefMapBuilder<'db> {
.insert(binding, place_state.declarations().clone());
place_state.record_binding(
def_id,
self.scope_start_visibility,
self.reachability,
self.is_class_scope,
is_place_name,
);
@ -798,136 +731,85 @@ impl<'db> UseDefMapBuilder<'db> {
}
}
pub(super) fn record_visibility_constraint(
&mut self,
constraint: ScopedVisibilityConstraintId,
) {
for state in &mut self.place_states {
state.record_visibility_constraint(&mut self.visibility_constraints, constraint);
}
self.scope_start_visibility = self
.visibility_constraints
.add_and_constraint(self.scope_start_visibility, constraint);
}
/// Snapshot the state of a single place at the current point in control flow.
///
/// This is only used for `*`-import visibility constraints, which are handled differently
/// to most other visibility constraints. See the doc-comment for
/// [`Self::record_and_negate_star_import_visibility_constraint`] for more details.
/// This is only used for `*`-import reachability constraints, which are handled differently
/// to most other reachability constraints. See the doc-comment for
/// [`Self::record_and_negate_star_import_reachability_constraint`] for more details.
pub(super) fn single_place_snapshot(&self, place: ScopedPlaceId) -> PlaceState {
self.place_states[place].clone()
}
/// This method exists solely for handling `*`-import visibility constraints.
/// This method exists solely for handling `*`-import reachability constraints.
///
/// The reason why we add visibility constraints for [`Definition`]s created by `*` imports
/// The reason why we add reachability constraints for [`Definition`]s created by `*` imports
/// is laid out in the doc-comment for [`StarImportPlaceholderPredicate`]. But treating these
/// visibility constraints in the use-def map the same way as all other visibility constraints
/// reachability constraints in the use-def map the same way as all other reachability constraints
/// was shown to lead to [significant regressions] for small codebases where typeshed
/// dominates. (Although `*` imports are not common generally, they are used in several
/// important places by typeshed.)
///
/// To solve these regressions, it was observed that we could do significantly less work for
/// `*`-import definitions. We do a number of things differently here to our normal handling of
/// visibility constraints:
/// reachability constraints:
///
/// - We only apply and negate the visibility constraints to a single symbol, rather than to
/// - We only apply and negate the reachability constraints to a single symbol, rather than to
/// all symbols. This is possible here because, unlike most definitions, we know in advance that
/// exactly one definition occurs inside the "if-true" predicate branch, and we know exactly
/// which definition it is.
///
/// Doing things this way is cheaper in and of itself. However, it also allows us to avoid
/// calling [`Self::simplify_visibility_constraints`] after the constraint has been applied to
/// the "if-predicate-true" branch and negated for the "if-predicate-false" branch. Simplifying
/// the visibility constraints is only important for places that did not have any new
/// definitions inside either the "if-predicate-true" branch or the "if-predicate-false" branch.
///
/// - We only snapshot the state for a single place prior to the definition, rather than doing
/// expensive calls to [`Self::snapshot`]. Again, this is possible because we know
/// that only a single definition occurs inside the "if-predicate-true" predicate branch.
///
/// - Normally we take care to check whether an "if-predicate-true" branch or an
/// "if-predicate-false" branch contains a terminal statement: these can affect the visibility
/// "if-predicate-false" branch contains a terminal statement: these can affect the reachability
/// of symbols defined inside either branch. However, in the case of `*`-import definitions,
/// this is unnecessary (and therefore not done in this method), since we know that a `*`-import
/// predicate cannot create a terminal statement inside either branch.
///
/// [significant regressions]: https://github.com/astral-sh/ruff/pull/17286#issuecomment-2786755746
pub(super) fn record_and_negate_star_import_visibility_constraint(
pub(super) fn record_and_negate_star_import_reachability_constraint(
&mut self,
star_import: StarImportPlaceholderPredicate<'db>,
symbol: ScopedPlaceId,
pre_definition_state: PlaceState,
) {
let predicate_id = self.add_predicate(star_import.into());
let visibility_id = self.visibility_constraints.add_atom(predicate_id);
let negated_visibility_id = self
.visibility_constraints
.add_not_constraint(visibility_id);
let reachability_id = self.reachability_constraints.add_atom(predicate_id);
let negated_reachability_id = self
.reachability_constraints
.add_not_constraint(reachability_id);
let mut post_definition_state =
std::mem::replace(&mut self.place_states[symbol], pre_definition_state);
post_definition_state
.record_visibility_constraint(&mut self.visibility_constraints, visibility_id);
.record_reachability_constraint(&mut self.reachability_constraints, reachability_id);
self.place_states[symbol]
.record_visibility_constraint(&mut self.visibility_constraints, negated_visibility_id);
self.place_states[symbol].record_reachability_constraint(
&mut self.reachability_constraints,
negated_reachability_id,
);
self.place_states[symbol].merge(
post_definition_state,
&mut self.narrowing_constraints,
&mut self.visibility_constraints,
&mut self.reachability_constraints,
);
}
/// This method resets the visibility constraints for all places to a previous state
/// *if* there have been no new declarations or bindings since then. Consider the
/// following example:
/// ```py
/// x = 0
/// y = 0
/// if test_a:
/// y = 1
/// elif test_b:
/// y = 2
/// elif test_c:
/// y = 3
///
/// # RESET
/// ```
/// We build a complex visibility constraint for the `y = 0` binding. We build the same
/// constraint for the `x = 0` binding as well, but at the `RESET` point, we can get rid
/// of it, as the `if`-`elif`-`elif` chain doesn't include any new bindings of `x`.
pub(super) fn simplify_visibility_constraints(&mut self, snapshot: FlowSnapshot) {
debug_assert!(self.place_states.len() >= snapshot.place_states.len());
// If there are any control flow paths that have become unreachable between `snapshot` and
// now, then it's not valid to simplify any visibility constraints to `snapshot`.
if self.scope_start_visibility != snapshot.scope_start_visibility {
return;
}
// Note that this loop terminates when we reach a place not present in the snapshot.
// This means we keep visibility constraints for all new places, which is intended,
// since these places have been introduced in the corresponding branch, which might
// be subject to visibility constraints. We only simplify/reset visibility constraints
// for places that have the same bindings and declarations present compared to the
// snapshot.
for (current, snapshot) in self.place_states.iter_mut().zip(snapshot.place_states) {
current.simplify_visibility_constraints(snapshot);
}
}
pub(super) fn record_reachability_constraint(
&mut self,
constraint: ScopedVisibilityConstraintId,
) -> ScopedVisibilityConstraintId {
constraint: ScopedReachabilityConstraintId,
) {
self.reachability = self
.visibility_constraints
.reachability_constraints
.add_and_constraint(self.reachability, constraint);
self.reachability
for state in &mut self.place_states {
state.record_reachability_constraint(&mut self.reachability_constraints, constraint);
}
}
pub(super) fn record_declaration(
@ -941,7 +823,7 @@ impl<'db> UseDefMapBuilder<'db> {
let place_state = &mut self.place_states[place];
self.bindings_by_declaration
.insert(declaration, place_state.bindings().clone());
place_state.record_declaration(def_id);
place_state.record_declaration(def_id, self.reachability);
}
pub(super) fn record_declaration_and_binding(
@ -956,10 +838,10 @@ impl<'db> UseDefMapBuilder<'db> {
.all_definitions
.push(DefinitionState::Defined(definition));
let place_state = &mut self.place_states[place];
place_state.record_declaration(def_id);
place_state.record_declaration(def_id, self.reachability);
place_state.record_binding(
def_id,
self.scope_start_visibility,
self.reachability,
self.is_class_scope,
is_place_name,
);
@ -970,7 +852,7 @@ impl<'db> UseDefMapBuilder<'db> {
let place_state = &mut self.place_states[place];
place_state.record_binding(
def_id,
self.scope_start_visibility,
self.reachability,
self.is_class_scope,
is_place_name,
);
@ -1024,7 +906,6 @@ impl<'db> UseDefMapBuilder<'db> {
pub(super) fn snapshot(&self) -> FlowSnapshot {
FlowSnapshot {
place_states: self.place_states.clone(),
scope_start_visibility: self.scope_start_visibility,
reachability: self.reachability,
}
}
@ -1039,16 +920,13 @@ impl<'db> UseDefMapBuilder<'db> {
// Restore the current visible-definitions state to the given snapshot.
self.place_states = snapshot.place_states;
self.scope_start_visibility = snapshot.scope_start_visibility;
self.reachability = snapshot.reachability;
// If the snapshot we are restoring is missing some places we've recorded since, we need
// to fill them in so the place IDs continue to line up. Since they don't exist in the
// snapshot, the correct state to fill them in with is "undefined".
self.place_states.resize(
num_places,
PlaceState::undefined(self.scope_start_visibility),
);
self.place_states
.resize(num_places, PlaceState::undefined(self.reachability));
}
/// Merge the given snapshot into the current state, reflecting that we might have taken either
@ -1059,13 +937,13 @@ impl<'db> UseDefMapBuilder<'db> {
// unreachable, we can leave it out of the merged result entirely. Note that we cannot
// perform any type inference at this point, so this is largely limited to unreachability
// via terminal statements. If a flow's reachability depends on an expression in the code,
// we will include the flow in the merged result; the visibility constraints of its
// we will include the flow in the merged result; the reachability constraints of its
// bindings will include this reachability condition, so that later during type inference,
// we can determine whether any particular binding is non-visible due to unreachability.
if snapshot.scope_start_visibility == ScopedVisibilityConstraintId::ALWAYS_FALSE {
if snapshot.reachability == ScopedReachabilityConstraintId::ALWAYS_FALSE {
return;
}
if self.scope_start_visibility == ScopedVisibilityConstraintId::ALWAYS_FALSE {
if self.reachability == ScopedReachabilityConstraintId::ALWAYS_FALSE {
self.restore(snapshot);
return;
}
@ -1081,24 +959,20 @@ impl<'db> UseDefMapBuilder<'db> {
current.merge(
snapshot,
&mut self.narrowing_constraints,
&mut self.visibility_constraints,
&mut self.reachability_constraints,
);
} else {
current.merge(
PlaceState::undefined(snapshot.scope_start_visibility),
PlaceState::undefined(snapshot.reachability),
&mut self.narrowing_constraints,
&mut self.visibility_constraints,
&mut self.reachability_constraints,
);
// Place not present in snapshot, so it's unbound/undeclared from that path.
}
}
self.scope_start_visibility = self
.visibility_constraints
.add_or_constraint(self.scope_start_visibility, snapshot.scope_start_visibility);
self.reachability = self
.visibility_constraints
.reachability_constraints
.add_or_constraint(self.reachability, snapshot.reachability);
}
@ -1115,14 +989,14 @@ impl<'db> UseDefMapBuilder<'db> {
all_definitions: self.all_definitions,
predicates: self.predicates.build(),
narrowing_constraints: self.narrowing_constraints.build(),
visibility_constraints: self.visibility_constraints.build(),
reachability_constraints: self.reachability_constraints.build(),
bindings_by_use: self.bindings_by_use,
node_reachability: self.node_reachability,
public_places: self.place_states,
declarations_by_binding: self.declarations_by_binding,
bindings_by_declaration: self.bindings_by_declaration,
eager_snapshots: self.eager_snapshots,
scope_start_visibility: self.scope_start_visibility,
end_of_scope_reachability: self.reachability,
}
}
}

261
crates/ty_python_semantic/src/semantic_index/use_def/place_state.rs
View file

@ -49,8 +49,8 @@ use smallvec::{SmallVec, smallvec};
use crate::semantic_index::narrowing_constraints::{
NarrowingConstraintsBuilder, ScopedNarrowingConstraint, ScopedNarrowingConstraintPredicate,
};
use crate::semantic_index::visibility_constraints::{
ScopedVisibilityConstraintId, VisibilityConstraintsBuilder,
use crate::semantic_index::reachability_constraints::{
ReachabilityConstraintsBuilder, ScopedReachabilityConstraintId,
};
/// A newtype-index for a definition in a particular scope.
@ -76,7 +76,7 @@ impl ScopedDefinitionId {
const INLINE_DEFINITIONS_PER_PLACE: usize = 4;
/// Live declarations for a single place at some point in control flow, with their
/// corresponding visibility constraints.
/// corresponding reachability constraints.
#[derive(Clone, Debug, Default, PartialEq, Eq, salsa::Update)]
pub(super) struct Declarations {
/// A list of live declarations for this place, sorted by their `ScopedDefinitionId`
@ -87,16 +87,16 @@ pub(super) struct Declarations {
#[derive(Clone, Debug, PartialEq, Eq)]
pub(super) struct LiveDeclaration {
pub(super) declaration: ScopedDefinitionId,
pub(super) visibility_constraint: ScopedVisibilityConstraintId,
pub(super) reachability_constraint: ScopedReachabilityConstraintId,
}
pub(super) type LiveDeclarationsIterator<'a> = std::slice::Iter<'a, LiveDeclaration>;
impl Declarations {
fn undeclared(scope_start_visibility: ScopedVisibilityConstraintId) -> Self {
fn undeclared(reachability_constraint: ScopedReachabilityConstraintId) -> Self {
let initial_declaration = LiveDeclaration {
declaration: ScopedDefinitionId::UNBOUND,
visibility_constraint: scope_start_visibility,
reachability_constraint,
};
Self {
live_declarations: smallvec![initial_declaration],
@ -104,24 +104,28 @@ impl Declarations {
}
/// Record a newly-encountered declaration for this place.
fn record_declaration(&mut self, declaration: ScopedDefinitionId) {
fn record_declaration(
&mut self,
declaration: ScopedDefinitionId,
reachability_constraint: ScopedReachabilityConstraintId,
) {
// The new declaration replaces all previous live declaration in this path.
self.live_declarations.clear();
self.live_declarations.push(LiveDeclaration {
declaration,
visibility_constraint: ScopedVisibilityConstraintId::ALWAYS_TRUE,
reachability_constraint,
});
}
/// Add given visibility constraint to all live declarations.
pub(super) fn record_visibility_constraint(
/// Add given reachability constraint to all live declarations.
pub(super) fn record_reachability_constraint(
&mut self,
visibility_constraints: &mut VisibilityConstraintsBuilder,
constraint: ScopedVisibilityConstraintId,
reachability_constraints: &mut ReachabilityConstraintsBuilder,
constraint: ScopedReachabilityConstraintId,
) {
for declaration in &mut self.live_declarations {
declaration.visibility_constraint = visibility_constraints
.add_and_constraint(declaration.visibility_constraint, constraint);
declaration.reachability_constraint = reachability_constraints
.add_and_constraint(declaration.reachability_constraint, constraint);
}
}
@ -130,46 +134,24 @@ impl Declarations {
self.live_declarations.iter()
}
/// Iterate over the IDs of each currently live declaration for this place
fn iter_declarations(&self) -> impl Iterator<Item = ScopedDefinitionId> + '_ {
self.iter().map(|lb| lb.declaration)
}
fn simplify_visibility_constraints(&mut self, other: Declarations) {
// If the set of live declarations hasn't changed, don't simplify.
if self.live_declarations.len() != other.live_declarations.len()
|| !self.iter_declarations().eq(other.iter_declarations())
{
return;
}
for (declaration, other_declaration) in self
.live_declarations
.iter_mut()
.zip(other.live_declarations)
{
declaration.visibility_constraint = other_declaration.visibility_constraint;
}
}
fn merge(&mut self, b: Self, visibility_constraints: &mut VisibilityConstraintsBuilder) {
fn merge(&mut self, b: Self, reachability_constraints: &mut ReachabilityConstraintsBuilder) {
let a = std::mem::take(self);
// Invariant: merge_join_by consumes the two iterators in sorted order, which ensures that
// the merged `live_declarations` vec remains sorted. If a definition is found in both `a`
// and `b`, we compose the constraints from the two paths in an appropriate way
// (intersection for narrowing constraints; ternary OR for visibility constraints). If a
// (intersection for narrowing constraints; ternary OR for reachability constraints). If a
// definition is found in only one path, it is used as-is.
let a = a.live_declarations.into_iter();
let b = b.live_declarations.into_iter();
for zipped in a.merge_join_by(b, |a, b| a.declaration.cmp(&b.declaration)) {
match zipped {
EitherOrBoth::Both(a, b) => {
let visibility_constraint = visibility_constraints
.add_or_constraint(a.visibility_constraint, b.visibility_constraint);
let reachability_constraint = reachability_constraints
.add_or_constraint(a.reachability_constraint, b.reachability_constraint);
self.live_declarations.push(LiveDeclaration {
declaration: a.declaration,
visibility_constraint,
reachability_constraint,
});
}
@ -193,7 +175,7 @@ pub(super) enum EagerSnapshot {
}
/// Live bindings for a single place at some point in control flow. Each live binding comes
/// with a set of narrowing constraints and a visibility constraint.
/// with a set of narrowing constraints and a reachability constraint.
#[derive(Clone, Debug, Default, PartialEq, Eq, salsa::Update)]
pub(super) struct Bindings {
/// The narrowing constraint applicable to the "unbound" binding, if we need access to it even
@ -217,17 +199,17 @@ impl Bindings {
pub(super) struct LiveBinding {
pub(super) binding: ScopedDefinitionId,
pub(super) narrowing_constraint: ScopedNarrowingConstraint,
pub(super) visibility_constraint: ScopedVisibilityConstraintId,
pub(super) reachability_constraint: ScopedReachabilityConstraintId,
}
pub(super) type LiveBindingsIterator<'a> = std::slice::Iter<'a, LiveBinding>;
impl Bindings {
fn unbound(scope_start_visibility: ScopedVisibilityConstraintId) -> Self {
fn unbound(reachability_constraint: ScopedReachabilityConstraintId) -> Self {
let initial_binding = LiveBinding {
binding: ScopedDefinitionId::UNBOUND,
narrowing_constraint: ScopedNarrowingConstraint::empty(),
visibility_constraint: scope_start_visibility,
reachability_constraint,
};
Self {
unbound_narrowing_constraint: None,
@ -239,7 +221,7 @@ impl Bindings {
pub(super) fn record_binding(
&mut self,
binding: ScopedDefinitionId,
visibility_constraint: ScopedVisibilityConstraintId,
reachability_constraint: ScopedReachabilityConstraintId,
is_class_scope: bool,
is_place_name: bool,
) {
@ -254,7 +236,7 @@ impl Bindings {
self.live_bindings.push(LiveBinding {
binding,
narrowing_constraint: ScopedNarrowingConstraint::empty(),
visibility_constraint,
reachability_constraint,
});
}
@ -270,15 +252,15 @@ impl Bindings {
}
}
/// Add given visibility constraint to all live bindings.
pub(super) fn record_visibility_constraint(
/// Add given reachability constraint to all live bindings.
pub(super) fn record_reachability_constraint(
&mut self,
visibility_constraints: &mut VisibilityConstraintsBuilder,
constraint: ScopedVisibilityConstraintId,
reachability_constraints: &mut ReachabilityConstraintsBuilder,
constraint: ScopedReachabilityConstraintId,
) {
for binding in &mut self.live_bindings {
binding.visibility_constraint = visibility_constraints
.add_and_constraint(binding.visibility_constraint, constraint);
binding.reachability_constraint = reachability_constraints
.add_and_constraint(binding.reachability_constraint, constraint);
}
}
@ -287,29 +269,11 @@ impl Bindings {
self.live_bindings.iter()
}
/// Iterate over the IDs of each currently live binding for this place
fn iter_bindings(&self) -> impl Iterator<Item = ScopedDefinitionId> + '_ {
self.iter().map(|lb| lb.binding)
}
fn simplify_visibility_constraints(&mut self, other: Bindings) {
// If the set of live bindings hasn't changed, don't simplify.
if self.live_bindings.len() != other.live_bindings.len()
|| !self.iter_bindings().eq(other.iter_bindings())
{
return;
}
for (binding, other_binding) in self.live_bindings.iter_mut().zip(other.live_bindings) {
binding.visibility_constraint = other_binding.visibility_constraint;
}
}
fn merge(
&mut self,
b: Self,
narrowing_constraints: &mut NarrowingConstraintsBuilder,
visibility_constraints: &mut VisibilityConstraintsBuilder,
reachability_constraints: &mut ReachabilityConstraintsBuilder,
) {
let a = std::mem::take(self);
@ -324,7 +288,7 @@ impl Bindings {
// Invariant: merge_join_by consumes the two iterators in sorted order, which ensures that
// the merged `live_bindings` vec remains sorted. If a definition is found in both `a` and
// `b`, we compose the constraints from the two paths in an appropriate way (intersection
// for narrowing constraints; ternary OR for visibility constraints). If a definition is
// for narrowing constraints; ternary OR for reachability constraints). If a definition is
// found in only one path, it is used as-is.
let a = a.live_bindings.into_iter();
let b = b.live_bindings.into_iter();
@ -337,14 +301,14 @@ impl Bindings {
let narrowing_constraint = narrowing_constraints
.intersect_constraints(a.narrowing_constraint, b.narrowing_constraint);
// For visibility constraints, we merge them using a ternary OR operation:
let visibility_constraint = visibility_constraints
.add_or_constraint(a.visibility_constraint, b.visibility_constraint);
// For reachability constraints, we merge them using a ternary OR operation:
let reachability_constraint = reachability_constraints
.add_or_constraint(a.reachability_constraint, b.reachability_constraint);
self.live_bindings.push(LiveBinding {
binding: a.binding,
narrowing_constraint,
visibility_constraint,
reachability_constraint,
});
}
@ -364,10 +328,10 @@ pub(in crate::semantic_index) struct PlaceState {
impl PlaceState {
/// Return a new [`PlaceState`] representing an unbound, undeclared place.
pub(super) fn undefined(scope_start_visibility: ScopedVisibilityConstraintId) -> Self {
pub(super) fn undefined(reachability: ScopedReachabilityConstraintId) -> Self {
Self {
declarations: Declarations::undeclared(scope_start_visibility),
bindings: Bindings::unbound(scope_start_visibility),
declarations: Declarations::undeclared(reachability),
bindings: Bindings::unbound(reachability),
}
}
@ -375,14 +339,14 @@ impl PlaceState {
pub(super) fn record_binding(
&mut self,
binding_id: ScopedDefinitionId,
visibility_constraint: ScopedVisibilityConstraintId,
reachability_constraint: ScopedReachabilityConstraintId,
is_class_scope: bool,
is_place_name: bool,
) {
debug_assert_ne!(binding_id, ScopedDefinitionId::UNBOUND);
self.bindings.record_binding(
binding_id,
visibility_constraint,
reachability_constraint,
is_class_scope,
is_place_name,
);
@ -398,31 +362,26 @@ impl PlaceState {
.record_narrowing_constraint(narrowing_constraints, constraint);
}
/// Add given visibility constraint to all live bindings.
pub(super) fn record_visibility_constraint(
/// Add given reachability constraint to all live bindings.
pub(super) fn record_reachability_constraint(
&mut self,
visibility_constraints: &mut VisibilityConstraintsBuilder,
constraint: ScopedVisibilityConstraintId,
reachability_constraints: &mut ReachabilityConstraintsBuilder,
constraint: ScopedReachabilityConstraintId,
) {
self.bindings
.record_visibility_constraint(visibility_constraints, constraint);
.record_reachability_constraint(reachability_constraints, constraint);
self.declarations
.record_visibility_constraint(visibility_constraints, constraint);
}
/// Simplifies this snapshot to have the same visibility constraints as a previous point in the
/// control flow, but only if the set of live bindings or declarations for this place hasn't
/// changed.
pub(super) fn simplify_visibility_constraints(&mut self, snapshot_state: PlaceState) {
self.bindings
.simplify_visibility_constraints(snapshot_state.bindings);
self.declarations
.simplify_visibility_constraints(snapshot_state.declarations);
.record_reachability_constraint(reachability_constraints, constraint);
}
/// Record a newly-encountered declaration of this place.
pub(super) fn record_declaration(&mut self, declaration_id: ScopedDefinitionId) {
self.declarations.record_declaration(declaration_id);
pub(super) fn record_declaration(
&mut self,
declaration_id: ScopedDefinitionId,
reachability_constraint: ScopedReachabilityConstraintId,
) {
self.declarations
.record_declaration(declaration_id, reachability_constraint);
}
/// Merge another [`PlaceState`] into this one.
@ -430,12 +389,12 @@ impl PlaceState {
&mut self,
b: PlaceState,
narrowing_constraints: &mut NarrowingConstraintsBuilder,
visibility_constraints: &mut VisibilityConstraintsBuilder,
reachability_constraints: &mut ReachabilityConstraintsBuilder,
) {
self.bindings
.merge(b.bindings, narrowing_constraints, visibility_constraints);
.merge(b.bindings, narrowing_constraints, reachability_constraints);
self.declarations
.merge(b.declarations, visibility_constraints);
.merge(b.declarations, reachability_constraints);
}
pub(super) fn bindings(&self) -> &Bindings {
@ -488,7 +447,7 @@ mod tests {
.map(
|LiveDeclaration {
declaration,
visibility_constraint: _,
reachability_constraint: _,
}| {
if *declaration == ScopedDefinitionId::UNBOUND {
"undeclared".into()
@ -504,7 +463,7 @@ mod tests {
#[test]
fn unbound() {
let narrowing_constraints = NarrowingConstraintsBuilder::default();
let sym = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
let sym = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
assert_bindings(&narrowing_constraints, &sym, &["unbound<>"]);
}
@ -512,10 +471,10 @@ mod tests {
#[test]
fn with() {
let narrowing_constraints = NarrowingConstraintsBuilder::default();
let mut sym = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
let mut sym = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym.record_binding(
ScopedDefinitionId::from_u32(1),
ScopedVisibilityConstraintId::ALWAYS_TRUE,
ScopedReachabilityConstraintId::ALWAYS_TRUE,
false,
true,
);
@ -526,10 +485,10 @@ mod tests {
#[test]
fn record_constraint() {
let mut narrowing_constraints = NarrowingConstraintsBuilder::default();
let mut sym = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
let mut sym = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym.record_binding(
ScopedDefinitionId::from_u32(1),
ScopedVisibilityConstraintId::ALWAYS_TRUE,
ScopedReachabilityConstraintId::ALWAYS_TRUE,
false,
true,
);
@ -542,23 +501,23 @@ mod tests {
#[test]
fn merge() {
let mut narrowing_constraints = NarrowingConstraintsBuilder::default();
let mut visibility_constraints = VisibilityConstraintsBuilder::default();
let mut reachability_constraints = ReachabilityConstraintsBuilder::default();
// merging the same definition with the same constraint keeps the constraint
let mut sym1a = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
let mut sym1a = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym1a.record_binding(
ScopedDefinitionId::from_u32(1),
ScopedVisibilityConstraintId::ALWAYS_TRUE,
ScopedReachabilityConstraintId::ALWAYS_TRUE,
false,
true,
);
let predicate = ScopedPredicateId::from_u32(0).into();
sym1a.record_narrowing_constraint(&mut narrowing_constraints, predicate);
let mut sym1b = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
let mut sym1b = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym1b.record_binding(
ScopedDefinitionId::from_u32(1),
ScopedVisibilityConstraintId::ALWAYS_TRUE,
ScopedReachabilityConstraintId::ALWAYS_TRUE,
false,
true,
);
@ -568,26 +527,26 @@ mod tests {
sym1a.merge(
sym1b,
&mut narrowing_constraints,
&mut visibility_constraints,
&mut reachability_constraints,
);
let mut sym1 = sym1a;
assert_bindings(&narrowing_constraints, &sym1, &["1<0>"]);
// merging the same definition with differing constraints drops all constraints
let mut sym2a = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
let mut sym2a = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym2a.record_binding(
ScopedDefinitionId::from_u32(2),
ScopedVisibilityConstraintId::ALWAYS_TRUE,
ScopedReachabilityConstraintId::ALWAYS_TRUE,
false,
true,
);
let predicate = ScopedPredicateId::from_u32(1).into();
sym2a.record_narrowing_constraint(&mut narrowing_constraints, predicate);
let mut sym1b = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
let mut sym1b = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym1b.record_binding(
ScopedDefinitionId::from_u32(2),
ScopedVisibilityConstraintId::ALWAYS_TRUE,
ScopedReachabilityConstraintId::ALWAYS_TRUE,
false,
true,
);
@ -597,28 +556,28 @@ mod tests {
sym2a.merge(
sym1b,
&mut narrowing_constraints,
&mut visibility_constraints,
&mut reachability_constraints,
);
let sym2 = sym2a;
assert_bindings(&narrowing_constraints, &sym2, &["2<>"]);
// merging a constrained definition with unbound keeps both
let mut sym3a = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
let mut sym3a = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym3a.record_binding(
ScopedDefinitionId::from_u32(3),
ScopedVisibilityConstraintId::ALWAYS_TRUE,
ScopedReachabilityConstraintId::ALWAYS_TRUE,
false,
true,
);
let predicate = ScopedPredicateId::from_u32(3).into();
sym3a.record_narrowing_constraint(&mut narrowing_constraints, predicate);
let sym2b = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
let sym2b = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym3a.merge(
sym2b,
&mut narrowing_constraints,
&mut visibility_constraints,
&mut reachability_constraints,
);
let sym3 = sym3a;
assert_bindings(&narrowing_constraints, &sym3, &["unbound<>", "3<3>"]);
@ -627,7 +586,7 @@ mod tests {
sym1.merge(
sym3,
&mut narrowing_constraints,
&mut visibility_constraints,
&mut reachability_constraints,
);
let sym = sym1;
assert_bindings(&narrowing_constraints, &sym, &["unbound<>", "1<0>", "3<3>"]);
@ -635,24 +594,33 @@ mod tests {
#[test]
fn no_declaration() {
let sym = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
let sym = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
assert_declarations(&sym, &["undeclared"]);
}
#[test]
fn record_declaration() {
let mut sym = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
sym.record_declaration(ScopedDefinitionId::from_u32(1));
let mut sym = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym.record_declaration(
ScopedDefinitionId::from_u32(1),
ScopedReachabilityConstraintId::ALWAYS_TRUE,
);
assert_declarations(&sym, &["1"]);
}
#[test]
fn record_declaration_override() {
let mut sym = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
sym.record_declaration(ScopedDefinitionId::from_u32(1));
sym.record_declaration(ScopedDefinitionId::from_u32(2));
let mut sym = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym.record_declaration(
ScopedDefinitionId::from_u32(1),
ScopedReachabilityConstraintId::ALWAYS_TRUE,
);
sym.record_declaration(
ScopedDefinitionId::from_u32(2),
ScopedReachabilityConstraintId::ALWAYS_TRUE,
);
assert_declarations(&sym, &["2"]);
}
@ -660,17 +628,23 @@ mod tests {
#[test]
fn record_declaration_merge() {
let mut narrowing_constraints = NarrowingConstraintsBuilder::default();
let mut visibility_constraints = VisibilityConstraintsBuilder::default();
let mut sym = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
sym.record_declaration(ScopedDefinitionId::from_u32(1));
let mut reachability_constraints = ReachabilityConstraintsBuilder::default();
let mut sym = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym.record_declaration(
ScopedDefinitionId::from_u32(1),
ScopedReachabilityConstraintId::ALWAYS_TRUE,
);
let mut sym2 = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
sym2.record_declaration(ScopedDefinitionId::from_u32(2));
let mut sym2 = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym2.record_declaration(
ScopedDefinitionId::from_u32(2),
ScopedReachabilityConstraintId::ALWAYS_TRUE,
);
sym.merge(
sym2,
&mut narrowing_constraints,
&mut visibility_constraints,
&mut reachability_constraints,
);
assert_declarations(&sym, &["1", "2"]);
@ -679,16 +653,19 @@ mod tests {
#[test]
fn record_declaration_merge_partial_undeclared() {
let mut narrowing_constraints = NarrowingConstraintsBuilder::default();
let mut visibility_constraints = VisibilityConstraintsBuilder::default();
let mut sym = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
sym.record_declaration(ScopedDefinitionId::from_u32(1));
let mut reachability_constraints = ReachabilityConstraintsBuilder::default();
let mut sym = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym.record_declaration(
ScopedDefinitionId::from_u32(1),
ScopedReachabilityConstraintId::ALWAYS_TRUE,
);
let sym2 = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
let sym2 = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym.merge(
sym2,
&mut narrowing_constraints,
&mut visibility_constraints,
&mut reachability_constraints,
);
assert_declarations(&sym, &["undeclared", "1"]);

4
crates/ty_python_semantic/src/types.rs
View file

@ -7052,6 +7052,10 @@ impl Truthiness {
matches!(self, Truthiness::AlwaysFalse)
}
pub(crate) const fn may_be_true(self) -> bool {
!self.is_always_false()
}
pub(crate) const fn is_always_true(self) -> bool {
matches!(self, Truthiness::AlwaysTrue)
}

28
crates/ty_python_semantic/src/types/class.rs
View file

@ -1637,8 +1637,8 @@ impl<'db> ClassLiteral<'db> {
let method_scope = method_scope_id.to_scope_id(db, file);
let method_map = use_def_map(db, method_scope);
// The attribute assignment inherits the visibility of the method which contains it
let is_method_visible =
// The attribute assignment inherits the reachability of the method which contains it
let is_method_reachable =
if let Some(method_def) = method_scope.node(db).as_function(&module) {
let method = index.expect_single_definition(method_def);
let method_place = class_table.place_id_by_name(&method_def.name).unwrap();
@ -1646,13 +1646,13 @@ impl<'db> ClassLiteral<'db> {
.public_bindings(method_place)
.find_map(|bind| {
(bind.binding.is_defined_and(|def| def == method))
.then(|| class_map.is_binding_visible(db, &bind))
.then(|| class_map.is_binding_reachable(db, &bind))
})
.unwrap_or(Truthiness::AlwaysFalse)
} else {
Truthiness::AlwaysFalse
};
if is_method_visible.is_always_false() {
if is_method_reachable.is_always_false() {
continue;
}
@ -1663,7 +1663,7 @@ impl<'db> ClassLiteral<'db> {
for attribute_assignment in attribute_assignments {
if let DefinitionState::Undefined = attribute_assignment.binding {
// Store the implicit unbound binding here so that we can delay the
// computation of `unbound_visibility` to the point when we actually
// computation of `unbound_reachability` to the point when we actually
// need it. This is an optimization for the common case where the
// `unbound` binding is the only binding of the `name` attribute,
// i.e. if there is no `self.name = …` assignment in this method.
@ -1675,8 +1675,8 @@ impl<'db> ClassLiteral<'db> {
continue;
};
match method_map
.is_binding_visible(db, &attribute_assignment)
.and(is_method_visible)
.is_binding_reachable(db, &attribute_assignment)
.and(is_method_reachable)
{
Truthiness::AlwaysTrue => {
is_attribute_bound = Truthiness::AlwaysTrue;
@ -1691,17 +1691,17 @@ impl<'db> ClassLiteral<'db> {
}
}
// There is at least one attribute assignment that may be visible,
// so if `unbound_visibility` is always false then this attribute is considered bound.
// TODO: this is incomplete logic since the attributes bound after termination are considered visible.
let unbound_visibility = unbound_binding
// There is at least one attribute assignment that may be reachable, so if `unbound_reachability` is
// always false then this attribute is considered bound.
// TODO: this is incomplete logic since the attributes bound after termination are considered reachable.
let unbound_reachability = unbound_binding
.as_ref()
.map(|binding| method_map.is_binding_visible(db, binding))
.map(|binding| method_map.is_binding_reachable(db, binding))
.unwrap_or(Truthiness::AlwaysFalse);
if unbound_visibility
if unbound_reachability
.negate()
.and(is_method_visible)
.and(is_method_reachable)
.is_always_true()
{
is_attribute_bound = Truthiness::AlwaysTrue;

25
crates/ty_python_semantic/src/types/infer.rs
View file

@ -1853,6 +1853,12 @@ impl<'db, 'ast> TypeInferenceBuilder<'db, 'ast> {
function.known(self.db()),
Some(KnownFunction::Overload | KnownFunction::AbstractMethod)
),
Type::Never => {
// In unreachable code, we infer `Never` for decorators like `typing.overload`.
// Return `true` here to avoid false positive `invalid-return-type` lints for
// `@overload`ed functions without a body in unreachable code.
true
}
_ => false,
}
})
@ -1967,7 +1973,7 @@ impl<'db, 'ast> TypeInferenceBuilder<'db, 'ast> {
);
}
let use_def = self.index.use_def_map(scope_id);
if use_def.can_implicit_return(self.db())
if use_def.can_implicitly_return_none(self.db())
&& !Type::none(self.db()).is_assignable_to(self.db(), expected_ty)
{
let no_return = self.return_types_and_ranges.is_empty();
@ -9200,6 +9206,15 @@ impl<'db> TypeInferenceBuilder<'db, '_> {
} = subscript;
match value_ty {
Type::Never => {
// This case can be entered when we use a type annotation like `Literal[1]`
// in unreachable code, since we infer `Never` for `Literal`. We call
// `infer_expression` (instead of `infer_type_expression`) here to avoid
// false-positive `invalid-type-form` diagnostics (`1` is not a valid type
// expression).
self.infer_expression(&subscript.slice);
Type::unknown()
}
Type::ClassLiteral(literal) if literal.is_known(self.db(), KnownClass::Any) => {
if let Some(builder) = self.context.report_lint(&INVALID_TYPE_FORM, subscript) {
builder.into_diagnostic("Type `typing.Any` expected no type parameter");
@ -10303,16 +10318,16 @@ mod tests {
}
/// Test that a symbol known to be unbound in a scope does not still trigger cycle-causing
/// visibility-constraint checks in that scope.
/// reachability-constraint checks in that scope.
#[test]
fn unbound_symbol_no_visibility_constraint_check() {
fn unbound_symbol_no_reachability_constraint_check() {
let mut db = setup_db();
// If the bug we are testing for is not fixed, what happens is that when inferring the
// `flag: bool = True` definitions, we look up `bool` as a deferred name (thus from end of
// scope), and because of the early return its "unbound" binding has a visibility
// scope), and because of the early return its "unbound" binding has a reachability
// constraint of `~flag`, which we evaluate, meaning we have to evaluate the definition of
// `flag` -- and we are in a cycle. With the fix, we short-circuit evaluating visibility
// `flag` -- and we are in a cycle. With the fix, we short-circuit evaluating reachability
// constraints on "unbound" if a symbol is otherwise not bound.
db.write_dedented(
"src/a.py",