mirrors/ruff
1
0
Fork 0
mirror of https://github.com/astral-sh/ruff.git synced 2025-09-30 05:44:56 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

[ty] Reachability constraints (#18621)

Browse source 
## Summary



* Completely removes the concept of visibility constraints. Reachability
constraints are now used to model the static visibility of bindings and
declarations. Reachability constraints are *much* easier to reason about
/ work with, since they are applied at the beginning of a branch, and
not applied retroactively. Removing the duplication between visibility
and reachability constraints also leads to major code simplifications
[^1]. For an overview of how the new constraint system works, see the
updated doc comment in `reachability_constraints.rs`.
* Fixes a [control-flow modeling bug
(panic)](https://github.com/astral-sh/ty/issues/365) involving `break`
statements in loops
* Fixes a [bug where](https://github.com/astral-sh/ty/issues/624) where
`elif` branches would have wrong reachability constraints
* Fixes a [bug where](https://github.com/astral-sh/ty/issues/648) code
after infinite loops would not be considered unreachble
* Fixes a panic on the `pywin32` ecosystem project, which we should be
able to move to `good.txt` once this has been merged.
* Removes some false positives in unreachable code because we infer
`Never` more often, due to the fact that reachability constraints now
apply retroactively to *all* active bindings, not just to bindings
inside a branch.
* As one example, this removes the `division-by-zero` diagnostic from
https://github.com/astral-sh/ty/issues/443 because we now infer `Never`
for the divisor.
* Supersedes and includes similar test changes as
https://github.com/astral-sh/ruff/pull/18392


closes https://github.com/astral-sh/ty/issues/365
closes https://github.com/astral-sh/ty/issues/624
closes https://github.com/astral-sh/ty/issues/642
closes https://github.com/astral-sh/ty/issues/648

## Benchmarks

Benchmarks on black, pandas, and sympy showed that this is neither a
performance improvement, nor a regression.

## Test Plan

Regression tests for:
- [x] https://github.com/astral-sh/ty/issues/365
- [x] https://github.com/astral-sh/ty/issues/624
- [x] https://github.com/astral-sh/ty/issues/642
- [x] https://github.com/astral-sh/ty/issues/648

[^1]: I'm afraid this is something that @carljm advocated for since the
beginning, and I'm not sure anymore why we have never seriously tried
this before. So I suggest we do *not* attempt to do a historical deep
dive to find out exactly why this ever became so complicated, and just
enjoy the fact that we eventually arrived here.

---------

Co-authored-by: Carl Meyer <carl@astral.sh>
This commit is contained in:
David Peter 2025-06-17 09:24:28 +02:00 • committed by GitHub
parent c22f809049
commit 3a77768f79
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
18 changed files with 683 additions and 806 deletions
Download patch file Download diff file Expand all files Collapse all files

136
crates/ty_python_semantic/src/place.rs
View file

@ -755,65 +755,63 @@ fn place_from_bindings_impl<'db>(
requires_explicit_reexport: RequiresExplicitReExport,
) -> Place<'db> {
let predicates = bindings_with_constraints.predicates;
let visibility_constraints = bindings_with_constraints.visibility_constraints;
let reachability_constraints = bindings_with_constraints.reachability_constraints;
let mut bindings_with_constraints = bindings_with_constraints.peekable();
let is_non_exported = |binding: Definition<'db>| {
requires_explicit_reexport.is_yes() && !is_reexported(db, binding)
};
let unbound_visibility_constraint = match bindings_with_constraints.peek() {
let unbound_reachability_constraint = match bindings_with_constraints.peek() {
Some(BindingWithConstraints {
binding,
visibility_constraint,
reachability_constraint,
narrowing_constraint: _,
}) if binding.is_undefined_or(is_non_exported) => Some(*visibility_constraint),
}) if binding.is_undefined_or(is_non_exported) => Some(*reachability_constraint),
_ => None,
};
let mut deleted_visibility = Truthiness::AlwaysFalse;
let mut deleted_reachability = Truthiness::AlwaysFalse;
// Evaluate this lazily because we don't always need it (for example, if there are no visible
// bindings at all, we don't need it), and it can cause us to evaluate visibility constraint
// bindings at all, we don't need it), and it can cause us to evaluate reachability constraint
// expressions, which is extra work and can lead to cycles.
let unbound_visibility = || {
unbound_visibility_constraint
.map(|visibility_constraint| {
visibility_constraints.evaluate(db, predicates, visibility_constraint)
})
.unwrap_or(Truthiness::AlwaysFalse)
let unbound_reachability = || {
unbound_reachability_constraint.map(|reachability_constraint| {
reachability_constraints.evaluate(db, predicates, reachability_constraint)
})
};
let mut types = bindings_with_constraints.filter_map(
|BindingWithConstraints {
binding,
narrowing_constraint,
visibility_constraint,
reachability_constraint,
}| {
let binding =
match binding {
DefinitionState::Defined(binding) => binding,
DefinitionState::Undefined => {
return None;
}
DefinitionState::Deleted => {
deleted_visibility = deleted_visibility.or(
visibility_constraints.evaluate(db, predicates, visibility_constraint)
);
return None;
}
};
let binding = match binding {
DefinitionState::Defined(binding) => binding,
DefinitionState::Undefined => {
return None;
}
DefinitionState::Deleted => {
deleted_reachability = deleted_reachability.or(
reachability_constraints.evaluate(db, predicates, reachability_constraint)
);
return None;
}
};
if is_non_exported(binding) {
return None;
}
let static_visibility =
visibility_constraints.evaluate(db, predicates, visibility_constraint);
let static_reachability =
reachability_constraints.evaluate(db, predicates, reachability_constraint);
if static_visibility.is_always_false() {
// We found a binding that we have statically determined to not be visible from
// the use of the place that we are investigating. There are three interesting
// cases to consider:
if static_reachability.is_always_false() {
// If the static reachability evaluates to false, the binding is either not reachable
// from the start of the scope, or there is no control flow path from that binding to
// the use of the place that we are investigating. There are three interesting cases
// to consider:
//
// ```py
// def f1():
@ -827,35 +825,37 @@ fn place_from_bindings_impl<'db>(
// use(y)
//
// def f3(flag: bool):
// z = 1
// if flag:
// z = 1
// else:
// z = 2
// return
// use(z)
// ```
//
// In the first case, there is a single binding for `x`, and due to the statically
// known `False` condition, it is not visible at the use of `x`. However, we *can*
// see/reach the start of the scope from `use(x)`. This means that `x` is unbound
// and we should return `None`.
// In the first case, there is a single binding for `x`, but it is not reachable from
// the start of the scope. However, the use of `x` is reachable (`unbound_reachability`
// is not always-false). This means that `x` is unbound and we should return `None`.
//
// In the second case, `y` is also not visible at the use of `y`, but here, we can
// not see/reach the start of the scope. There is only one path of control flow,
// and it passes through that binding of `y` (which we can not see). This implies
// that we are in an unreachable section of code. We return `Never` in order to
// silence the `unresolve-reference` diagnostic that would otherwise be emitted at
// the use of `y`.
// In the second case, the binding of `y` is reachable, but there is no control flow
// path from the beginning of the scope, through that binding, to the use of `y` that
// we are investigating. There is also no control flow path from the start of the
// scope, through the implicit `y = <unbound>` binding, to the use of `y`. This means
// that `unbound_reachability` is always false. Since there are no other bindings, no
// control flow path can reach this use of `y`, implying that we are in unreachable
// section of code. We return `Never` in order to silence the `unresolve-reference`
// diagnostic that would otherwise be emitted at the use of `y`.
//
// In the third case, we have two bindings for `z`. The first one is visible, so we
// consider the case that we now encounter the second binding `z = 2`, which is not
// visible due to the early return. We *also* can not see the start of the scope
// from `use(z)` because both paths of control flow pass through a binding of `z`.
// The `z = 1` binding is visible, and so we are *not* in an unreachable section of
// code. However, it is still okay to return `Never` in this case, because we will
// union the types of all bindings, and `Never` will be eliminated automatically.
// In the third case, we have two bindings for `z`. The first one is visible (there
// is a path of control flow from the start of the scope, through that binding, to
// the use of `z`). So we consider the case that we now encounter the second binding
// `z = 2`, which is not visible due to the early return. The `z = <unbound>` binding
// is not live (shadowed by the other bindings), so `unbound_reachability` is `None`.
// Here, we are *not* in an unreachable section of code. However, it is still okay to
// return `Never` in this case, because we will union the types of all bindings, and
// `Never` will be eliminated automatically.
if unbound_visibility().is_always_false() {
// The scope-start is not visible
if unbound_reachability().is_none_or(Truthiness::is_always_false) {
return Some(Type::Never);
}
return None;
@ -867,14 +867,14 @@ fn place_from_bindings_impl<'db>(
);
if let Some(first) = types.next() {
let boundness = match unbound_visibility() {
Truthiness::AlwaysTrue => {
let boundness = match unbound_reachability() {
Some(Truthiness::AlwaysTrue) => {
unreachable!(
"If we have at least one binding, the scope-start should not be definitely visible"
"If we have at least one binding, the implicit `unbound` binding should not be definitely visible"
)
}
Truthiness::AlwaysFalse => Boundness::Bound,
Truthiness::Ambiguous => Boundness::PossiblyUnbound,
Some(Truthiness::AlwaysFalse) | None => Boundness::Bound,
Some(Truthiness::Ambiguous) => Boundness::PossiblyUnbound,
};
let ty = if let Some(second) = types.next() {
@ -882,7 +882,7 @@ fn place_from_bindings_impl<'db>(
} else {
first
};
match deleted_visibility {
match deleted_reachability {
Truthiness::AlwaysFalse => Place::Type(ty, boundness),
Truthiness::AlwaysTrue => Place::Unbound,
Truthiness::Ambiguous => Place::Type(ty, Boundness::PossiblyUnbound),
@ -903,19 +903,19 @@ fn place_from_declarations_impl<'db>(
requires_explicit_reexport: RequiresExplicitReExport,
) -> PlaceFromDeclarationsResult<'db> {
let predicates = declarations.predicates;
let visibility_constraints = declarations.visibility_constraints;
let reachability_constraints = declarations.reachability_constraints;
let mut declarations = declarations.peekable();
let is_non_exported = |declaration: Definition<'db>| {
requires_explicit_reexport.is_yes() && !is_reexported(db, declaration)
};
let undeclared_visibility = match declarations.peek() {
let undeclared_reachability = match declarations.peek() {
Some(DeclarationWithConstraint {
declaration,
visibility_constraint,
reachability_constraint,
}) if declaration.is_undefined_or(is_non_exported) => {
visibility_constraints.evaluate(db, predicates, *visibility_constraint)
reachability_constraints.evaluate(db, predicates, *reachability_constraint)
}
_ => Truthiness::AlwaysFalse,
};
@ -923,7 +923,7 @@ fn place_from_declarations_impl<'db>(
let mut types = declarations.filter_map(
|DeclarationWithConstraint {
declaration,
visibility_constraint,
reachability_constraint,
}| {
let DefinitionState::Defined(declaration) = declaration else {
return None;
@ -933,10 +933,10 @@ fn place_from_declarations_impl<'db>(
return None;
}
let static_visibility =
visibility_constraints.evaluate(db, predicates, visibility_constraint);
let static_reachability =
reachability_constraints.evaluate(db, predicates, reachability_constraint);
if static_visibility.is_always_false() {
if static_reachability.is_always_false() {
None
} else {
Some(declaration_type(db, declaration))
@ -964,10 +964,10 @@ fn place_from_declarations_impl<'db>(
first
};
if conflicting.is_empty() {
let boundness = match undeclared_visibility {
let boundness = match undeclared_reachability {
Truthiness::AlwaysTrue => {
unreachable!(
"If we have at least one declaration, the scope-start should not be definitely visible"
"If we have at least one declaration, the implicit `unbound` binding should not be definitely visible"
)
}
Truthiness::AlwaysFalse => Boundness::Bound,

2
crates/ty_python_semantic/src/semantic_index.rs
View file

@ -33,8 +33,8 @@ pub(crate) mod narrowing_constraints;
pub mod place;
pub(crate) mod predicate;
mod re_exports;
mod reachability_constraints;
mod use_def;
mod visibility_constraints;
pub(crate) use self::use_def::{
BindingWithConstraints, BindingWithConstraintsIterator, DeclarationWithConstraint,

222
crates/ty_python_semantic/src/semantic_index/builder.rs
View file

@ -40,12 +40,12 @@ use crate::semantic_index::predicate::{
StarImportPlaceholderPredicate,
};
use crate::semantic_index::re_exports::exported_names;
use crate::semantic_index::reachability_constraints::{
ReachabilityConstraintsBuilder, ScopedReachabilityConstraintId,
};
use crate::semantic_index::use_def::{
EagerSnapshotKey, FlowSnapshot, ScopedEagerSnapshotId, UseDefMapBuilder,
};
use crate::semantic_index::visibility_constraints::{
ScopedVisibilityConstraintId, VisibilityConstraintsBuilder,
};
use crate::unpack::{Unpack, UnpackKind, UnpackPosition, UnpackValue};
use crate::{Db, Program};
@ -157,7 +157,7 @@ impl<'db, 'ast> SemanticIndexBuilder<'db, 'ast> {
builder.push_scope_with_parent(
NodeWithScopeRef::Module,
None,
ScopedVisibilityConstraintId::ALWAYS_TRUE,
ScopedReachabilityConstraintId::ALWAYS_TRUE,
);
builder
@ -237,7 +237,7 @@ impl<'db, 'ast> SemanticIndexBuilder<'db, 'ast> {
&mut self,
node: NodeWithScopeRef,
parent: Option<FileScopeId>,
reachability: ScopedVisibilityConstraintId,
reachability: ScopedReachabilityConstraintId,
) {
let children_start = self.scopes.next_index() + 1;
@ -354,9 +354,9 @@ impl<'db, 'ast> SemanticIndexBuilder<'db, 'ast> {
&self.use_def_maps[scope_id]
}
fn current_visibility_constraints_mut(&mut self) -> &mut VisibilityConstraintsBuilder {
fn current_reachability_constraints_mut(&mut self) -> &mut ReachabilityConstraintsBuilder {
let scope_id = self.current_scope();
&mut self.use_def_maps[scope_id].visibility_constraints
&mut self.use_def_maps[scope_id].reachability_constraints
}
fn current_ast_ids(&mut self) -> &mut AstIdsBuilder {
@ -576,55 +576,15 @@ impl<'db, 'ast> SemanticIndexBuilder<'db, 'ast> {
id
}
/// Records a previously added visibility constraint by applying it to all live bindings
/// and declarations.
fn record_visibility_constraint_id(&mut self, constraint: ScopedVisibilityConstraintId) {
self.current_use_def_map_mut()
.record_visibility_constraint(constraint);
}
/// Negates the given visibility constraint and then adds it to all live bindings and declarations.
fn record_negated_visibility_constraint(
&mut self,
constraint: ScopedVisibilityConstraintId,
) -> ScopedVisibilityConstraintId {
let id = self
.current_visibility_constraints_mut()
.add_not_constraint(constraint);
self.record_visibility_constraint_id(id);
id
}
/// Records a visibility constraint by applying it to all live bindings and declarations.
fn record_visibility_constraint(
&mut self,
predicate: Predicate<'db>,
) -> ScopedVisibilityConstraintId {
let predicate_id = self.current_use_def_map_mut().add_predicate(predicate);
let id = self
.current_visibility_constraints_mut()
.add_atom(predicate_id);
self.record_visibility_constraint_id(id);
id
}
/// Records that all remaining statements in the current block are unreachable, and therefore
/// not visible.
/// Records that all remaining statements in the current block are unreachable.
fn mark_unreachable(&mut self) {
self.current_use_def_map_mut().mark_unreachable();
}
/// Records a visibility constraint that always evaluates to "ambiguous".
fn record_ambiguous_visibility(&mut self) {
/// Records a reachability constraint that always evaluates to "ambiguous".
fn record_ambiguous_reachability(&mut self) {
self.current_use_def_map_mut()
.record_visibility_constraint(ScopedVisibilityConstraintId::AMBIGUOUS);
}
/// Simplifies (resets) visibility constraints on all live bindings and declarations that did
/// not see any new definitions since the given snapshot.
fn simplify_visibility_constraints(&mut self, snapshot: FlowSnapshot) {
self.current_use_def_map_mut()
.simplify_visibility_constraints(snapshot);
.record_reachability_constraint(ScopedReachabilityConstraintId::AMBIGUOUS);
}
/// Record a constraint that affects the reachability of the current position in the semantic
@ -634,7 +594,7 @@ impl<'db, 'ast> SemanticIndexBuilder<'db, 'ast> {
fn record_reachability_constraint(
&mut self,
predicate: Predicate<'db>,
) -> ScopedVisibilityConstraintId {
) -> ScopedReachabilityConstraintId {
let predicate_id = self.add_predicate(predicate);
self.record_reachability_constraint_id(predicate_id)
}
@ -643,22 +603,22 @@ impl<'db, 'ast> SemanticIndexBuilder<'db, 'ast> {
fn record_reachability_constraint_id(
&mut self,
predicate_id: ScopedPredicateId,
) -> ScopedVisibilityConstraintId {
let visibility_constraint = self
.current_visibility_constraints_mut()
) -> ScopedReachabilityConstraintId {
let reachability_constraint = self
.current_reachability_constraints_mut()
.add_atom(predicate_id);
self.current_use_def_map_mut()
.record_reachability_constraint(visibility_constraint);
visibility_constraint
.record_reachability_constraint(reachability_constraint);
reachability_constraint
}
/// Record the negation of a given reachability/visibility constraint.
/// Record the negation of a given reachability constraint.
fn record_negated_reachability_constraint(
&mut self,
reachability_constraint: ScopedVisibilityConstraintId,
reachability_constraint: ScopedReachabilityConstraintId,
) {
let negated_constraint = self
.current_visibility_constraints_mut()
.current_reachability_constraints_mut()
.add_not_constraint(reachability_constraint);
self.current_use_def_map_mut()
.record_reachability_constraint(negated_constraint);
@ -1139,12 +1099,10 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
let pre_return_state = matches!(last_stmt, ast::Stmt::Return(_))
.then(|| builder.flow_snapshot());
builder.visit_stmt(last_stmt);
let scope_start_visibility =
builder.current_use_def_map().scope_start_visibility;
let reachability = builder.current_use_def_map().reachability;
if let Some(pre_return_state) = pre_return_state {
builder.flow_restore(pre_return_state);
builder.current_use_def_map_mut().scope_start_visibility =
scope_start_visibility;
builder.current_use_def_map_mut().reachability = reachability;
}
}
@ -1297,11 +1255,11 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
continue;
};
// In order to understand the visibility of definitions created by a `*` import,
// we need to know the visibility of the global-scope definitions in the
// In order to understand the reachability of definitions created by a `*` import,
// we need to know the reachability of the global-scope definitions in the
// `referenced_module` the symbols imported from. Much like predicates for `if`
// statements can only have their visibility constraints resolved at type-inference
// time, the visibility of these global-scope definitions in the external module
// statements can only have their reachability constraints resolved at type-inference
// time, the reachability of these global-scope definitions in the external module
// cannot be resolved at this point. As such, we essentially model each definition
// stemming from a `from exporter *` import as something like:
//
@ -1328,7 +1286,7 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
self.current_use_def_map().single_place_snapshot(symbol_id);
self.push_additional_definition(symbol_id, node_ref);
self.current_use_def_map_mut()
.record_and_negate_star_import_visibility_constraint(
.record_and_negate_star_import_reachability_constraint(
star_import,
symbol_id,
pre_definition,
@ -1387,7 +1345,7 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
// reachability constraint on the `msg` expression.
//
// The other important part is the `<halt>`. This lets us skip the usual merging of
// flow states and simplification of visibility constraints, since there is no way
// flow states and simplification of reachability constraints, since there is no way
// of getting out of that `msg` branch. We simply restore to the post-test state.
self.visit_expr(test);
@ -1399,12 +1357,10 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
self.record_narrowing_constraint(negated_predicate);
self.record_reachability_constraint(negated_predicate);
self.visit_expr(msg);
self.record_visibility_constraint(negated_predicate);
self.flow_restore(post_test);
}
self.record_narrowing_constraint(predicate);
self.record_visibility_constraint(predicate);
self.record_reachability_constraint(predicate);
}
@ -1479,13 +1435,10 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
self.visit_expr(&node.test);
let mut no_branch_taken = self.flow_snapshot();
let mut last_predicate = self.record_expression_narrowing_constraint(&node.test);
let mut reachability_constraint =
let mut last_reachability_constraint =
self.record_reachability_constraint(last_predicate);
self.visit_body(&node.body);
let visibility_constraint_id = self.record_visibility_constraint(last_predicate);
let mut vis_constraints = vec![visibility_constraint_id];
let mut post_clauses: Vec<FlowSnapshot> = vec![];
let elif_else_clauses = node
.elif_else_clauses
@ -1509,38 +1462,27 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
// we can only take an elif/else branch if none of the previous ones were
// taken
self.flow_restore(no_branch_taken.clone());
self.record_negated_narrowing_constraint(last_predicate);
self.record_negated_reachability_constraint(reachability_constraint);
let elif_predicate = if let Some(elif_test) = clause_test {
self.record_negated_narrowing_constraint(last_predicate);
self.record_negated_reachability_constraint(last_reachability_constraint);
if let Some(elif_test) = clause_test {
self.visit_expr(elif_test);
// A test expression is evaluated whether the branch is taken or not
no_branch_taken = self.flow_snapshot();
reachability_constraint =
last_predicate = self.record_expression_narrowing_constraint(elif_test);
last_reachability_constraint =
self.record_reachability_constraint(last_predicate);
let predicate = self.record_expression_narrowing_constraint(elif_test);
Some(predicate)
} else {
None
};
}
self.visit_body(clause_body);
for id in &vis_constraints {
self.record_negated_visibility_constraint(*id);
}
if let Some(elif_predicate) = elif_predicate {
last_predicate = elif_predicate;
let id = self.record_visibility_constraint(elif_predicate);
vis_constraints.push(id);
}
}
for post_clause_state in post_clauses {
self.flow_merge(post_clause_state);
}
self.simplify_visibility_constraints(no_branch_taken);
}
ast::Stmt::While(ast::StmtWhile {
test,
@ -1555,57 +1497,49 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
let predicate = self.record_expression_narrowing_constraint(test);
self.record_reachability_constraint(predicate);
// We need multiple copies of the visibility constraint for the while condition,
// We need multiple copies of the reachability constraint for the while condition,
// since we need to model situations where the first evaluation of the condition
// returns True, but a later evaluation returns False.
let first_predicate_id = self.current_use_def_map_mut().add_predicate(predicate);
let later_predicate_id = self.current_use_def_map_mut().add_predicate(predicate);
let first_vis_constraint_id = self
.current_visibility_constraints_mut()
.add_atom(first_predicate_id);
let later_vis_constraint_id = self
.current_visibility_constraints_mut()
.current_reachability_constraints_mut()
.add_atom(later_predicate_id);
// If the body is executed, we know that we've evaluated the condition at least
// once, and that the first evaluation was True. We might not have evaluated the
// condition more than once, so we can't assume that later evaluations were True.
// So the body's full reachability constraint is `first`.
self.record_reachability_constraint_id(first_predicate_id);
let outer_loop = self.push_loop();
self.visit_body(body);
let this_loop = self.pop_loop(outer_loop);
// If the body is executed, we know that we've evaluated the condition at least
// once, and that the first evaluation was True. We might not have evaluated the
// condition more than once, so we can't assume that later evaluations were True.
// So the body's full visibility constraint is `first`.
let body_vis_constraint_id = first_vis_constraint_id;
self.record_visibility_constraint_id(body_vis_constraint_id);
// We execute the `else` once the condition evaluates to false. This could happen
// without ever executing the body, if the condition is false the first time it's
// tested. So the starting flow state of the `else` clause is the union of:
// - the pre-loop state with a visibility constraint that the first evaluation of
// - the pre-loop state with a reachability constraint that the first evaluation of
// the while condition was false,
// - the post-body state (which already has a visibility constraint that the
// first evaluation was true) with a visibility constraint that a _later_
// - the post-body state (which already has a reachability constraint that the
// first evaluation was true) with a reachability constraint that a _later_
// evaluation of the while condition was false.
// To model this correctly, we need two copies of the while condition constraint,
// since the first and later evaluations might produce different results.
let post_body = self.flow_snapshot();
self.flow_restore(pre_loop.clone());
self.record_negated_visibility_constraint(first_vis_constraint_id);
self.flow_restore(pre_loop);
self.flow_merge(post_body);
self.record_negated_narrowing_constraint(predicate);
self.record_negated_reachability_constraint(later_vis_constraint_id);
self.visit_body(orelse);
self.record_negated_visibility_constraint(later_vis_constraint_id);
// Breaking out of a while loop bypasses the `else` clause, so merge in the break
// states after visiting `else`.
for break_state in this_loop.break_states {
let snapshot = self.flow_snapshot();
self.flow_restore(break_state);
self.record_visibility_constraint_id(body_vis_constraint_id);
self.flow_merge(snapshot);
}
self.simplify_visibility_constraints(pre_loop);
}
ast::Stmt::With(ast::StmtWith {
items,
@ -1652,7 +1586,7 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
let iter_expr = self.add_standalone_expression(iter);
self.visit_expr(iter);
self.record_ambiguous_visibility();
self.record_ambiguous_reachability();
let pre_loop = self.flow_snapshot();
@ -1709,10 +1643,15 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
&case.pattern,
case.guard.as_deref(),
);
let vis_constraint_id = self.record_reachability_constraint(match_predicate);
let reachability_constraint =
self.record_reachability_constraint(match_predicate);
let match_success_guard_failure = case.guard.as_ref().map(|guard| {
let guard_expr = self.add_standalone_expression(guard);
// We could also add the guard expression as a reachability constraint, but
// it seems unlikely that both the case predicate as well as the guard are
// statically known conditions, so we currently don't model that.
self.record_ambiguous_reachability();
self.visit_expr(guard);
let post_guard_eval = self.flow_snapshot();
let predicate = Predicate {
@ -1726,8 +1665,6 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
match_success_guard_failure
});
self.record_visibility_constraint_id(vis_constraint_id);
self.visit_body(&case.body);
post_case_snapshots.push(self.flow_snapshot());
@ -1738,6 +1675,7 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
// snapshots into.
self.flow_restore(no_case_matched.clone());
self.record_negated_narrowing_constraint(match_predicate);
self.record_negated_reachability_constraint(reachability_constraint);
if let Some(match_success_guard_failure) = match_success_guard_failure {
self.flow_merge(match_success_guard_failure);
} else {
@ -1748,15 +1686,12 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
debug_assert!(case.guard.is_none());
}
self.record_negated_visibility_constraint(vis_constraint_id);
no_case_matched = self.flow_snapshot();
}
for post_clause_state in post_case_snapshots {
self.flow_merge(post_clause_state);
}
self.simplify_visibility_constraints(no_case_matched);
}
ast::Stmt::Try(ast::StmtTry {
body,
@ -1767,7 +1702,7 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
range: _,
node_index: _,
}) => {
self.record_ambiguous_visibility();
self.record_ambiguous_reachability();
// Save the state prior to visiting any of the `try` block.
//
@ -2136,16 +2071,13 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
let predicate = self.record_expression_narrowing_constraint(test);
let reachability_constraint = self.record_reachability_constraint(predicate);
self.visit_expr(body);
let visibility_constraint = self.record_visibility_constraint(predicate);
let post_body = self.flow_snapshot();
self.flow_restore(pre_if.clone());
self.flow_restore(pre_if);
self.record_negated_narrowing_constraint(predicate);
self.record_negated_reachability_constraint(reachability_constraint);
self.visit_expr(orelse);
self.record_negated_visibility_constraint(visibility_constraint);
self.flow_merge(post_body);
self.simplify_visibility_constraints(pre_if);
}
ast::Expr::ListComp(
list_comprehension @ ast::ExprListComp {
@ -2203,18 +2135,17 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
node_index: _,
op,
}) => {
let pre_op = self.flow_snapshot();
let mut snapshots = vec![];
let mut visibility_constraints = vec![];
let mut reachability_constraints = vec![];
for (index, value) in values.iter().enumerate() {
self.visit_expr(value);
for vid in &visibility_constraints {
self.record_visibility_constraint_id(*vid);
for id in &reachability_constraints {
self.current_use_def_map_mut()
.record_reachability_constraint(*id); // TODO: nicer API
}
self.visit_expr(value);
// For the last value, we don't need to model control flow. There is no short-circuiting
// anymore.
if index < values.len() - 1 {
@ -2223,37 +2154,32 @@ impl<'ast> Visitor<'ast> for SemanticIndexBuilder<'_, 'ast> {
ast::BoolOp::And => self.add_predicate(predicate),
ast::BoolOp::Or => self.add_negated_predicate(predicate),
};
let visibility_constraint = self
.current_visibility_constraints_mut()
let reachability_constraint = self
.current_reachability_constraints_mut()
.add_atom(predicate_id);
let after_expr = self.flow_snapshot();
// We first model the short-circuiting behavior. We take the short-circuit
// path here if all of the previous short-circuit paths were not taken, so
// we record all previously existing visibility constraints, and negate the
// we record all previously existing reachability constraints, and negate the
// one for the current expression.
for vid in &visibility_constraints {
self.record_visibility_constraint_id(*vid);
}
self.record_negated_visibility_constraint(visibility_constraint);
self.record_negated_reachability_constraint(reachability_constraint);
snapshots.push(self.flow_snapshot());
// Then we model the non-short-circuiting behavior. Here, we need to delay
// the application of the visibility constraint until after the expression
// the application of the reachability constraint until after the expression
// has been evaluated, so we only push it onto the stack here.
self.flow_restore(after_expr);
self.record_narrowing_constraint_id(predicate_id);
self.record_reachability_constraint_id(predicate_id);
visibility_constraints.push(visibility_constraint);
reachability_constraints.push(reachability_constraint);
}
}
for snapshot in snapshots {
self.flow_merge(snapshot);
}
self.simplify_visibility_constraints(pre_op);
}
ast::Expr::StringLiteral(_) => {
// Track reachability of string literals, as they could be a stringified annotation

8
crates/ty_python_semantic/src/semantic_index/place.rs
View file

@ -15,7 +15,7 @@ use smallvec::{SmallVec, smallvec};
use crate::Db;
use crate::ast_node_ref::AstNodeRef;
use crate::node_key::NodeKey;
use crate::semantic_index::visibility_constraints::ScopedVisibilityConstraintId;
use crate::semantic_index::reachability_constraints::ScopedReachabilityConstraintId;
use crate::semantic_index::{PlaceSet, SemanticIndex, semantic_index};
#[derive(Debug, Clone, PartialEq, Eq, Hash, salsa::Update)]
@ -480,7 +480,7 @@ pub struct Scope {
parent: Option<FileScopeId>,
node: NodeWithScopeKind,
descendants: Range<FileScopeId>,
reachability: ScopedVisibilityConstraintId,
reachability: ScopedReachabilityConstraintId,
}
impl Scope {
@ -488,7 +488,7 @@ impl Scope {
parent: Option<FileScopeId>,
node: NodeWithScopeKind,
descendants: Range<FileScopeId>,
reachability: ScopedVisibilityConstraintId,
reachability: ScopedReachabilityConstraintId,
) -> Self {
Scope {
parent,
@ -522,7 +522,7 @@ impl Scope {
self.kind().is_eager()
}
pub(crate) fn reachability(&self) -> ScopedVisibilityConstraintId {
pub(crate) fn reachability(&self) -> ScopedReachabilityConstraintId {
self.reachability
}
}

8
crates/ty_python_semantic/src/semantic_index/predicate.rs
View file

@ -4,8 +4,8 @@
//!
//! - [_Narrowing constraints_][crate::semantic_index::narrowing_constraints] constrain the type of
//! a binding that is visible at a particular use.
//! - [_Visibility constraints_][crate::semantic_index::visibility_constraints] determine the
//! static visibility of a binding, and the reachability of a statement.
//! - [_Reachability constraints_][crate::semantic_index::reachability_constraints] determine the
//! static reachability of a binding, and the reachability of a statement or expression.
use ruff_db::files::File;
use ruff_index::{IndexVec, newtype_index};
@ -65,7 +65,7 @@ pub(crate) enum PredicateNode<'db> {
StarImportPlaceholder(StarImportPlaceholderPredicate<'db>),
}
/// Pattern kinds for which we support type narrowing and/or static visibility analysis.
/// Pattern kinds for which we support type narrowing and/or static reachability analysis.
#[derive(Debug, Clone, Hash, PartialEq, salsa::Update)]
pub(crate) enum PatternPredicateKind<'db> {
Singleton(Singleton),
@ -99,7 +99,7 @@ impl<'db> PatternPredicate<'db> {
/// A "placeholder predicate" that is used to model the fact that the boundness of a
/// (possible) definition or declaration caused by a `*` import cannot be fully determined
/// until type-inference time. This is essentially the same as a standard visibility constraint,
/// until type-inference time. This is essentially the same as a standard reachability constraint,
/// so we reuse the [`Predicate`] infrastructure to model it.
///
/// To illustrate, say we have a module `exporter.py` like so:

336
crates/ty_python_semantic/src/semantic_index/visibility_constraints.rs → crates/ty_python_semantic/src/semantic_index/reachability_constraints.rs
View file

@ -1,61 +1,37 @@
//! # Visibility constraints
//! # Reachability constraints
//!
//! During semantic index building, we collect visibility constraints for each binding and
//! declaration. These constraints are then used during type-checking to determine the static
//! visibility of a certain definition. This allows us to re-analyze control flow during type
//! checking, potentially "hiding" some branches that we can statically determine to never be
//! taken. Consider the following example first. We added implicit "unbound" definitions at the
//! start of the scope. Note how visibility constraints can apply to bindings outside of the
//! if-statement:
//! During semantic index building, we record so-called reachability constraints that keep track
//! of a set of conditions that need to apply in order for a certain statement or expression to
//! be reachable from the start of the scope. As an example, consider the following situation where
//! we have just processed two `if`-statements:
//! ```py
//! x = <unbound> # not a live binding for the use of x below, shadowed by `x = 1`
//! y = <unbound> # visibility constraint: ~test
//!
//! x = 1 # visibility constraint: ~test
//! if test:
//! x = 2 # visibility constraint: test
//!
//! y = 2 # visibility constraint: test
//!
//! use(x)
//! use(y)
//! <is this reachable?>
//! ```
//! The static truthiness of the `test` condition can either be always-false, ambiguous, or
//! always-true. Similarly, we have the same three options when evaluating a visibility constraint.
//! This outcome determines the visibility of a definition: always-true means that the definition
//! is definitely visible for a given use, always-false means that the definition is definitely
//! not visible, and ambiguous means that we might see this definition or not. In the latter case,
//! we need to consider both options during type inference and boundness analysis. For the example
//! above, these are the possible type inference / boundness results for the uses of `x` and `y`:
//! In this case, we would record a reachability constraint of `test`, which would later allow us
//! to re-analyze the control flow during type-checking, once we actually know the static truthiness
//! of `test`. When evaluating a constraint, there are three possible outcomes: always true, always
//! false, or ambiguous. For a simple constraint like this, always-true and always-false correspond
//! to the case in which we can infer that the type of `test` is `Literal[True]` or `Literal[False]`.
//! In any other case, like if the type of `test` is `bool` or `Unknown`, we can not statically
//! determine whether `test` is truthy or falsy, so the outcome would be "ambiguous".
//!
//! ```text
//! | `test` truthiness | `~test` truthiness | type of `x` | boundness of `y` |
//! |-------------------|--------------------|-----------------|------------------|
//! | always false | always true | `Literal[1]` | unbound |
//! | ambiguous | ambiguous | `Literal[1, 2]` | possibly unbound |
//! | always true | always false | `Literal[2]` | bound |
//! ```
//!
//! ### Sequential constraints (ternary AND)
//! ## Sequential constraints (ternary AND)
//!
//! As we have seen above, visibility constraints can apply outside of a control flow element.
//! So we need to consider the possibility that multiple constraints apply to the same binding.
//! Here, we consider what happens if multiple `if`-statements lead to a sequence of constraints.
//! Consider the following example:
//! Whenever control flow branches, we record reachability constraints. If we already have a
//! constraint, we create a new one using a ternary AND operation. Consider the following example:
//! ```py
//! x = 0
//!
//! if test1:
//! x = 1
//!
//! if test2:
//! x = 2
//! if test2:
//! <is this reachable?>
//! ```
//! The binding `x = 2` is easy to analyze. Its visibility corresponds to the truthiness of `test2`.
//! For the `x = 1` binding, things are a bit more interesting. It is always visible if `test1` is
//! always-true *and* `test2` is always-false. It is never visible if `test1` is always-false *or*
//! `test2` is always-true. And it is ambiguous otherwise. This corresponds to a ternary *test1 AND
//! ~test2* operation in three-valued Kleene logic [Kleene]:
//! Here, we would accumulate a reachability constraint of `test1 AND test2`. We can statically
//! determine that this position is *always* reachable only if both `test1` and `test2` are
//! always true. On the other hand, we can statically determine that this position is *never*
//! reachable if *either* `test1` or `test2` is always false. In any other case, we can not
//! determine whether this position is reachable or not, so the outcome is "ambiguous". This
//! corresponds to a ternary *AND* operation in [Kleene] logic:
//!
//! ```text
//! | AND | always-false | ambiguous | always-true |
@ -65,38 +41,27 @@
//! | always true | always-false | ambiguous | always-true |
//! ```
//!
//! The `x = 0` binding can be handled similarly, with the difference that both `test1` and `test2`
//! are negated:
//!
//! ## Merged constraints (ternary OR)
//!
//! We also need to consider the case where control flow merges again. Consider a case like this:
//! ```py
//! x = 0 # ~test1 AND ~test2
//!
//! if test1:
//! x = 1 # test1 AND ~test2
//!
//! if test2:
//! x = 2 # test2
//! ```
//!
//! ### Merged constraints (ternary OR)
//!
//! Finally, we consider what happens in "parallel" control flow. Consider the following example
//! where we have omitted the test condition for the outer `if` for clarity:
//! ```py
//! x = 0
//!
//! if <…>:
//! def _():
//! if test1:
//! x = 1
//! else:
//! if test2:
//! x = 2
//! pass
//! elif test2:
//! pass
//! else:
//! return
//!
//! use(x)
//! <is this reachable?>
//! ```
//! At the usage of `x`, i.e. after control flow has been merged again, the visibility of the `x =
//! 0` binding behaves as follows: the binding is always visible if `test1` is always-false *or*
//! `test2` is always-false; and it is never visible if `test1` is always-true *and* `test2` is
//! always-true. This corresponds to a ternary *OR* operation in Kleene logic:
//! Here, the first branch has a `test1` constraint, and the second branch has a `test2` constraint.
//! The third branch ends in a terminal statement [^1]. When we merge control flow, we need to consider
//! the reachability through either the first or the second branch. The current position is only
//! *definitely* unreachable if both `test1` and `test2` are always false. It is definitely
//! reachable if *either* `test1` or `test2` is always true. In any other case, we can not statically
//! determine whether it is reachable or not. This operation corresponds to a ternary *OR* operation:
//!
//! ```text
//! | OR | always-false | ambiguous | always-true |
@ -106,40 +71,95 @@
//! | always true | always-true | always-true | always-true |
//! ```
//!
//! Using this, we can annotate the visibility constraints for the example above:
//! ```py
//! x = 0 # ~test1 OR ~test2
//! [^1]: What's actually happening here is that we merge all three branches using a ternary OR. The
//! third branch has a reachability constraint of `always-false`, and `t OR always-false` is equal
//! to `t` (see first column in that table), so it was okay to omit the third branch in the discussion
//! above.
//!
//! if <…>:
//! if test1:
//! x = 1 # test1
//!
//! ## Negation
//!
//! Control flow elements like `if-elif-else` or `match` statements can also lead to negated
//! constraints. For example, we record a constraint of `~test` for the `else` branch here:
//! ```py
//! if test:
//! pass
//! else:
//! if test2:
//! x = 2 # test2
//! <is this reachable?>
//! ```
//!
//! ## Explicit ambiguity
//!
//! In some cases, we explicitly record an “ambiguous” constraint. We do this when branching on
//! something that we can not (or intentionally do not want to) analyze statically. `for` loops are
//! one example:
//! ```py
//! def _():
//! for _ in range(2):
//! return
//!
//! <is this reachable?>
//! ```
//! If we would not record any constraints at the branching point, we would have an `always-true`
//! reachability for the no-loop branch, and a `always-false` reachability for the branch which enters
//! the loop. Merging those would lead to a reachability of `always-true OR always-false = always-true`,
//! i.e. we would consider the end of the scope to be unconditionally reachable, which is not correct.
//!
//! Recording an ambiguous constraint at the branching point modifies the constraints in both branches to
//! `always-true AND ambiguous = ambiguous` and `always-false AND ambiguous = always-false`, respectively.
//! Merging these two using OR correctly leads to `ambiguous` for the end-of-scope reachability.
//!
//!
//! ## Reachability constraints and bindings
//!
//! To understand how reachability constraints apply to bindings in particular, consider the following
//! example:
//! ```py
//! x = <unbound> # not a live binding for the use of x below, shadowed by `x = 1`
//! y = <unbound> # reachability constraint: ~test
//!
//! x = 1 # reachability constraint: ~test
//! if test:
//! x = 2 # reachability constraint: test
//!
//! y = 2 # reachability constraint: test
//!
//! use(x)
//! use(y)
//! ```
//! Both the type and the boundness of `x` and `y` are affected by reachability constraints:
//!
//! ```text
//! | `test` truthiness | type of `x` | boundness of `y` |
//! |-------------------|-----------------|------------------|
//! | always false | `Literal[1]` | unbound |
//! | ambiguous | `Literal[1, 2]` | possibly unbound |
//! | always true | `Literal[2]` | bound |
//! ```
//!
//! ### Explicit ambiguity
//! To achieve this, we apply reachability constraints retroactively to bindings that came before
//! the branching point. In the example above, the `x = 1` binding has a `test` constraint in the
//! `if` branch, and a `~test` constraint in the implicit `else` branch. Since it is shadowed by
//! `x = 2` in the `if` branch, we are only left with the `~test` constraint after control flow
//! has merged again.
//!
//! In some cases, we explicitly add an “ambiguous” constraint to all bindings
//! in a certain control flow path. We do this when branching on something that we can not (or
//! intentionally do not want to) analyze statically. `for` loops are one example:
//! ```py
//! x = <unbound>
//! For live bindings, the reachability constraint therefore refers to the following question:
//! Is the binding reachable from the start of the scope, and is there a control flow path from
//! that binding to a use of that symbol at the current position?
//!
//! for _ in range(2):
//! x = 1
//! ```
//! Here, we report an ambiguous visibility constraint before branching off. If we don't do this,
//! the `x = <unbound>` binding would be considered unconditionally visible in the no-loop case.
//! And since the other branch does not have the live `x = <unbound>` binding, we would incorrectly
//! create a state where the `x = <unbound>` binding is always visible.
//! In the example above, `x = 1` is always reachable, but that binding can only reach the use of
//! `x` at the current position if `test` is falsy.
//!
//! To handle boundness correctly, we also add implicit `y = <unbound>` bindings at the start of
//! the scope. This allows us to determine whether a symbol is definitely bound (if that implicit
//! `y = <unbound>` binding is not visible), possibly unbound (if the reachability constraint
//! evaluates to `Ambiguous`), or definitely unbound (in case the `y = <unbound>` binding is
//! always visible).
//!
//!
//! ### Representing formulas
//!
//! Given everything above, we can represent a visibility constraint as a _ternary formula_. This
//! Given everything above, we can represent a reachability constraint as a _ternary formula_. This
//! is like a boolean formula (which maps several true/false variables to a single true/false
//! result), but which allows the third "ambiguous" value in addition to "true" and "false".
//!
@ -166,7 +186,7 @@
//! formulas (which have the same outputs for every combination of inputs) are represented by
//! exactly the same graph node. (Because of interning, this is not _equal_ nodes, but _identical_
//! ones.) That means that we can compare formulas for equivalence in constant time, and in
//! particular, can check whether a visibility constraint is statically always true or false,
//! particular, can check whether a reachability constraint is statically always true or false,
//! regardless of any Python program state, by seeing if the constraint's formula is the "true" or
//! "false" leaf node.
//!
@ -204,18 +224,18 @@ use crate::types::{Truthiness, Type, infer_expression_type};
/// for a particular [`Predicate`], if your formula needs to consider how a particular runtime
/// property might be different at different points in the execution of the program.
///
/// Visibility constraints are normalized, so equivalent constraints are guaranteed to have equal
/// reachability constraints are normalized, so equivalent constraints are guaranteed to have equal
/// IDs.
#[derive(Clone, Copy, Eq, Hash, PartialEq)]
pub(crate) struct ScopedVisibilityConstraintId(u32);
pub(crate) struct ScopedReachabilityConstraintId(u32);
impl std::fmt::Debug for ScopedVisibilityConstraintId {
impl std::fmt::Debug for ScopedReachabilityConstraintId {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
let mut f = f.debug_tuple("ScopedVisibilityConstraintId");
let mut f = f.debug_tuple("ScopedReachabilityConstraintId");
match *self {
// We use format_args instead of rendering the strings directly so that we don't get
// any quotes in the output: ScopedVisibilityConstraintId(AlwaysTrue) instead of
// ScopedVisibilityConstraintId("AlwaysTrue").
// any quotes in the output: ScopedReachabilityConstraintId(AlwaysTrue) instead of
// ScopedReachabilityConstraintId("AlwaysTrue").
ALWAYS_TRUE => f.field(&format_args!("AlwaysTrue")),
AMBIGUOUS => f.field(&format_args!("Ambiguous")),
ALWAYS_FALSE => f.field(&format_args!("AlwaysFalse")),
@ -237,34 +257,34 @@ impl std::fmt::Debug for ScopedVisibilityConstraintId {
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
struct InteriorNode {
/// A "variable" that is evaluated as part of a TDD ternary function. For visibility
/// A "variable" that is evaluated as part of a TDD ternary function. For reachability
/// constraints, this is a `Predicate` that represents some runtime property of the Python
/// code that we are evaluating.
atom: ScopedPredicateId,
if_true: ScopedVisibilityConstraintId,
if_ambiguous: ScopedVisibilityConstraintId,
if_false: ScopedVisibilityConstraintId,
if_true: ScopedReachabilityConstraintId,
if_ambiguous: ScopedReachabilityConstraintId,
if_false: ScopedReachabilityConstraintId,
}
impl ScopedVisibilityConstraintId {
impl ScopedReachabilityConstraintId {
/// A special ID that is used for an "always true" / "always visible" constraint.
pub(crate) const ALWAYS_TRUE: ScopedVisibilityConstraintId =
ScopedVisibilityConstraintId(0xffff_ffff);
pub(crate) const ALWAYS_TRUE: ScopedReachabilityConstraintId =
ScopedReachabilityConstraintId(0xffff_ffff);
/// A special ID that is used for an ambiguous constraint.
pub(crate) const AMBIGUOUS: ScopedVisibilityConstraintId =
ScopedVisibilityConstraintId(0xffff_fffe);
pub(crate) const AMBIGUOUS: ScopedReachabilityConstraintId =
ScopedReachabilityConstraintId(0xffff_fffe);
/// A special ID that is used for an "always false" / "never visible" constraint.
pub(crate) const ALWAYS_FALSE: ScopedVisibilityConstraintId =
ScopedVisibilityConstraintId(0xffff_fffd);
pub(crate) const ALWAYS_FALSE: ScopedReachabilityConstraintId =
ScopedReachabilityConstraintId(0xffff_fffd);
fn is_terminal(self) -> bool {
self.0 >= SMALLEST_TERMINAL.0
}
}
impl Idx for ScopedVisibilityConstraintId {
impl Idx for ScopedReachabilityConstraintId {
#[inline]
fn new(value: usize) -> Self {
assert!(value <= (SMALLEST_TERMINAL.0 as usize));
@ -280,35 +300,41 @@ impl Idx for ScopedVisibilityConstraintId {
}
// Rebind some constants locally so that we don't need as many qualifiers below.
const ALWAYS_TRUE: ScopedVisibilityConstraintId = ScopedVisibilityConstraintId::ALWAYS_TRUE;
const AMBIGUOUS: ScopedVisibilityConstraintId = ScopedVisibilityConstraintId::AMBIGUOUS;
const ALWAYS_FALSE: ScopedVisibilityConstraintId = ScopedVisibilityConstraintId::ALWAYS_FALSE;
const SMALLEST_TERMINAL: ScopedVisibilityConstraintId = ALWAYS_FALSE;
const ALWAYS_TRUE: ScopedReachabilityConstraintId = ScopedReachabilityConstraintId::ALWAYS_TRUE;
const AMBIGUOUS: ScopedReachabilityConstraintId = ScopedReachabilityConstraintId::AMBIGUOUS;
const ALWAYS_FALSE: ScopedReachabilityConstraintId = ScopedReachabilityConstraintId::ALWAYS_FALSE;
const SMALLEST_TERMINAL: ScopedReachabilityConstraintId = ALWAYS_FALSE;
/// A collection of visibility constraints for a given scope.
/// A collection of reachability constraints for a given scope.
#[derive(Debug, PartialEq, Eq, salsa::Update)]
pub(crate) struct VisibilityConstraints {
interiors: IndexVec<ScopedVisibilityConstraintId, InteriorNode>,
pub(crate) struct ReachabilityConstraints {
interiors: IndexVec<ScopedReachabilityConstraintId, InteriorNode>,
}
#[derive(Debug, Default, PartialEq, Eq)]
pub(crate) struct VisibilityConstraintsBuilder {
interiors: IndexVec<ScopedVisibilityConstraintId, InteriorNode>,
interior_cache: FxHashMap<InteriorNode, ScopedVisibilityConstraintId>,
not_cache: FxHashMap<ScopedVisibilityConstraintId, ScopedVisibilityConstraintId>,
pub(crate) struct ReachabilityConstraintsBuilder {
interiors: IndexVec<ScopedReachabilityConstraintId, InteriorNode>,
interior_cache: FxHashMap<InteriorNode, ScopedReachabilityConstraintId>,
not_cache: FxHashMap<ScopedReachabilityConstraintId, ScopedReachabilityConstraintId>,
and_cache: FxHashMap<
(ScopedVisibilityConstraintId, ScopedVisibilityConstraintId),
ScopedVisibilityConstraintId,
(
ScopedReachabilityConstraintId,
ScopedReachabilityConstraintId,
),
ScopedReachabilityConstraintId,
>,
or_cache: FxHashMap<
(ScopedVisibilityConstraintId, ScopedVisibilityConstraintId),
ScopedVisibilityConstraintId,
(
ScopedReachabilityConstraintId,
ScopedReachabilityConstraintId,
),
ScopedReachabilityConstraintId,
>,
}
impl VisibilityConstraintsBuilder {
pub(crate) fn build(self) -> VisibilityConstraints {
VisibilityConstraints {
impl ReachabilityConstraintsBuilder {
pub(crate) fn build(self) -> ReachabilityConstraints {
ReachabilityConstraints {
interiors: self.interiors,
}
}
@ -318,8 +344,8 @@ impl VisibilityConstraintsBuilder {
/// any internal node, since they are leaf nodes.
fn cmp_atoms(
&self,
a: ScopedVisibilityConstraintId,
b: ScopedVisibilityConstraintId,
a: ScopedReachabilityConstraintId,
b: ScopedReachabilityConstraintId,
) -> Ordering {
if a == b || (a.is_terminal() && b.is_terminal()) {
Ordering::Equal
@ -332,9 +358,9 @@ impl VisibilityConstraintsBuilder {
}
}
/// Adds an interior node, ensuring that we always use the same visibility constraint ID for
/// Adds an interior node, ensuring that we always use the same reachability constraint ID for
/// equal nodes.
fn add_interior(&mut self, node: InteriorNode) -> ScopedVisibilityConstraintId {
fn add_interior(&mut self, node: InteriorNode) -> ScopedReachabilityConstraintId {
// If the true and false branches lead to the same node, we can override the ambiguous
// branch to go there too. And this node is then redundant and can be reduced.
if node.if_true == node.if_false {
@ -347,7 +373,7 @@ impl VisibilityConstraintsBuilder {
.or_insert_with(|| self.interiors.push(node))
}
/// Adds a new visibility constraint that checks a single [`Predicate`].
/// Adds a new reachability constraint that checks a single [`Predicate`].
///
/// [`ScopedPredicateId`]s are the “variables” that are evaluated by a TDD. A TDD variable has
/// the same value no matter how many times it appears in the ternary formula that the TDD
@ -361,7 +387,7 @@ impl VisibilityConstraintsBuilder {
pub(crate) fn add_atom(
&mut self,
predicate: ScopedPredicateId,
) -> ScopedVisibilityConstraintId {
) -> ScopedReachabilityConstraintId {
self.add_interior(InteriorNode {
atom: predicate,
if_true: ALWAYS_TRUE,
@ -370,11 +396,11 @@ impl VisibilityConstraintsBuilder {
})
}
/// Adds a new visibility constraint that is the ternary NOT of an existing one.
/// Adds a new reachability constraint that is the ternary NOT of an existing one.
pub(crate) fn add_not_constraint(
&mut self,
a: ScopedVisibilityConstraintId,
) -> ScopedVisibilityConstraintId {
a: ScopedReachabilityConstraintId,
) -> ScopedReachabilityConstraintId {
if a == ALWAYS_TRUE {
return ALWAYS_FALSE;
} else if a == AMBIGUOUS {
@ -400,12 +426,12 @@ impl VisibilityConstraintsBuilder {
result
}
/// Adds a new visibility constraint that is the ternary OR of two existing ones.
/// Adds a new reachability constraint that is the ternary OR of two existing ones.
pub(crate) fn add_or_constraint(
&mut self,
a: ScopedVisibilityConstraintId,
b: ScopedVisibilityConstraintId,
) -> ScopedVisibilityConstraintId {
a: ScopedReachabilityConstraintId,
b: ScopedReachabilityConstraintId,
) -> ScopedReachabilityConstraintId {
match (a, b) {
(ALWAYS_TRUE, _) | (_, ALWAYS_TRUE) => return ALWAYS_TRUE,
(ALWAYS_FALSE, other) | (other, ALWAYS_FALSE) => return other,
@ -466,12 +492,12 @@ impl VisibilityConstraintsBuilder {
result
}
/// Adds a new visibility constraint that is the ternary AND of two existing ones.
/// Adds a new reachability constraint that is the ternary AND of two existing ones.
pub(crate) fn add_and_constraint(
&mut self,
a: ScopedVisibilityConstraintId,
b: ScopedVisibilityConstraintId,
) -> ScopedVisibilityConstraintId {
a: ScopedReachabilityConstraintId,
b: ScopedReachabilityConstraintId,
) -> ScopedReachabilityConstraintId {
match (a, b) {
(ALWAYS_FALSE, _) | (_, ALWAYS_FALSE) => return ALWAYS_FALSE,
(ALWAYS_TRUE, other) | (other, ALWAYS_TRUE) => return other,
@ -533,13 +559,13 @@ impl VisibilityConstraintsBuilder {
}
}
impl VisibilityConstraints {
/// Analyze the statically known visibility for a given visibility constraint.
impl ReachabilityConstraints {
/// Analyze the statically known reachability for a given constraint.
pub(crate) fn evaluate<'db>(
&self,
db: &'db dyn Db,
predicates: &Predicates<'db>,
mut id: ScopedVisibilityConstraintId,
mut id: ScopedReachabilityConstraintId,
) -> Truthiness {
loop {
let node = match id {

332
crates/ty_python_semantic/src/semantic_index/use_def.rs
View file

@ -221,43 +221,8 @@
//! snapshot (which has `x = 3` as the only live binding). The result of this merge is that we now
//! have two live bindings of `x`: `x = 3` and `x = 4`.
//!
//! Another piece of information that the `UseDefMap` needs to provide are visibility constraints.
//! These are similar to the narrowing constraints, but apply to bindings and declarations within a
//! control flow path. Consider the following example:
//! ```py
//! x = 1
//! if test:
//! x = 2
//! y = "y"
//! ```
//! In principle, there are two possible control flow paths here. However, if we can statically
//! infer `test` to be always truthy or always falsy (that is, `__bool__` of `test` is of type
//! `Literal[True]` or `Literal[False]`), we can rule out one of the possible paths. To support
//! this feature, we record a visibility constraint of `test` to all live bindings and declarations
//! *after* visiting the body of the `if` statement. And we record a negative visibility constraint
//! `~test` to all live bindings/declarations in the (implicit) `else` branch. For the example
//! above, we would record the following visibility constraints (adding the implicit "unbound"
//! definitions for clarity):
//! ```py
//! x = <unbound> # not live, shadowed by `x = 1`
//! y = <unbound> # visibility constraint: ~test
//!
//! x = 1 # visibility constraint: ~test
//! if test:
//! x = 2 # visibility constraint: test
//! y = "y" # visibility constraint: test
//! ```
//! When we encounter a use of `x` after this `if` statement, we would record two live bindings: `x
//! = 1` with a constraint of `~test`, and `x = 2` with a constraint of `test`. In type inference,
//! when we iterate over all live bindings, we can evaluate these constraints to determine if a
//! particular binding is actually visible. For example, if `test` is always truthy, we only see
//! the `x = 2` binding. If `test` is always falsy, we only see the `x = 1` binding. And if the
//! `__bool__` method of `test` returns type `bool`, we can see both bindings.
//!
//! Note that we also record visibility constraints for the start of the scope. This is important
//! to determine if a place is definitely bound, possibly unbound, or definitely unbound. In the
//! example above, The `y = <unbound>` binding is constrained by `~test`, so `y` would only be
//! definitely-bound if `test` is always truthy.
//! Another piece of information that the `UseDefMap` needs to provide are reachability constraints.
//! See [`reachability_constraints.rs`] for more details, in particular how they apply to bindings.
//!
//! The [`UseDefMapBuilder`] itself just exposes methods for taking a snapshot, resetting to a
//! snapshot, and merging a snapshot into the current state. The logic using these methods lives in
@ -282,8 +247,8 @@ use crate::semantic_index::place::{FileScopeId, PlaceExpr, ScopeKind, ScopedPlac
use crate::semantic_index::predicate::{
Predicate, Predicates, PredicatesBuilder, ScopedPredicateId, StarImportPlaceholderPredicate,
};
use crate::semantic_index::visibility_constraints::{
ScopedVisibilityConstraintId, VisibilityConstraints, VisibilityConstraintsBuilder,
use crate::semantic_index::reachability_constraints::{
ReachabilityConstraints, ReachabilityConstraintsBuilder, ScopedReachabilityConstraintId,
};
use crate::types::{IntersectionBuilder, Truthiness, Type, infer_narrowing_constraint};
@ -302,14 +267,14 @@ pub(crate) struct UseDefMap<'db> {
/// Array of narrowing constraints in this scope.
narrowing_constraints: NarrowingConstraints,
/// Array of visibility constraints in this scope.
visibility_constraints: VisibilityConstraints,
/// Array of reachability constraints in this scope.
reachability_constraints: ReachabilityConstraints,
/// [`Bindings`] reaching a [`ScopedUseId`].
bindings_by_use: IndexVec<ScopedUseId, Bindings>,
/// Tracks whether or not a given AST node is reachable from the start of the scope.
node_reachability: FxHashMap<NodeKey, ScopedVisibilityConstraintId>,
node_reachability: FxHashMap<NodeKey, ScopedReachabilityConstraintId>,
/// If the definition is a binding (only) -- `x = 1` for example -- then we need
/// [`Declarations`] to know whether this binding is permitted by the live declarations.
@ -335,20 +300,24 @@ pub(crate) struct UseDefMap<'db> {
/// eager scope.
eager_snapshots: EagerSnapshots,
/// Whether or not the start of the scope is visible.
/// Whether or not the end of the scope is reachable.
///
/// This is used to check if the function can implicitly return `None`.
/// For example:
///
/// ```python
/// def f(cond: bool) -> int:
/// ```py
/// def f(cond: bool) -> int | None:
/// if cond:
/// return 1
///
/// def g() -> int:
/// if True:
/// return 1
/// ```
///
/// In this case, the function may implicitly return `None`.
/// Function `f` may implicitly return `None`, but `g` cannot.
///
/// This is used by `UseDefMap::can_implicit_return`.
scope_start_visibility: ScopedVisibilityConstraintId,
/// This is used by [`UseDefMap::can_implicitly_return_none`].
end_of_scope_reachability: ScopedReachabilityConstraintId,
}
impl<'db> UseDefMap<'db> {
@ -378,12 +347,11 @@ impl<'db> UseDefMap<'db> {
pub(super) fn is_reachable(
&self,
db: &dyn crate::Db,
reachability: ScopedVisibilityConstraintId,
reachability: ScopedReachabilityConstraintId,
) -> bool {
!self
.visibility_constraints
self.reachability_constraints
.evaluate(db, &self.predicates, reachability)
.is_always_false()
.may_be_true()
}
/// Check whether or not a given expression is reachable from the start of the scope. This
@ -392,8 +360,8 @@ impl<'db> UseDefMap<'db> {
/// analysis.
#[track_caller]
pub(super) fn is_node_reachable(&self, db: &dyn crate::Db, node_key: NodeKey) -> bool {
!self
.visibility_constraints
self
.reachability_constraints
.evaluate(
db,
&self.predicates,
@ -402,7 +370,7 @@ impl<'db> UseDefMap<'db> {
.get(&node_key)
.expect("`is_node_reachable` should only be called on AST nodes with recorded reachability"),
)
.is_always_false()
.may_be_true()
}
pub(crate) fn public_bindings(
@ -467,20 +435,23 @@ impl<'db> UseDefMap<'db> {
}
/// This function is intended to be called only once inside `TypeInferenceBuilder::infer_function_body`.
pub(crate) fn can_implicit_return(&self, db: &dyn crate::Db) -> bool {
pub(crate) fn can_implicitly_return_none(&self, db: &dyn crate::Db) -> bool {
!self
.visibility_constraints
.evaluate(db, &self.predicates, self.scope_start_visibility)
.reachability_constraints
.evaluate(db, &self.predicates, self.end_of_scope_reachability)
.is_always_false()
}
pub(crate) fn is_binding_visible(
pub(crate) fn is_binding_reachable(
&self,
db: &dyn crate::Db,
binding: &BindingWithConstraints<'_, 'db>,
) -> Truthiness {
self.visibility_constraints
.evaluate(db, &self.predicates, binding.visibility_constraint)
self.reachability_constraints.evaluate(
db,
&self.predicates,
binding.reachability_constraint,
)
}
fn bindings_iterator<'map>(
@ -491,7 +462,7 @@ impl<'db> UseDefMap<'db> {
all_definitions: &self.all_definitions,
predicates: &self.predicates,
narrowing_constraints: &self.narrowing_constraints,
visibility_constraints: &self.visibility_constraints,
reachability_constraints: &self.reachability_constraints,
inner: bindings.iter(),
}
}
@ -503,7 +474,7 @@ impl<'db> UseDefMap<'db> {
DeclarationsIterator {
all_definitions: &self.all_definitions,
predicates: &self.predicates,
visibility_constraints: &self.visibility_constraints,
reachability_constraints: &self.reachability_constraints,
inner: declarations.iter(),
}
}
@ -538,7 +509,7 @@ pub(crate) struct BindingWithConstraintsIterator<'map, 'db> {
all_definitions: &'map IndexVec<ScopedDefinitionId, DefinitionState<'db>>,
pub(crate) predicates: &'map Predicates<'db>,
pub(crate) narrowing_constraints: &'map NarrowingConstraints,
pub(crate) visibility_constraints: &'map VisibilityConstraints,
pub(crate) reachability_constraints: &'map ReachabilityConstraints,
inner: LiveBindingsIterator<'map>,
}
@ -558,7 +529,7 @@ impl<'map, 'db> Iterator for BindingWithConstraintsIterator<'map, 'db> {
constraint_ids: narrowing_constraints
.iter_predicates(live_binding.narrowing_constraint),
},
visibility_constraint: live_binding.visibility_constraint,
reachability_constraint: live_binding.reachability_constraint,
})
}
}
@ -568,7 +539,7 @@ impl std::iter::FusedIterator for BindingWithConstraintsIterator<'_, '_> {}
pub(crate) struct BindingWithConstraints<'map, 'db> {
pub(crate) binding: DefinitionState<'db>,
pub(crate) narrowing_constraint: ConstraintsIterator<'map, 'db>,
pub(crate) visibility_constraint: ScopedVisibilityConstraintId,
pub(crate) reachability_constraint: ScopedReachabilityConstraintId,
}
pub(crate) struct ConstraintsIterator<'map, 'db> {
@ -618,13 +589,13 @@ impl<'db> ConstraintsIterator<'_, 'db> {
pub(crate) struct DeclarationsIterator<'map, 'db> {
all_definitions: &'map IndexVec<ScopedDefinitionId, DefinitionState<'db>>,
pub(crate) predicates: &'map Predicates<'db>,
pub(crate) visibility_constraints: &'map VisibilityConstraints,
pub(crate) reachability_constraints: &'map ReachabilityConstraints,
inner: LiveDeclarationsIterator<'map>,
}
pub(crate) struct DeclarationWithConstraint<'db> {
pub(crate) declaration: DefinitionState<'db>,
pub(crate) visibility_constraint: ScopedVisibilityConstraintId,
pub(crate) reachability_constraint: ScopedReachabilityConstraintId,
}
impl<'db> Iterator for DeclarationsIterator<'_, 'db> {
@ -634,11 +605,11 @@ impl<'db> Iterator for DeclarationsIterator<'_, 'db> {
self.inner.next().map(
|LiveDeclaration {
declaration,
visibility_constraint,
reachability_constraint,
}| {
DeclarationWithConstraint {
declaration: self.all_definitions[*declaration],
visibility_constraint: *visibility_constraint,
reachability_constraint: *reachability_constraint,
}
},
)
@ -651,8 +622,7 @@ impl std::iter::FusedIterator for DeclarationsIterator<'_, '_> {}
#[derive(Clone, Debug)]
pub(super) struct FlowSnapshot {
place_states: IndexVec<ScopedPlaceId, PlaceState>,
scope_start_visibility: ScopedVisibilityConstraintId,
reachability: ScopedVisibilityConstraintId,
reachability: ScopedReachabilityConstraintId,
}
#[derive(Debug)]
@ -666,60 +636,18 @@ pub(super) struct UseDefMapBuilder<'db> {
/// Builder of narrowing constraints.
pub(super) narrowing_constraints: NarrowingConstraintsBuilder,
/// Builder of visibility constraints.
pub(super) visibility_constraints: VisibilityConstraintsBuilder,
/// A constraint which describes the visibility of the unbound/undeclared state, i.e.
/// whether or not a use of a place at the current point in control flow would see
/// the fake `x = <unbound>` binding at the start of the scope. This is important for
/// cases like the following, where we need to hide the implicit unbound binding in
/// the "else" branch:
/// ```py
/// # x = <unbound>
///
/// if True:
/// x = 1
///
/// use(x) # the `x = <unbound>` binding is not visible here
/// ```
pub(super) scope_start_visibility: ScopedVisibilityConstraintId,
/// Builder of reachability constraints.
pub(super) reachability_constraints: ReachabilityConstraintsBuilder,
/// Live bindings at each so-far-recorded use.
bindings_by_use: IndexVec<ScopedUseId, Bindings>,
/// Tracks whether or not the scope start is visible at the current point in control flow.
/// This is subtly different from `scope_start_visibility`, as we apply these constraints
/// at the beginning of a branch. Visibility constraints, on the other hand, need to be
/// applied at the end of a branch, as we apply them retroactively to all live bindings:
/// ```py
/// y = 1
///
/// if test:
/// # we record a reachability constraint of [test] here,
/// # so that it can affect the use of `x`:
///
/// x # we store a reachability constraint of [test] for this use of `x`
///
/// y = 2
///
/// # we record a visibility constraint of [test] here, which retroactively affects
/// # the `y = 1` and the `y = 2` binding.
/// else:
/// # we record a reachability constraint of [~test] here.
///
/// pass
///
/// # we record a visibility constraint of [~test] here, which retroactively affects
/// # the `y = 1` binding.
///
/// use(y)
/// ```
/// Depending on the value of `test`, the `y = 1`, `y = 2`, or both bindings may be visible.
/// The use of `x` is recorded with a reachability constraint of `[test]`.
pub(super) reachability: ScopedVisibilityConstraintId,
/// Tracks whether or not the current point in control flow is reachable from the
/// start of the scope.
pub(super) reachability: ScopedReachabilityConstraintId,
/// Tracks whether or not a given AST node is reachable from the start of the scope.
node_reachability: FxHashMap<NodeKey, ScopedVisibilityConstraintId>,
node_reachability: FxHashMap<NodeKey, ScopedReachabilityConstraintId>,
/// Live declarations for each so-far-recorded binding.
declarations_by_binding: FxHashMap<Definition<'db>, Declarations>,
@ -744,10 +672,9 @@ impl<'db> UseDefMapBuilder<'db> {
all_definitions: IndexVec::from_iter([DefinitionState::Undefined]),
predicates: PredicatesBuilder::default(),
narrowing_constraints: NarrowingConstraintsBuilder::default(),
visibility_constraints: VisibilityConstraintsBuilder::default(),
scope_start_visibility: ScopedVisibilityConstraintId::ALWAYS_TRUE,
reachability_constraints: ReachabilityConstraintsBuilder::default(),
bindings_by_use: IndexVec::new(),
reachability: ScopedVisibilityConstraintId::ALWAYS_TRUE,
reachability: ScopedReachabilityConstraintId::ALWAYS_TRUE,
node_reachability: FxHashMap::default(),
declarations_by_binding: FxHashMap::default(),
bindings_by_declaration: FxHashMap::default(),
@ -757,14 +684,20 @@ impl<'db> UseDefMapBuilder<'db> {
}
}
pub(super) fn mark_unreachable(&mut self) {
self.record_visibility_constraint(ScopedVisibilityConstraintId::ALWAYS_FALSE);
self.reachability = ScopedVisibilityConstraintId::ALWAYS_FALSE;
self.reachability = ScopedReachabilityConstraintId::ALWAYS_FALSE;
for state in &mut self.place_states {
state.record_reachability_constraint(
&mut self.reachability_constraints,
ScopedReachabilityConstraintId::ALWAYS_FALSE,
);
}
}
pub(super) fn add_place(&mut self, place: ScopedPlaceId) {
let new_place = self
.place_states
.push(PlaceState::undefined(self.scope_start_visibility));
.push(PlaceState::undefined(self.reachability));
debug_assert_eq!(place, new_place);
}
@ -780,7 +713,7 @@ impl<'db> UseDefMapBuilder<'db> {
.insert(binding, place_state.declarations().clone());
place_state.record_binding(
def_id,
self.scope_start_visibility,
self.reachability,
self.is_class_scope,
is_place_name,
);
@ -798,136 +731,85 @@ impl<'db> UseDefMapBuilder<'db> {
}
}
pub(super) fn record_visibility_constraint(
&mut self,
constraint: ScopedVisibilityConstraintId,
) {
for state in &mut self.place_states {
state.record_visibility_constraint(&mut self.visibility_constraints, constraint);
}
self.scope_start_visibility = self
.visibility_constraints
.add_and_constraint(self.scope_start_visibility, constraint);
}
/// Snapshot the state of a single place at the current point in control flow.
///
/// This is only used for `*`-import visibility constraints, which are handled differently
/// to most other visibility constraints. See the doc-comment for
/// [`Self::record_and_negate_star_import_visibility_constraint`] for more details.
/// This is only used for `*`-import reachability constraints, which are handled differently
/// to most other reachability constraints. See the doc-comment for
/// [`Self::record_and_negate_star_import_reachability_constraint`] for more details.
pub(super) fn single_place_snapshot(&self, place: ScopedPlaceId) -> PlaceState {
self.place_states[place].clone()
}
/// This method exists solely for handling `*`-import visibility constraints.
/// This method exists solely for handling `*`-import reachability constraints.
///
/// The reason why we add visibility constraints for [`Definition`]s created by `*` imports
/// The reason why we add reachability constraints for [`Definition`]s created by `*` imports
/// is laid out in the doc-comment for [`StarImportPlaceholderPredicate`]. But treating these
/// visibility constraints in the use-def map the same way as all other visibility constraints
/// reachability constraints in the use-def map the same way as all other reachability constraints
/// was shown to lead to [significant regressions] for small codebases where typeshed
/// dominates. (Although `*` imports are not common generally, they are used in several
/// important places by typeshed.)
///
/// To solve these regressions, it was observed that we could do significantly less work for
/// `*`-import definitions. We do a number of things differently here to our normal handling of
/// visibility constraints:
/// reachability constraints:
///
/// - We only apply and negate the visibility constraints to a single symbol, rather than to
/// - We only apply and negate the reachability constraints to a single symbol, rather than to
/// all symbols. This is possible here because, unlike most definitions, we know in advance that
/// exactly one definition occurs inside the "if-true" predicate branch, and we know exactly
/// which definition it is.
///
/// Doing things this way is cheaper in and of itself. However, it also allows us to avoid
/// calling [`Self::simplify_visibility_constraints`] after the constraint has been applied to
/// the "if-predicate-true" branch and negated for the "if-predicate-false" branch. Simplifying
/// the visibility constraints is only important for places that did not have any new
/// definitions inside either the "if-predicate-true" branch or the "if-predicate-false" branch.
///
/// - We only snapshot the state for a single place prior to the definition, rather than doing
/// expensive calls to [`Self::snapshot`]. Again, this is possible because we know
/// that only a single definition occurs inside the "if-predicate-true" predicate branch.
///
/// - Normally we take care to check whether an "if-predicate-true" branch or an
/// "if-predicate-false" branch contains a terminal statement: these can affect the visibility
/// "if-predicate-false" branch contains a terminal statement: these can affect the reachability
/// of symbols defined inside either branch. However, in the case of `*`-import definitions,
/// this is unnecessary (and therefore not done in this method), since we know that a `*`-import
/// predicate cannot create a terminal statement inside either branch.
///
/// [significant regressions]: https://github.com/astral-sh/ruff/pull/17286#issuecomment-2786755746
pub(super) fn record_and_negate_star_import_visibility_constraint(
pub(super) fn record_and_negate_star_import_reachability_constraint(
&mut self,
star_import: StarImportPlaceholderPredicate<'db>,
symbol: ScopedPlaceId,
pre_definition_state: PlaceState,
) {
let predicate_id = self.add_predicate(star_import.into());
let visibility_id = self.visibility_constraints.add_atom(predicate_id);
let negated_visibility_id = self
.visibility_constraints
.add_not_constraint(visibility_id);
let reachability_id = self.reachability_constraints.add_atom(predicate_id);
let negated_reachability_id = self
.reachability_constraints
.add_not_constraint(reachability_id);
let mut post_definition_state =
std::mem::replace(&mut self.place_states[symbol], pre_definition_state);
post_definition_state
.record_visibility_constraint(&mut self.visibility_constraints, visibility_id);
.record_reachability_constraint(&mut self.reachability_constraints, reachability_id);
self.place_states[symbol]
.record_visibility_constraint(&mut self.visibility_constraints, negated_visibility_id);
self.place_states[symbol].record_reachability_constraint(
&mut self.reachability_constraints,
negated_reachability_id,
);
self.place_states[symbol].merge(
post_definition_state,
&mut self.narrowing_constraints,
&mut self.visibility_constraints,
&mut self.reachability_constraints,
);
}
/// This method resets the visibility constraints for all places to a previous state
/// *if* there have been no new declarations or bindings since then. Consider the
/// following example:
/// ```py
/// x = 0
/// y = 0
/// if test_a:
/// y = 1
/// elif test_b:
/// y = 2
/// elif test_c:
/// y = 3
///
/// # RESET
/// ```
/// We build a complex visibility constraint for the `y = 0` binding. We build the same
/// constraint for the `x = 0` binding as well, but at the `RESET` point, we can get rid
/// of it, as the `if`-`elif`-`elif` chain doesn't include any new bindings of `x`.
pub(super) fn simplify_visibility_constraints(&mut self, snapshot: FlowSnapshot) {
debug_assert!(self.place_states.len() >= snapshot.place_states.len());
// If there are any control flow paths that have become unreachable between `snapshot` and
// now, then it's not valid to simplify any visibility constraints to `snapshot`.
if self.scope_start_visibility != snapshot.scope_start_visibility {
return;
}
// Note that this loop terminates when we reach a place not present in the snapshot.
// This means we keep visibility constraints for all new places, which is intended,
// since these places have been introduced in the corresponding branch, which might
// be subject to visibility constraints. We only simplify/reset visibility constraints
// for places that have the same bindings and declarations present compared to the
// snapshot.
for (current, snapshot) in self.place_states.iter_mut().zip(snapshot.place_states) {
current.simplify_visibility_constraints(snapshot);
}
}
pub(super) fn record_reachability_constraint(
&mut self,
constraint: ScopedVisibilityConstraintId,
) -> ScopedVisibilityConstraintId {
constraint: ScopedReachabilityConstraintId,
) {
self.reachability = self
.visibility_constraints
.reachability_constraints
.add_and_constraint(self.reachability, constraint);
self.reachability
for state in &mut self.place_states {
state.record_reachability_constraint(&mut self.reachability_constraints, constraint);
}
}
pub(super) fn record_declaration(
@ -941,7 +823,7 @@ impl<'db> UseDefMapBuilder<'db> {
let place_state = &mut self.place_states[place];
self.bindings_by_declaration
.insert(declaration, place_state.bindings().clone());
place_state.record_declaration(def_id);
place_state.record_declaration(def_id, self.reachability);
}
pub(super) fn record_declaration_and_binding(
@ -956,10 +838,10 @@ impl<'db> UseDefMapBuilder<'db> {
.all_definitions
.push(DefinitionState::Defined(definition));
let place_state = &mut self.place_states[place];
place_state.record_declaration(def_id);
place_state.record_declaration(def_id, self.reachability);
place_state.record_binding(
def_id,
self.scope_start_visibility,
self.reachability,
self.is_class_scope,
is_place_name,
);
@ -970,7 +852,7 @@ impl<'db> UseDefMapBuilder<'db> {
let place_state = &mut self.place_states[place];
place_state.record_binding(
def_id,
self.scope_start_visibility,
self.reachability,
self.is_class_scope,
is_place_name,
);
@ -1024,7 +906,6 @@ impl<'db> UseDefMapBuilder<'db> {
pub(super) fn snapshot(&self) -> FlowSnapshot {
FlowSnapshot {
place_states: self.place_states.clone(),
scope_start_visibility: self.scope_start_visibility,
reachability: self.reachability,
}
}
@ -1039,16 +920,13 @@ impl<'db> UseDefMapBuilder<'db> {
// Restore the current visible-definitions state to the given snapshot.
self.place_states = snapshot.place_states;
self.scope_start_visibility = snapshot.scope_start_visibility;
self.reachability = snapshot.reachability;
// If the snapshot we are restoring is missing some places we've recorded since, we need
// to fill them in so the place IDs continue to line up. Since they don't exist in the
// snapshot, the correct state to fill them in with is "undefined".
self.place_states.resize(
num_places,
PlaceState::undefined(self.scope_start_visibility),
);
self.place_states
.resize(num_places, PlaceState::undefined(self.reachability));
}
/// Merge the given snapshot into the current state, reflecting that we might have taken either
@ -1059,13 +937,13 @@ impl<'db> UseDefMapBuilder<'db> {
// unreachable, we can leave it out of the merged result entirely. Note that we cannot
// perform any type inference at this point, so this is largely limited to unreachability
// via terminal statements. If a flow's reachability depends on an expression in the code,
// we will include the flow in the merged result; the visibility constraints of its
// we will include the flow in the merged result; the reachability constraints of its
// bindings will include this reachability condition, so that later during type inference,
// we can determine whether any particular binding is non-visible due to unreachability.
if snapshot.scope_start_visibility == ScopedVisibilityConstraintId::ALWAYS_FALSE {
if snapshot.reachability == ScopedReachabilityConstraintId::ALWAYS_FALSE {
return;
}
if self.scope_start_visibility == ScopedVisibilityConstraintId::ALWAYS_FALSE {
if self.reachability == ScopedReachabilityConstraintId::ALWAYS_FALSE {
self.restore(snapshot);
return;
}
@ -1081,24 +959,20 @@ impl<'db> UseDefMapBuilder<'db> {
current.merge(
snapshot,
&mut self.narrowing_constraints,
&mut self.visibility_constraints,
&mut self.reachability_constraints,
);
} else {
current.merge(
PlaceState::undefined(snapshot.scope_start_visibility),
PlaceState::undefined(snapshot.reachability),
&mut self.narrowing_constraints,
&mut self.visibility_constraints,
&mut self.reachability_constraints,
);
// Place not present in snapshot, so it's unbound/undeclared from that path.
}
}
self.scope_start_visibility = self
.visibility_constraints
.add_or_constraint(self.scope_start_visibility, snapshot.scope_start_visibility);
self.reachability = self
.visibility_constraints
.reachability_constraints
.add_or_constraint(self.reachability, snapshot.reachability);
}
@ -1115,14 +989,14 @@ impl<'db> UseDefMapBuilder<'db> {
all_definitions: self.all_definitions,
predicates: self.predicates.build(),
narrowing_constraints: self.narrowing_constraints.build(),
visibility_constraints: self.visibility_constraints.build(),
reachability_constraints: self.reachability_constraints.build(),
bindings_by_use: self.bindings_by_use,
node_reachability: self.node_reachability,
public_places: self.place_states,
declarations_by_binding: self.declarations_by_binding,
bindings_by_declaration: self.bindings_by_declaration,
eager_snapshots: self.eager_snapshots,
scope_start_visibility: self.scope_start_visibility,
end_of_scope_reachability: self.reachability,
}
}
}

261
crates/ty_python_semantic/src/semantic_index/use_def/place_state.rs
View file

@ -49,8 +49,8 @@ use smallvec::{SmallVec, smallvec};
use crate::semantic_index::narrowing_constraints::{
NarrowingConstraintsBuilder, ScopedNarrowingConstraint, ScopedNarrowingConstraintPredicate,
};
use crate::semantic_index::visibility_constraints::{
ScopedVisibilityConstraintId, VisibilityConstraintsBuilder,
use crate::semantic_index::reachability_constraints::{
ReachabilityConstraintsBuilder, ScopedReachabilityConstraintId,
};
/// A newtype-index for a definition in a particular scope.
@ -76,7 +76,7 @@ impl ScopedDefinitionId {
const INLINE_DEFINITIONS_PER_PLACE: usize = 4;
/// Live declarations for a single place at some point in control flow, with their
/// corresponding visibility constraints.
/// corresponding reachability constraints.
#[derive(Clone, Debug, Default, PartialEq, Eq, salsa::Update)]
pub(super) struct Declarations {
/// A list of live declarations for this place, sorted by their `ScopedDefinitionId`
@ -87,16 +87,16 @@ pub(super) struct Declarations {
#[derive(Clone, Debug, PartialEq, Eq)]
pub(super) struct LiveDeclaration {
pub(super) declaration: ScopedDefinitionId,
pub(super) visibility_constraint: ScopedVisibilityConstraintId,
pub(super) reachability_constraint: ScopedReachabilityConstraintId,
}
pub(super) type LiveDeclarationsIterator<'a> = std::slice::Iter<'a, LiveDeclaration>;
impl Declarations {
fn undeclared(scope_start_visibility: ScopedVisibilityConstraintId) -> Self {
fn undeclared(reachability_constraint: ScopedReachabilityConstraintId) -> Self {
let initial_declaration = LiveDeclaration {
declaration: ScopedDefinitionId::UNBOUND,
visibility_constraint: scope_start_visibility,
reachability_constraint,
};
Self {
live_declarations: smallvec![initial_declaration],
@ -104,24 +104,28 @@ impl Declarations {
}
/// Record a newly-encountered declaration for this place.
fn record_declaration(&mut self, declaration: ScopedDefinitionId) {
fn record_declaration(
&mut self,
declaration: ScopedDefinitionId,
reachability_constraint: ScopedReachabilityConstraintId,
) {
// The new declaration replaces all previous live declaration in this path.
self.live_declarations.clear();
self.live_declarations.push(LiveDeclaration {
declaration,
visibility_constraint: ScopedVisibilityConstraintId::ALWAYS_TRUE,
reachability_constraint,
});
}
/// Add given visibility constraint to all live declarations.
pub(super) fn record_visibility_constraint(
/// Add given reachability constraint to all live declarations.
pub(super) fn record_reachability_constraint(
&mut self,
visibility_constraints: &mut VisibilityConstraintsBuilder,
constraint: ScopedVisibilityConstraintId,
reachability_constraints: &mut ReachabilityConstraintsBuilder,
constraint: ScopedReachabilityConstraintId,
) {
for declaration in &mut self.live_declarations {
declaration.visibility_constraint = visibility_constraints
.add_and_constraint(declaration.visibility_constraint, constraint);
declaration.reachability_constraint = reachability_constraints
.add_and_constraint(declaration.reachability_constraint, constraint);
}
}
@ -130,46 +134,24 @@ impl Declarations {
self.live_declarations.iter()
}
/// Iterate over the IDs of each currently live declaration for this place
fn iter_declarations(&self) -> impl Iterator<Item = ScopedDefinitionId> + '_ {
self.iter().map(|lb| lb.declaration)
}
fn simplify_visibility_constraints(&mut self, other: Declarations) {
// If the set of live declarations hasn't changed, don't simplify.
if self.live_declarations.len() != other.live_declarations.len()
|| !self.iter_declarations().eq(other.iter_declarations())
{
return;
}
for (declaration, other_declaration) in self
.live_declarations
.iter_mut()
.zip(other.live_declarations)
{
declaration.visibility_constraint = other_declaration.visibility_constraint;
}
}
fn merge(&mut self, b: Self, visibility_constraints: &mut VisibilityConstraintsBuilder) {
fn merge(&mut self, b: Self, reachability_constraints: &mut ReachabilityConstraintsBuilder) {
let a = std::mem::take(self);
// Invariant: merge_join_by consumes the two iterators in sorted order, which ensures that
// the merged `live_declarations` vec remains sorted. If a definition is found in both `a`
// and `b`, we compose the constraints from the two paths in an appropriate way
// (intersection for narrowing constraints; ternary OR for visibility constraints). If a
// (intersection for narrowing constraints; ternary OR for reachability constraints). If a
// definition is found in only one path, it is used as-is.
let a = a.live_declarations.into_iter();
let b = b.live_declarations.into_iter();
for zipped in a.merge_join_by(b, |a, b| a.declaration.cmp(&b.declaration)) {
match zipped {
EitherOrBoth::Both(a, b) => {
let visibility_constraint = visibility_constraints
.add_or_constraint(a.visibility_constraint, b.visibility_constraint);
let reachability_constraint = reachability_constraints
.add_or_constraint(a.reachability_constraint, b.reachability_constraint);
self.live_declarations.push(LiveDeclaration {
declaration: a.declaration,
visibility_constraint,
reachability_constraint,
});
}
@ -193,7 +175,7 @@ pub(super) enum EagerSnapshot {
}
/// Live bindings for a single place at some point in control flow. Each live binding comes
/// with a set of narrowing constraints and a visibility constraint.
/// with a set of narrowing constraints and a reachability constraint.
#[derive(Clone, Debug, Default, PartialEq, Eq, salsa::Update)]
pub(super) struct Bindings {
/// The narrowing constraint applicable to the "unbound" binding, if we need access to it even
@ -217,17 +199,17 @@ impl Bindings {
pub(super) struct LiveBinding {
pub(super) binding: ScopedDefinitionId,
pub(super) narrowing_constraint: ScopedNarrowingConstraint,
pub(super) visibility_constraint: ScopedVisibilityConstraintId,
pub(super) reachability_constraint: ScopedReachabilityConstraintId,
}
pub(super) type LiveBindingsIterator<'a> = std::slice::Iter<'a, LiveBinding>;
impl Bindings {
fn unbound(scope_start_visibility: ScopedVisibilityConstraintId) -> Self {
fn unbound(reachability_constraint: ScopedReachabilityConstraintId) -> Self {
let initial_binding = LiveBinding {
binding: ScopedDefinitionId::UNBOUND,
narrowing_constraint: ScopedNarrowingConstraint::empty(),
visibility_constraint: scope_start_visibility,
reachability_constraint,
};
Self {
unbound_narrowing_constraint: None,
@ -239,7 +221,7 @@ impl Bindings {
pub(super) fn record_binding(
&mut self,
binding: ScopedDefinitionId,
visibility_constraint: ScopedVisibilityConstraintId,
reachability_constraint: ScopedReachabilityConstraintId,
is_class_scope: bool,
is_place_name: bool,
) {
@ -254,7 +236,7 @@ impl Bindings {
self.live_bindings.push(LiveBinding {
binding,
narrowing_constraint: ScopedNarrowingConstraint::empty(),
visibility_constraint,
reachability_constraint,
});
}
@ -270,15 +252,15 @@ impl Bindings {
}
}
/// Add given visibility constraint to all live bindings.
pub(super) fn record_visibility_constraint(
/// Add given reachability constraint to all live bindings.
pub(super) fn record_reachability_constraint(
&mut self,
visibility_constraints: &mut VisibilityConstraintsBuilder,
constraint: ScopedVisibilityConstraintId,
reachability_constraints: &mut ReachabilityConstraintsBuilder,
constraint: ScopedReachabilityConstraintId,
) {
for binding in &mut self.live_bindings {
binding.visibility_constraint = visibility_constraints
.add_and_constraint(binding.visibility_constraint, constraint);
binding.reachability_constraint = reachability_constraints
.add_and_constraint(binding.reachability_constraint, constraint);
}
}
@ -287,29 +269,11 @@ impl Bindings {
self.live_bindings.iter()
}
/// Iterate over the IDs of each currently live binding for this place
fn iter_bindings(&self) -> impl Iterator<Item = ScopedDefinitionId> + '_ {
self.iter().map(|lb| lb.binding)
}
fn simplify_visibility_constraints(&mut self, other: Bindings) {
// If the set of live bindings hasn't changed, don't simplify.
if self.live_bindings.len() != other.live_bindings.len()
|| !self.iter_bindings().eq(other.iter_bindings())
{
return;
}
for (binding, other_binding) in self.live_bindings.iter_mut().zip(other.live_bindings) {
binding.visibility_constraint = other_binding.visibility_constraint;
}
}
fn merge(
&mut self,
b: Self,
narrowing_constraints: &mut NarrowingConstraintsBuilder,
visibility_constraints: &mut VisibilityConstraintsBuilder,
reachability_constraints: &mut ReachabilityConstraintsBuilder,
) {
let a = std::mem::take(self);
@ -324,7 +288,7 @@ impl Bindings {
// Invariant: merge_join_by consumes the two iterators in sorted order, which ensures that
// the merged `live_bindings` vec remains sorted. If a definition is found in both `a` and
// `b`, we compose the constraints from the two paths in an appropriate way (intersection
// for narrowing constraints; ternary OR for visibility constraints). If a definition is
// for narrowing constraints; ternary OR for reachability constraints). If a definition is
// found in only one path, it is used as-is.
let a = a.live_bindings.into_iter();
let b = b.live_bindings.into_iter();
@ -337,14 +301,14 @@ impl Bindings {
let narrowing_constraint = narrowing_constraints
.intersect_constraints(a.narrowing_constraint, b.narrowing_constraint);
// For visibility constraints, we merge them using a ternary OR operation:
let visibility_constraint = visibility_constraints
.add_or_constraint(a.visibility_constraint, b.visibility_constraint);
// For reachability constraints, we merge them using a ternary OR operation:
let reachability_constraint = reachability_constraints
.add_or_constraint(a.reachability_constraint, b.reachability_constraint);
self.live_bindings.push(LiveBinding {
binding: a.binding,
narrowing_constraint,
visibility_constraint,
reachability_constraint,
});
}
@ -364,10 +328,10 @@ pub(in crate::semantic_index) struct PlaceState {
impl PlaceState {
/// Return a new [`PlaceState`] representing an unbound, undeclared place.
pub(super) fn undefined(scope_start_visibility: ScopedVisibilityConstraintId) -> Self {
pub(super) fn undefined(reachability: ScopedReachabilityConstraintId) -> Self {
Self {
declarations: Declarations::undeclared(scope_start_visibility),
bindings: Bindings::unbound(scope_start_visibility),
declarations: Declarations::undeclared(reachability),
bindings: Bindings::unbound(reachability),
}
}
@ -375,14 +339,14 @@ impl PlaceState {
pub(super) fn record_binding(
&mut self,
binding_id: ScopedDefinitionId,
visibility_constraint: ScopedVisibilityConstraintId,
reachability_constraint: ScopedReachabilityConstraintId,
is_class_scope: bool,
is_place_name: bool,
) {
debug_assert_ne!(binding_id, ScopedDefinitionId::UNBOUND);
self.bindings.record_binding(
binding_id,
visibility_constraint,
reachability_constraint,
is_class_scope,
is_place_name,
);
@ -398,31 +362,26 @@ impl PlaceState {
.record_narrowing_constraint(narrowing_constraints, constraint);
}
/// Add given visibility constraint to all live bindings.
pub(super) fn record_visibility_constraint(
/// Add given reachability constraint to all live bindings.
pub(super) fn record_reachability_constraint(
&mut self,
visibility_constraints: &mut VisibilityConstraintsBuilder,
constraint: ScopedVisibilityConstraintId,
reachability_constraints: &mut ReachabilityConstraintsBuilder,
constraint: ScopedReachabilityConstraintId,
) {
self.bindings
.record_visibility_constraint(visibility_constraints, constraint);
.record_reachability_constraint(reachability_constraints, constraint);
self.declarations
.record_visibility_constraint(visibility_constraints, constraint);
}
/// Simplifies this snapshot to have the same visibility constraints as a previous point in the
/// control flow, but only if the set of live bindings or declarations for this place hasn't
/// changed.
pub(super) fn simplify_visibility_constraints(&mut self, snapshot_state: PlaceState) {
self.bindings
.simplify_visibility_constraints(snapshot_state.bindings);
self.declarations
.simplify_visibility_constraints(snapshot_state.declarations);
.record_reachability_constraint(reachability_constraints, constraint);
}
/// Record a newly-encountered declaration of this place.
pub(super) fn record_declaration(&mut self, declaration_id: ScopedDefinitionId) {
self.declarations.record_declaration(declaration_id);
pub(super) fn record_declaration(
&mut self,
declaration_id: ScopedDefinitionId,
reachability_constraint: ScopedReachabilityConstraintId,
) {
self.declarations
.record_declaration(declaration_id, reachability_constraint);
}
/// Merge another [`PlaceState`] into this one.
@ -430,12 +389,12 @@ impl PlaceState {
&mut self,
b: PlaceState,
narrowing_constraints: &mut NarrowingConstraintsBuilder,
visibility_constraints: &mut VisibilityConstraintsBuilder,
reachability_constraints: &mut ReachabilityConstraintsBuilder,
) {
self.bindings
.merge(b.bindings, narrowing_constraints, visibility_constraints);
.merge(b.bindings, narrowing_constraints, reachability_constraints);
self.declarations
.merge(b.declarations, visibility_constraints);
.merge(b.declarations, reachability_constraints);
}
pub(super) fn bindings(&self) -> &Bindings {
@ -488,7 +447,7 @@ mod tests {
.map(
|LiveDeclaration {
declaration,
visibility_constraint: _,
reachability_constraint: _,
}| {
if *declaration == ScopedDefinitionId::UNBOUND {
"undeclared".into()
@ -504,7 +463,7 @@ mod tests {
#[test]
fn unbound() {
let narrowing_constraints = NarrowingConstraintsBuilder::default();
let sym = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
let sym = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
assert_bindings(&narrowing_constraints, &sym, &["unbound<>"]);
}
@ -512,10 +471,10 @@ mod tests {
#[test]
fn with() {
let narrowing_constraints = NarrowingConstraintsBuilder::default();
let mut sym = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
let mut sym = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym.record_binding(
ScopedDefinitionId::from_u32(1),
ScopedVisibilityConstraintId::ALWAYS_TRUE,
ScopedReachabilityConstraintId::ALWAYS_TRUE,
false,
true,
);
@ -526,10 +485,10 @@ mod tests {
#[test]
fn record_constraint() {
let mut narrowing_constraints = NarrowingConstraintsBuilder::default();
let mut sym = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
let mut sym = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym.record_binding(
ScopedDefinitionId::from_u32(1),
ScopedVisibilityConstraintId::ALWAYS_TRUE,
ScopedReachabilityConstraintId::ALWAYS_TRUE,
false,
true,
);
@ -542,23 +501,23 @@ mod tests {
#[test]
fn merge() {
let mut narrowing_constraints = NarrowingConstraintsBuilder::default();
let mut visibility_constraints = VisibilityConstraintsBuilder::default();
let mut reachability_constraints = ReachabilityConstraintsBuilder::default();
// merging the same definition with the same constraint keeps the constraint
let mut sym1a = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
let mut sym1a = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym1a.record_binding(
ScopedDefinitionId::from_u32(1),
ScopedVisibilityConstraintId::ALWAYS_TRUE,
ScopedReachabilityConstraintId::ALWAYS_TRUE,
false,
true,
);
let predicate = ScopedPredicateId::from_u32(0).into();
sym1a.record_narrowing_constraint(&mut narrowing_constraints, predicate);
let mut sym1b = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
let mut sym1b = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym1b.record_binding(
ScopedDefinitionId::from_u32(1),
ScopedVisibilityConstraintId::ALWAYS_TRUE,
ScopedReachabilityConstraintId::ALWAYS_TRUE,
false,
true,
);
@ -568,26 +527,26 @@ mod tests {
sym1a.merge(
sym1b,
&mut narrowing_constraints,
&mut visibility_constraints,
&mut reachability_constraints,
);
let mut sym1 = sym1a;
assert_bindings(&narrowing_constraints, &sym1, &["1<0>"]);
// merging the same definition with differing constraints drops all constraints
let mut sym2a = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
let mut sym2a = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym2a.record_binding(
ScopedDefinitionId::from_u32(2),
ScopedVisibilityConstraintId::ALWAYS_TRUE,
ScopedReachabilityConstraintId::ALWAYS_TRUE,
false,
true,
);
let predicate = ScopedPredicateId::from_u32(1).into();
sym2a.record_narrowing_constraint(&mut narrowing_constraints, predicate);
let mut sym1b = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
let mut sym1b = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym1b.record_binding(
ScopedDefinitionId::from_u32(2),
ScopedVisibilityConstraintId::ALWAYS_TRUE,
ScopedReachabilityConstraintId::ALWAYS_TRUE,
false,
true,
);
@ -597,28 +556,28 @@ mod tests {
sym2a.merge(
sym1b,
&mut narrowing_constraints,
&mut visibility_constraints,
&mut reachability_constraints,
);
let sym2 = sym2a;
assert_bindings(&narrowing_constraints, &sym2, &["2<>"]);
// merging a constrained definition with unbound keeps both
let mut sym3a = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
let mut sym3a = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym3a.record_binding(
ScopedDefinitionId::from_u32(3),
ScopedVisibilityConstraintId::ALWAYS_TRUE,
ScopedReachabilityConstraintId::ALWAYS_TRUE,
false,
true,
);
let predicate = ScopedPredicateId::from_u32(3).into();
sym3a.record_narrowing_constraint(&mut narrowing_constraints, predicate);
let sym2b = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
let sym2b = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym3a.merge(
sym2b,
&mut narrowing_constraints,
&mut visibility_constraints,
&mut reachability_constraints,
);
let sym3 = sym3a;
assert_bindings(&narrowing_constraints, &sym3, &["unbound<>", "3<3>"]);
@ -627,7 +586,7 @@ mod tests {
sym1.merge(
sym3,
&mut narrowing_constraints,
&mut visibility_constraints,
&mut reachability_constraints,
);
let sym = sym1;
assert_bindings(&narrowing_constraints, &sym, &["unbound<>", "1<0>", "3<3>"]);
@ -635,24 +594,33 @@ mod tests {
#[test]
fn no_declaration() {
let sym = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
let sym = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
assert_declarations(&sym, &["undeclared"]);
}
#[test]
fn record_declaration() {
let mut sym = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
sym.record_declaration(ScopedDefinitionId::from_u32(1));
let mut sym = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym.record_declaration(
ScopedDefinitionId::from_u32(1),
ScopedReachabilityConstraintId::ALWAYS_TRUE,
);
assert_declarations(&sym, &["1"]);
}
#[test]
fn record_declaration_override() {
let mut sym = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
sym.record_declaration(ScopedDefinitionId::from_u32(1));
sym.record_declaration(ScopedDefinitionId::from_u32(2));
let mut sym = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym.record_declaration(
ScopedDefinitionId::from_u32(1),
ScopedReachabilityConstraintId::ALWAYS_TRUE,
);
sym.record_declaration(
ScopedDefinitionId::from_u32(2),
ScopedReachabilityConstraintId::ALWAYS_TRUE,
);
assert_declarations(&sym, &["2"]);
}
@ -660,17 +628,23 @@ mod tests {
#[test]
fn record_declaration_merge() {
let mut narrowing_constraints = NarrowingConstraintsBuilder::default();
let mut visibility_constraints = VisibilityConstraintsBuilder::default();
let mut sym = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
sym.record_declaration(ScopedDefinitionId::from_u32(1));
let mut reachability_constraints = ReachabilityConstraintsBuilder::default();
let mut sym = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym.record_declaration(
ScopedDefinitionId::from_u32(1),
ScopedReachabilityConstraintId::ALWAYS_TRUE,
);
let mut sym2 = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
sym2.record_declaration(ScopedDefinitionId::from_u32(2));
let mut sym2 = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym2.record_declaration(
ScopedDefinitionId::from_u32(2),
ScopedReachabilityConstraintId::ALWAYS_TRUE,
);
sym.merge(
sym2,
&mut narrowing_constraints,
&mut visibility_constraints,
&mut reachability_constraints,
);
assert_declarations(&sym, &["1", "2"]);
@ -679,16 +653,19 @@ mod tests {
#[test]
fn record_declaration_merge_partial_undeclared() {
let mut narrowing_constraints = NarrowingConstraintsBuilder::default();
let mut visibility_constraints = VisibilityConstraintsBuilder::default();
let mut sym = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
sym.record_declaration(ScopedDefinitionId::from_u32(1));
let mut reachability_constraints = ReachabilityConstraintsBuilder::default();
let mut sym = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym.record_declaration(
ScopedDefinitionId::from_u32(1),
ScopedReachabilityConstraintId::ALWAYS_TRUE,
);
let sym2 = PlaceState::undefined(ScopedVisibilityConstraintId::ALWAYS_TRUE);
let sym2 = PlaceState::undefined(ScopedReachabilityConstraintId::ALWAYS_TRUE);
sym.merge(
sym2,
&mut narrowing_constraints,
&mut visibility_constraints,
&mut reachability_constraints,
);
assert_declarations(&sym, &["undeclared", "1"]);

4
crates/ty_python_semantic/src/types.rs
View file

@ -7052,6 +7052,10 @@ impl Truthiness {
matches!(self, Truthiness::AlwaysFalse)
}
pub(crate) const fn may_be_true(self) -> bool {
!self.is_always_false()
}
pub(crate) const fn is_always_true(self) -> bool {
matches!(self, Truthiness::AlwaysTrue)
}

28
crates/ty_python_semantic/src/types/class.rs
View file

@ -1637,8 +1637,8 @@ impl<'db> ClassLiteral<'db> {
let method_scope = method_scope_id.to_scope_id(db, file);
let method_map = use_def_map(db, method_scope);
// The attribute assignment inherits the visibility of the method which contains it
let is_method_visible =
// The attribute assignment inherits the reachability of the method which contains it
let is_method_reachable =
if let Some(method_def) = method_scope.node(db).as_function(&module) {
let method = index.expect_single_definition(method_def);
let method_place = class_table.place_id_by_name(&method_def.name).unwrap();
@ -1646,13 +1646,13 @@ impl<'db> ClassLiteral<'db> {
.public_bindings(method_place)
.find_map(|bind| {
(bind.binding.is_defined_and(|def| def == method))
.then(|| class_map.is_binding_visible(db, &bind))
.then(|| class_map.is_binding_reachable(db, &bind))
})
.unwrap_or(Truthiness::AlwaysFalse)
} else {
Truthiness::AlwaysFalse
};
if is_method_visible.is_always_false() {
if is_method_reachable.is_always_false() {
continue;
}
@ -1663,7 +1663,7 @@ impl<'db> ClassLiteral<'db> {
for attribute_assignment in attribute_assignments {
if let DefinitionState::Undefined = attribute_assignment.binding {
// Store the implicit unbound binding here so that we can delay the
// computation of `unbound_visibility` to the point when we actually
// computation of `unbound_reachability` to the point when we actually
// need it. This is an optimization for the common case where the
// `unbound` binding is the only binding of the `name` attribute,
// i.e. if there is no `self.name = …` assignment in this method.
@ -1675,8 +1675,8 @@ impl<'db> ClassLiteral<'db> {
continue;
};
match method_map
.is_binding_visible(db, &attribute_assignment)
.and(is_method_visible)
.is_binding_reachable(db, &attribute_assignment)
.and(is_method_reachable)
{
Truthiness::AlwaysTrue => {
is_attribute_bound = Truthiness::AlwaysTrue;
@ -1691,17 +1691,17 @@ impl<'db> ClassLiteral<'db> {
}
}
// There is at least one attribute assignment that may be visible,
// so if `unbound_visibility` is always false then this attribute is considered bound.
// TODO: this is incomplete logic since the attributes bound after termination are considered visible.
let unbound_visibility = unbound_binding
// There is at least one attribute assignment that may be reachable, so if `unbound_reachability` is
// always false then this attribute is considered bound.
// TODO: this is incomplete logic since the attributes bound after termination are considered reachable.
let unbound_reachability = unbound_binding
.as_ref()
.map(|binding| method_map.is_binding_visible(db, binding))
.map(|binding| method_map.is_binding_reachable(db, binding))
.unwrap_or(Truthiness::AlwaysFalse);
if unbound_visibility
if unbound_reachability
.negate()
.and(is_method_visible)
.and(is_method_reachable)
.is_always_true()
{
is_attribute_bound = Truthiness::AlwaysTrue;

25
crates/ty_python_semantic/src/types/infer.rs
View file

@ -1853,6 +1853,12 @@ impl<'db, 'ast> TypeInferenceBuilder<'db, 'ast> {
function.known(self.db()),
Some(KnownFunction::Overload | KnownFunction::AbstractMethod)
),
Type::Never => {
// In unreachable code, we infer `Never` for decorators like `typing.overload`.
// Return `true` here to avoid false positive `invalid-return-type` lints for
// `@overload`ed functions without a body in unreachable code.
true
}
_ => false,
}
})
@ -1967,7 +1973,7 @@ impl<'db, 'ast> TypeInferenceBuilder<'db, 'ast> {
);
}
let use_def = self.index.use_def_map(scope_id);
if use_def.can_implicit_return(self.db())
if use_def.can_implicitly_return_none(self.db())
&& !Type::none(self.db()).is_assignable_to(self.db(), expected_ty)
{
let no_return = self.return_types_and_ranges.is_empty();
@ -9200,6 +9206,15 @@ impl<'db> TypeInferenceBuilder<'db, '_> {
} = subscript;
match value_ty {
Type::Never => {
// This case can be entered when we use a type annotation like `Literal[1]`
// in unreachable code, since we infer `Never` for `Literal`. We call
// `infer_expression` (instead of `infer_type_expression`) here to avoid
// false-positive `invalid-type-form` diagnostics (`1` is not a valid type
// expression).
self.infer_expression(&subscript.slice);
Type::unknown()
}
Type::ClassLiteral(literal) if literal.is_known(self.db(), KnownClass::Any) => {
if let Some(builder) = self.context.report_lint(&INVALID_TYPE_FORM, subscript) {
builder.into_diagnostic("Type `typing.Any` expected no type parameter");
@ -10303,16 +10318,16 @@ mod tests {
}
/// Test that a symbol known to be unbound in a scope does not still trigger cycle-causing
/// visibility-constraint checks in that scope.
/// reachability-constraint checks in that scope.
#[test]
fn unbound_symbol_no_visibility_constraint_check() {
fn unbound_symbol_no_reachability_constraint_check() {
let mut db = setup_db();
// If the bug we are testing for is not fixed, what happens is that when inferring the
// `flag: bool = True` definitions, we look up `bool` as a deferred name (thus from end of
// scope), and because of the early return its "unbound" binding has a visibility
// scope), and because of the early return its "unbound" binding has a reachability
// constraint of `~flag`, which we evaluate, meaning we have to evaluate the definition of
// `flag` -- and we are in a cycle. With the fix, we short-circuit evaluating visibility
// `flag` -- and we are in a cycle. With the fix, we short-circuit evaluating reachability
// constraints on "unbound" if a symbol is otherwise not bound.
db.write_dedented(
"src/a.py",