[ty] Add caching for submodule completion suggestions (#19408)

This change makes it so we aren't doing a directory traversal every time
we ask for completions from a module. Specifically, submodules that
aren't attributes of their parent module can only be discovered by
looking at the directory tree. But we want to avoid doing a directory
scan unless we think there are changes.

To make this work, this change does a little bit of surgery to
`FileRoot`. Previously, a `FileRoot` was only used for library search
paths. Its revision was bumped whenever a file in that tree was added,
deleted or even modified (to support the discovery of `pth` files and
changes to its contents). This generally seems fine since these are
presumably dependency paths that shouldn't change frequently.

In this change, we add a `FileRoot` for the project. But having the
`FileRoot`'s revision bumped for every change in the project makes
caching based on that `FileRoot` rather ineffective. That is, cache
invalidation will occur too aggressively. To the point that there is
little point in adding caching in the first place. To mitigate this, a
`FileRoot`'s revision is only bumped on a change to a child file's
contents when the `FileRoot` is a `LibrarySearchPath`. Otherwise, we
only bump the revision when a file is created or added.

The effect is that, at least in VS Code, when a new module is added or
removed, this change is picked up and the cache is properly invalidated.
Other LSP clients with worse support for file watching (which seems to
be the case for the CoC vim plugin that I use) don't work as well. Here,
the cache is less likely to be invalidated which might cause completions
to have stale results. Unless there's an obvious way to fix or improve
this, I propose punting on improvements here for now.

crates/ty_python_semantic/src/module_resolver/module.rs

@@ -17,6 +17,7 @@ pub struct Module {
     inner: Arc<ModuleInner>,
 }
 
+#[salsa::tracked]
 impl Module {
     pub(crate) fn file_module(
         name: ModuleName,
@@ -97,11 +98,16 @@ impl Module {
     ///
     /// The names returned correspond to the "base" name of the module.
     /// That is, `{self.name}.{basename}` should give the full module name.
-    pub fn all_submodules(&self, db: &dyn Db) -> Vec<Name> {
-        self.all_submodules_inner(db).unwrap_or_default()
+    pub fn all_submodules<'db>(&self, db: &'db dyn Db) -> &'db [Name] {
+        self.clone()
+            .all_submodules_inner(db, ())
+            .as_deref()
+            .unwrap_or_default()
     }
 
-    fn all_submodules_inner(&self, db: &dyn Db) -> Option<Vec<Name>> {
+    #[allow(clippy::ref_option, clippy::used_underscore_binding)]
+    #[salsa::tracked(returns(ref))]
+    fn all_submodules_inner(self, db: &dyn Db, _dummy: ()) -> Option<Vec<Name>> {
         fn is_submodule(
             is_dir: bool,
             is_file: bool,
@@ -136,32 +142,42 @@ impl Module {
         );
 
         Some(match path.parent()? {
-            SystemOrVendoredPathRef::System(parent_directory) => db
-                .system()
-                .read_directory(parent_directory)
-                .inspect_err(|err| {
-                    tracing::debug!(
-                        "Failed to read {parent_directory:?} when looking for \
-                         its possible submodules: {err}"
-                    );
-                })
-                .ok()?
-                .flatten()
-                .filter(|entry| {
-                    let ty = entry.file_type();
-                    let path = entry.path();
-                    is_submodule(
-                        ty.is_directory(),
-                        ty.is_file(),
-                        path.file_name(),
-                        path.extension(),
-                    )
-                })
-                .filter_map(|entry| {
-                    let stem = entry.path().file_stem()?;
-                    is_identifier(stem).then(|| Name::from(stem))
-                })
-                .collect(),
+            SystemOrVendoredPathRef::System(parent_directory) => {
+                // Read the revision on the corresponding file root to
+                // register an explicit dependency on this directory
+                // tree. When the revision gets bumped, the cache
+                // that Salsa creates does for this routine will be
+                // invalidated.
+                if let Some(root) = db.files().root(db, parent_directory) {
+                    let _ = root.revision(db);
+                }
+
+                db.system()
+                    .read_directory(parent_directory)
+                    .inspect_err(|err| {
+                        tracing::debug!(
+                            "Failed to read {parent_directory:?} when looking for \
+                             its possible submodules: {err}"
+                        );
+                    })
+                    .ok()?
+                    .flatten()
+                    .filter(|entry| {
+                        let ty = entry.file_type();
+                        let path = entry.path();
+                        is_submodule(
+                            ty.is_directory(),
+                            ty.is_file(),
+                            path.file_name(),
+                            path.extension(),
+                        )
+                    })
+                    .filter_map(|entry| {
+                        let stem = entry.path().file_stem()?;
+                        is_identifier(stem).then(|| Name::from(stem))
+                    })
+                    .collect()
+            }
             SystemOrVendoredPathRef::Vendored(parent_directory) => db
                 .vendored()
                 .read_directory(parent_directory)