[ty] Ask the LSP client to watch all project search paths

This change rejiggers how we register globs for file watching with the
LSP client. Previously, we registered a few globs like `**/*.py`,
`**/pyproject.toml` and more. There were two problems with this
approach.

Firstly, it only watches files within the project root. Search paths may
be outside the project root. Such as virtualenv directory.

Secondly, there is variation on how tools interact with virtual
environments. In the case of uv, depending on its link mode, we might
not get any file change notifications after running `uv add foo` or
`uv remove foo`.

To remedy this, we instead just list for file change notifications on
all files for all search paths. This simplifies the globs we use, but
does potentially increase the number of notifications we'll get.
However, given the somewhat simplistic interface supported by the LSP
protocol, I think this is unavoidable (unless we used our own file
watcher, which has its own considerably downsides). Moreover, this is
seemingly consistent with how `ty check --watch` works.

This also required moving file watcher registration to *after*
workspaces are initialized, or else we don't know what the right search
paths are.

This change is in service of #19883, which in order for cache
invalidation to work right, the LSP client needs to send notifications
whenever a dependency is added or removed. This change should make that
possible.

I tried this patch with #19883 in addition to my work to activate Salsa
caching, and everything seems to work as I'd expect. That is,
completions no longer show stale results after a dependency is added or
removed.

crates/ty_server/src/capabilities.rs

@@ -30,9 +30,10 @@ bitflags::bitflags! {
         const HIERARCHICAL_DOCUMENT_SYMBOL_SUPPORT = 1 << 10;
         const WORK_DONE_PROGRESS = 1 << 11;
         const FILE_WATCHER_SUPPORT = 1 << 12;
-        const DIAGNOSTIC_DYNAMIC_REGISTRATION = 1 << 13;
+        const RELATIVE_FILE_WATCHER_SUPPORT = 1 << 13;
-        const WORKSPACE_CONFIGURATION = 1 << 14;
+        const DIAGNOSTIC_DYNAMIC_REGISTRATION = 1 << 14;
-        const RENAME_DYNAMIC_REGISTRATION = 1 << 15;
+        const WORKSPACE_CONFIGURATION = 1 << 15;
+        const RENAME_DYNAMIC_REGISTRATION = 1 << 16;
     }
 }
 
@@ -107,6 +108,14 @@ impl ResolvedClientCapabilities {
         self.contains(Self::FILE_WATCHER_SUPPORT)
     }
 
+    /// Returns `true` if the client supports relative file watcher capabilities.
+    ///
+    /// This permits specifying a "base uri" that a glob is interpreted
+    /// relative to.
+    pub(crate) const fn supports_relative_file_watcher(self) -> bool {
+        self.contains(Self::RELATIVE_FILE_WATCHER_SUPPORT)
+    }
+
     /// Returns `true` if the client supports dynamic registration for diagnostic capabilities.
     pub(crate) const fn supports_diagnostic_dynamic_registration(self) -> bool {
         self.contains(Self::DIAGNOSTIC_DYNAMIC_REGISTRATION)
@@ -144,11 +153,15 @@ impl ResolvedClientCapabilities {
             flags |= Self::INLAY_HINT_REFRESH;
         }
 
-        if workspace
+        if let Some(capabilities) =
-            .and_then(|workspace| workspace.did_change_watched_files?.dynamic_registration)
+            workspace.and_then(|workspace| workspace.did_change_watched_files.as_ref())
-            .unwrap_or_default()
         {
-            flags |= Self::FILE_WATCHER_SUPPORT;
+            if capabilities.dynamic_registration == Some(true) {
+                flags |= Self::FILE_WATCHER_SUPPORT;
+            }
+            if capabilities.relative_pattern_support == Some(true) {
+                flags |= Self::RELATIVE_FILE_WATCHER_SUPPORT;
+            }
         }
 
         if text_document.is_some_and(|text_document| text_document.diagnostic.is_some()) {

crates/ty_server/src/server/main_loop.rs

@@ -5,10 +5,8 @@ use crate::session::{ClientOptions, SuspendedWorkspaceDiagnosticRequest};
 use anyhow::anyhow;
 use crossbeam::select;
 use lsp_server::Message;
-use lsp_types::notification::{DidChangeWatchedFiles, Notification};
+use lsp_types::notification::Notification;
-use lsp_types::{
+use lsp_types::{ConfigurationParams, Url};
-    ConfigurationParams, DidChangeWatchedFilesRegistrationOptions, FileSystemWatcher, Url,
-};
 use serde_json::Value;
 
 pub(crate) type ConnectionSender = crossbeam::channel::Sender<Message>;
@@ -154,6 +152,10 @@ impl Server {
                     Action::InitializeWorkspaces(workspaces_with_options) => {
                         self.session
                             .initialize_workspaces(workspaces_with_options, &client);
+                        // We do this here after workspaces have been initialized
+                        // so that the file watcher globs can take project search
+                        // paths into account.
+                        // self.try_register_file_watcher(&client);
                     }
                 },
             }
@@ -195,7 +197,6 @@ impl Server {
 
     fn initialize(&mut self, client: &Client) {
         self.request_workspace_configurations(client);
-        self.try_register_file_watcher(client);
     }
 
     /// Requests workspace configurations from the client for all the workspaces in the session.
@@ -282,68 +283,6 @@ impl Server {
             },
         );
     }
-
-    /// Try to register the file watcher provided by the client if the client supports it.
-    fn try_register_file_watcher(&mut self, client: &Client) {
-        static FILE_WATCHER_REGISTRATION_ID: &str = "ty/workspace/didChangeWatchedFiles";
-
-        if !self.session.client_capabilities().supports_file_watcher() {
-            tracing::warn!("Client does not support file system watching");
-            return;
-        }
-
-        let registration = lsp_types::Registration {
-            id: FILE_WATCHER_REGISTRATION_ID.to_owned(),
-            method: DidChangeWatchedFiles::METHOD.to_owned(),
-            register_options: Some(
-                serde_json::to_value(DidChangeWatchedFilesRegistrationOptions {
-                    watchers: vec![
-                        FileSystemWatcher {
-                            glob_pattern: lsp_types::GlobPattern::String("**/ty.toml".into()),
-                            kind: None,
-                        },
-                        FileSystemWatcher {
-                            glob_pattern: lsp_types::GlobPattern::String("**/.gitignore".into()),
-                            kind: None,
-                        },
-                        FileSystemWatcher {
-                            glob_pattern: lsp_types::GlobPattern::String("**/.ignore".into()),
-                            kind: None,
-                        },
-                        FileSystemWatcher {
-                            glob_pattern: lsp_types::GlobPattern::String(
-                                "**/pyproject.toml".into(),
-                            ),
-                            kind: None,
-                        },
-                        FileSystemWatcher {
-                            glob_pattern: lsp_types::GlobPattern::String("**/*.py".into()),
-                            kind: None,
-                        },
-                        FileSystemWatcher {
-                            glob_pattern: lsp_types::GlobPattern::String("**/*.pyi".into()),
-                            kind: None,
-                        },
-                        FileSystemWatcher {
-                            glob_pattern: lsp_types::GlobPattern::String("**/*.ipynb".into()),
-                            kind: None,
-                        },
-                    ],
-                })
-                .unwrap(),
-            ),
-        };
-
-        client.send_request::<lsp_types::request::RegisterCapability>(
-            &self.session,
-            lsp_types::RegistrationParams {
-                registrations: vec![registration],
-            },
-            |_: &Client, ()| {
-                tracing::info!("File watcher registration completed successfully");
-            },
-        );
-    }
 }
 
 /// An action that should be performed on the main loop.

crates/ty_server/src/session.rs

@@ -3,13 +3,14 @@
 use anyhow::{Context, anyhow};
 use index::DocumentQueryError;
 use lsp_server::{Message, RequestId};
-use lsp_types::notification::{Exit, Notification};
+use lsp_types::notification::{DidChangeWatchedFiles, Exit, Notification};
 use lsp_types::request::{
     DocumentDiagnosticRequest, RegisterCapability, Rename, Request, Shutdown, UnregisterCapability,
     WorkspaceDiagnosticRequest,
 };
 use lsp_types::{
-    DiagnosticRegistrationOptions, DiagnosticServerCapabilities, Registration, RegistrationParams,
+    DiagnosticRegistrationOptions, DiagnosticServerCapabilities,
+    DidChangeWatchedFilesRegistrationOptions, FileSystemWatcher, Registration, RegistrationParams,
     TextDocumentContentChangeEvent, Unregistration, UnregistrationParams, Url,
 };
 use options::GlobalOptions;
@@ -308,6 +309,14 @@ impl Session {
         &self.project_state(path).db
     }
 
+    /// Returns an iterator, in arbitrary order, over all project databases
+    /// in this session.
+    pub(crate) fn project_dbs(&self) -> impl Iterator<Item = &ProjectDatabase> {
+        self.projects
+            .values()
+            .map(|project_state| &project_state.db)
+    }
+
     /// Returns a mutable reference to the project's [`ProjectDatabase`] in which the given `path`
     /// belongs.
     ///
@@ -600,6 +609,7 @@ impl Session {
     fn register_capabilities(&mut self, client: &Client) {
         static DIAGNOSTIC_REGISTRATION_ID: &str = "ty/textDocument/diagnostic";
         static RENAME_REGISTRATION_ID: &str = "ty/textDocument/rename";
+        static FILE_WATCHER_REGISTRATION_ID: &str = "ty/workspace/didChangeWatchedFiles";
 
         let mut registrations = vec![];
         let mut unregistrations = vec![];
@@ -665,6 +675,20 @@ impl Session {
             }
         }
 
+        if let Some(register_options) = self.file_watcher_registration_options() {
+            if self.registrations.contains(DidChangeWatchedFiles::METHOD) {
+                unregistrations.push(Unregistration {
+                    id: FILE_WATCHER_REGISTRATION_ID.into(),
+                    method: DidChangeWatchedFiles::METHOD.into(),
+                });
+            }
+            registrations.push(Registration {
+                id: FILE_WATCHER_REGISTRATION_ID.into(),
+                method: DidChangeWatchedFiles::METHOD.into(),
+                register_options: Some(serde_json::to_value(register_options).unwrap()),
+            });
+        }
+
         // First, unregister any existing capabilities and then register or re-register them.
         self.unregister_dynamic_capability(client, unregistrations);
         self.register_dynamic_capability(client, registrations);
@@ -719,6 +743,82 @@ impl Session {
         );
     }
 
+    /// Try to register the file watcher provided by the client if the client supports it.
+    ///
+    /// Note that this should be called *after* workspaces/projects have been initialized.
+    /// This is required because the globs we use for registering file watching take
+    /// project search paths into account.
+    fn file_watcher_registration_options(
+        &self,
+    ) -> Option<DidChangeWatchedFilesRegistrationOptions> {
+        fn make_watcher(glob: &str) -> FileSystemWatcher {
+            FileSystemWatcher {
+                glob_pattern: lsp_types::GlobPattern::String(glob.into()),
+                kind: Some(lsp_types::WatchKind::all()),
+            }
+        }
+
+        fn make_relative_watcher(relative_to: &SystemPath, glob: &str) -> FileSystemWatcher {
+            let base_uri = Url::from_file_path(relative_to.as_std_path())
+                .expect("system path must be a valid URI");
+            let glob_pattern = lsp_types::GlobPattern::Relative(lsp_types::RelativePattern {
+                base_uri: lsp_types::OneOf::Right(base_uri),
+                pattern: glob.to_string(),
+            });
+            FileSystemWatcher {
+                glob_pattern,
+                kind: Some(lsp_types::WatchKind::all()),
+            }
+        }
+
+        if !self.client_capabilities().supports_file_watcher() {
+            tracing::warn!(
+                "Your LSP client doesn't support file watching: \
+                 You may see stale results when files change outside the editor"
+            );
+            return None;
+        }
+
+        // We also want to watch everything in the search paths as
+        // well. But this seems to require "relative" watcher support.
+        // I had trouble getting this working without using a base uri.
+        //
+        // Specifically, I tried this for each search path:
+        //
+        //     make_watcher(&format!("{path}/**"))
+        //
+        // But while this seemed to work for the project root, it
+        // simply wouldn't result in any file notifications for changes
+        // to files outside of the project root.
+        #[allow(clippy::if_not_else)] // no! it reads better this way ---AG
+        let watchers = if !self.client_capabilities().supports_relative_file_watcher() {
+            tracing::warn!(
+                "Your LSP client doesn't support file watching outside of project: \
+                 You may see stale results when dependencies change"
+            );
+            // Initialize our list of watchers with the standard globs relative
+            // to the project root if we can't use relative globs.
+            vec![make_watcher("**")]
+        } else {
+            // Gather up all of our project roots and all of the corresponding
+            // project root system paths, then deduplicate them relative to
+            // one another. Then listen to everything.
+            let roots = self.project_dbs().map(|db| db.project().root(db));
+            let paths = self
+                .project_dbs()
+                .flat_map(|db| {
+                    ty_python_semantic::system_module_search_paths(db).map(move |path| (db, path))
+                })
+                .filter(|(db, path)| !path.starts_with(db.project().root(*db)))
+                .map(|(_, path)| path)
+                .chain(roots);
+            ruff_db::system::deduplicate_nested_paths(paths)
+                .map(|path| make_relative_watcher(path, "**"))
+                .collect()
+        };
+        Some(DidChangeWatchedFilesRegistrationOptions { watchers })
+    }
+
     /// Creates a document snapshot with the URL referencing the document to snapshot.
     pub(crate) fn take_document_snapshot(&self, url: Url) -> DocumentSnapshot {
         let key = self