[ty] First cut at semantic token provider (#19108)

This PR implements a basic semantic token provider for ty's language
server. This allows for more accurate semantic highlighting / coloring
within editors that support this LSP functionality.

Here are screen shots that show how code appears in VS Code using the
"rainbow" theme both before and after this change.


![461737617-15630625-d4a9-4ec5-9886-77b00eb7a41a](https://github.com/user-attachments/assets/f963b55b-3195-41d1-ba38-ac2e7508d5f5)


![461737624-d6dcf5f0-7b9b-47de-a410-e202c63e2058](https://github.com/user-attachments/assets/111ca2c5-bb4f-4c8a-a0b5-6c1b2b6f246b)

The token types and modifier tags in this implementation largely mirror
those used in Microsoft's default language server for Python.

The implementation supports two LSP interfaces. The first provides
semantic tokens for an entire document, and the second returns semantic
tokens for a requested range within a document.

The PR includes unit tests. It also includes comments that document
known limitations and areas for future improvements.

---------

Co-authored-by: UnboundVariable <unbound@gmail.com>
This commit is contained in:
UnboundVariable 2025-07-07 15:34:47 -07:00 committed by GitHub
parent 4dd2c03144
commit 278f93022a
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
13 changed files with 2221 additions and 3 deletions

View file

@ -5,6 +5,7 @@ mod goto;
mod hover;
mod inlay_hints;
mod markup;
mod semantic_tokens;
pub use completion::completion;
pub use db::Db;
@ -12,6 +13,9 @@ pub use goto::goto_type_definition;
pub use hover::hover;
pub use inlay_hints::inlay_hints;
pub use markup::MarkupKind;
pub use semantic_tokens::{
SemanticToken, SemanticTokenModifier, SemanticTokenType, SemanticTokens, semantic_tokens,
};
use ruff_db::files::{File, FileRange};
use ruff_text_size::{Ranged, TextRange};