language-servers/tinymist
1
0
Fork 0
mirror of https://github.com/Myriad-Dreamin/tinymist.git synced 2025-08-04 10:18:16 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 9

docs: add base documentation (#344)

Browse source
This commit is contained in:
Myriad-Dreamin 2024-06-26 16:33:33 +08:00 • committed by GitHub
parent c7d0ebd3bd
commit 39d6c0affe
No known key found for this signature in database
GPG key ID: B5690EEEBB952194
26 changed files with 2069 additions and 328 deletions
Download patch file Download diff file Expand all files Collapse all files

325
docs/overview.typ
View file

@ -1,325 +0,0 @@
#import "pageless.typ": *
#import "@preview/fletcher:0.4.4" as fletcher: *
#let colors = (blue.lighten(10%), olive, eastern)
#import fletcher.shapes: diamond
#show: project.with(title: "Tinymist")
/// This function is to render a text string in monospace style and function
/// color in your defining themes.
///
/// ## Examples
///
/// ```typc
/// typst-func("list.item")
/// ```
///
/// Note: it doesn't check whether input is a valid function identifier or path.
#let typst-func(it) = [
#raw(it + "()", lang: "typc") <typst-raw-func>
]
#show <typst-raw-func>: it => {
it.lines.at(0).body.children.slice(0, -2).join()
}
This document gives an overview of tinymist service, which provides a single integrated language service for Typst. This document doesn't dive in details unless necessary.
== Principles
Four principles are followed:
/ Multiple Actors: The main component, #link("https://github.com/Myriad-Dreamin/tinymist/tree/main/crates/tinymist")[tinymist], starts as a thread or process, obeying the #link("https://microsoft.github.io/language-server-protocol/")[Language Server Protocol]. tinymist will bootstrap multiple actors, each of which provides some typst feature.
The each actor holds and maintains some resources exclusively. For example, the compile server actor holds the well known ```rs trait World``` resource.
The actors communicate with each other by channels. An actor should own many receivers as its input, and many senders as output. The actor will take input from receivers _sequentially_. For example, when some LSP request or notification is coming as an LSP event, multiple actors serve the event collaboratively, as shown in @fig:actor-serve-lsp-requests.
#figure(
align(
center,
diagram(
edge-stroke: 0.85pt,
node-corner-radius: 3pt,
edge-corner-radius: 4pt,
mark-scale: 80%,
node((0, 0), [LSP Requests/\ Notifications\ (Channel)], fill: colors.at(0), shape: fletcher.shapes.hexagon),
node((2, +1), [RenderActor], fill: colors.at(1)),
node((2, 0), align(center)[`CompileServerActor`], fill: colors.at(1)),
node((2, -1), [`LspActor` (Main Thread)], fill: colors.at(1)),
node((4, 0), [LSP Responses\ (Channel)], fill: colors.at(2), shape: fletcher.shapes.hexagon),
edge((0, 0), "r,u,r", "-}>"),
edge((2, -1), "r,d,r", "-}>"),
edge((2, 0), "rr", "-}>"),
edge((2, 1), "r,u,r", "-}>"),
edge((2, 0), (2, 1), align(center)[Rendering\ Requests], "-}>"),
edge((2, -1), (2, 0), align(center)[Analysis\ Requests], "-}>"),
),
),
caption: [The IO Graph of actors serving a LSP request or notification],
) <fig:actor-serve-lsp-requests>
A _Hover_ request is taken as example of that events.
A global unique `LspActor` takes the event and _mutates_ a global server state by the event. If the event requires some additional code analysis, it is converted into an analysis request, #link("https://github.com/search?q=repo%3AMyriad-Dreamin/tinymist%20CompilerQueryRequest&type=code")[```rs struct CompilerQueryRequest```], and pushed to the actors owning compiler resources. Otherwise, `LspActor` responds to the event directly. Obviously, the _Hover_ on code request requires code analysis.
The `CompileServerActor`s are created for workspaces and main entries (files/documents) in workspaces. When a compiler query is coming, a subset of that actors will take it and give project-specific responses, combining into a final concluded LSP response. Some analysis requests even require rendering features, and those requests will be pushed to the actors owning rendering resources. If you enable the periscope feature, a `Hover` on content request requires rendering on documents.
The `RenderActor`s don't do compilations, but own project-specific rendering cache. They are designed for rendering documents in _low latency_. This is the last sink of `Hover` requests. A `RenderActor` will receive an additional compiled `Document` object, and render the compiled frames in needed. After finishing rendering, a response attached with the rendered picture is sent to the LSP response channel intermediately.
/ Multi-level Analysis: The most critical features are lsp functions, built on the #link("https://github.com/Myriad-Dreamin/tinymist/tree/main/crates/tinymist-query")[tinymist-query] crate. To achieve higher concurrency, functions are classified into different levels of analysis.
// + `query_token_cache` – `TokenRequest` – locks and accesses token cache.
+ `query_source` – `SyntaxRequest` – locks and accesses a single source unit.
+ `query_world` – `SemanticRequest` – locks and accesses multiple source units.
+ `query_state` – `StatefulRequest` – acquires to accesses a specific version of compile results.
When an analysis request is coming, tinymist _upgrades_ it to a suitable level as needed, as shown in @fig:analysis-upgrading-level. A higher level requires to hold more resources and takes longer time to prepare.
#let pg-node = node.with(corner-radius: 2pt, shape: "rect");
#figure(
align(
center,
diagram(
node-stroke: 1pt,
edge-stroke: 1pt,
edge("-|>", align(center)[Analysis\ Request], label-pos: 0.1),
pg-node((1, 0), [Syntax\ Level]),
edge("-|>", []),
pg-node((3, 0), [Semantic\ Level]),
edge("-|>"),
pg-node((5, 0), [Stateful\ Level]),
edge((5, 0), (6, 0), "-|>", align(center)[Analysis\ Response], label-pos: 1),
for i in (1, 3, 5) {
edge((i, 0), (i, -0.5), (5.5, -0.5), (5.6, 0), "-|>")
},
edge(
(0.3, 0.4),
(0.3, 0),
"-|>",
align(center)[clone #typst-func("Source")],
label-anchor: "center",
label-pos: -0.5,
),
edge(
(2, 0.4),
(2, 0),
"-|>",
align(center)[snapshot ```rs trait World```],
label-anchor: "center",
label-pos: -0.5,
),
edge(
(4, 0.4),
(4, 0),
"-|>",
align(center)[acquire #typst-func("Document")],
label-anchor: "center",
label-pos: -0.5,
),
),
),
caption: [The analyzer upgrades the level to acquire necessary resources],
) <fig:analysis-upgrading-level>
/ Optional Non-LSP Features: All non-LSP features in tinymist are *optional*. They are optional, as they can be disabled *totally* on compiling the tinymist binary. The significant features are enabled by default, but you can disable them with feature flags. For example, `tinymist` provides preview server features powered by `typst-preview`.
/ Minimal Editor Frontends: Leveraging the interface of LSP, tinymist provides frontends to each editor, located in the #link("https://github.com/Myriad-Dreamin/tinymist/tree/main/editors")[editor folders]. They are minimal, meaning that LSP should finish its main LSP features as many as possible without help of editor frontends. The editor frontends just enhances your code experience. For example, the vscode frontend takes responsibility on providing some nice editor tools. It is recommended to install these editors frontend for your editors.
== Command System
The extra features are exposed via LSP's #link("https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#workspace_executeCommand")[`workspace/executeCommand`] request, forming a command system. The commands in the system share a name convention.
- `export`#text(olive, `Fmt`). these commands perform export on some document, with a specific format (#text(olive, `Fmt`)), e.g. `exportPdf`.
- `interactCodeContext({`#text(olive, `kind`)`}[])`. The code context requests are useful for _Editor Frontends_ to extend some semantic actions. A batch of requests are sent at the same time, to get code context _atomically_.
- `getResources(`#text(olive, `"path/to/resource/"`)`, `#text(red, `opts`)`)`. The resources required by _Editor Frontends_ should be arranged in #text(olive, "paths"). A second arguments can be passed as options to request a resource. This resemebles a restful `POST` action to LSP, with a url #text(olive, "path") and a HTTP #text(red, "body"), or a RPC with a #text(olive, "method name") and #text(red, "params").
Note you can also hide some commands in list of commands in UI by putting them in `getResources` command.
- `do`#text(olive, `Xxx`). these commands are internally for _Editor Frontends_, and you'd better not to invoke them directly. You can still invoke them manually, as long as you know what would happen.
- The rest commands are public and tend to be user-friendly.
// === Stateful Commands
// Two styles are made for stateful commands.
=== Code Context
The code context requests are useful for _Editor Frontends_ to check syntax and semantic the multiple positions. For example an editor frontend can filter some completion list by acquire the code context at current position.
== Handling Input Events
The compilation triggers many side effects, but the behavior of compiler actor is still easy to predicate. This is achieved by accepting all compile inputs by events.
Let us take reading files from physical file system as example of processing compile inputs, as shown in @fig:overlay-vfs. The upper access models take precedence over the lower access models. The memory access model is updated _sequentially_ by `LspActor` receiving source change notifications, assigned with logical ticks $t_(L,n)$. The notify access model is also updated in same way by `NotifyActor`. When there is an absent access, the system access model initiates the request for the file system directly. The read contents from fs are assigned with logical access time $t_(M,n)$.
#let pg-hori-sep = 1.5
#let pg-vert-sep = 0.7
#let pg-adjust = 18pt
#figure(
align(
center,
move(
dx: pg-adjust,
diagram(
edge-stroke: 0.85pt,
node-corner-radius: 3pt,
edge-corner-radius: 4pt,
mark-scale: 80%,
node((pg-hori-sep, +pg-vert-sep), [SystemAccessModel], fill: colors.at(1)),
node((pg-hori-sep, 0), align(center)[`NotifyAccessModel`], fill: colors.at(1)),
node((pg-hori-sep, -pg-vert-sep), [MemoryAccessModel], fill: colors.at(1)),
node((0, 0), align(center)[`NotifyActor`], fill: colors.at(0)),
node((0, -pg-vert-sep), align(center)[`LspActor`], fill: colors.at(0)),
edge((0, 0), (pg-hori-sep, 0), "-}>"),
edge((0, -pg-vert-sep), (pg-hori-sep, -pg-vert-sep), "-}>"),
edge(
(-1, -pg-vert-sep),
(0, -pg-vert-sep),
"-}>",
[didChange, \ didOpen, etc.],
label-anchor: "center",
label-pos: 0,
),
edge(
(-0.8, pg-vert-sep),
(0, pg-vert-sep),
(0, 0),
"-}>",
[readFile\ readDir, etc.],
label-anchor: "center",
label-pos: 0,
),
edge((-1, pg-vert-sep), (pg-hori-sep, pg-vert-sep), "-}>"),
edge((pg-hori-sep, 0), (pg-hori-sep, pg-vert-sep), "-}>"),
edge((pg-hori-sep, -pg-vert-sep), (pg-hori-sep, 0), "-}>"),
edge(
(pg-hori-sep * 1.59, -pg-vert-sep * 1.6),
(pg-hori-sep, -pg-vert-sep * 1.6),
(pg-hori-sep, -pg-vert-sep),
"-}>",
[sourceOf(path)],
label-pos: 0.2,
),
for i in (-1, 0, 1) {
edge(
(pg-hori-sep * 1.2, i * pg-vert-sep),
(pg-hori-sep * 1.7, i * pg-vert-sep),
"-}>",
[source],
label-pos: 1,
)
},
node(
(-1.3, 0),
rotate(-90deg, rect(stroke: (bottom: (thickness: 1pt, dash: "dashed")), width: 120pt)[Input Sources]),
),
node(
(pg-hori-sep + 1.45, 0),
rotate(
90deg,
move(
dy: pg-adjust * 2,
rect(stroke: (bottom: (thickness: 1pt, dash: "dashed")), width: 120pt)[Compiler World],
),
),
),
),
),
),
caption: [The overlay virtual file system (VFS)],
) <fig:overlay-vfs>
The problem is to ensure that the compiler can read the content correctly from access models at the time.
If there is an only active input source in a _small time window_, we can know the problem is solved, as the logical ticks $t_(L,n)$ and $t_(M,n)$ keep increasing, enforced by actors. For example, if there is only `LspActor` active at the _small time window_, the memory access model receives the source changes in the order of $t_(L,n)$, i.e. the _sequential_ order of receiving notifications. The cases of two rest access models is more complicated, but are also ensured that compiler reads content in order of $t_(M,n)$.
Otherwise, the two input sources are both active in a _small time window_ on a *same file*. However, this indicates that, the file is in already the memory access model at most time. Since the precedence, the compiler reads content in order of $t_(L,n)$ at the time.
The only bad case can happen is that: When the two input sources are both active in a _small time window_ $delta$ on a *same file*:
- first `LspActor` removes the file from the memory access model, then compiler doesn't read content from file system in time $delta$.
- first `NotifyActor` inserts the file from the inotify thread, then the LSP client (editor) overlays an older content in time $delta$.
This is handled by tinymist by some tricks.
=== Record and Replay
Tinymist can record these input events with assigned the logic ticks. By replaying the events, tinymist can reproduce the server state for debugging. This technique is learnt from the well-known LSP, clangd, and the well known emulator, QEMU.
== Additional Concepts for Typst Language
=== AST Matchers
Many analyzers don't check AST node relationships directly. The AST matchers provide some indirect structure for analyzers.
- Most code checks the syntax object matched by `get_deref_target` or `get_check_target`.
- The folding range analyzer and def-use analyzer check the source file on the structure named _lexical hierarchy_.
- The type checker checks constraint collected by a trivial node-to-type converter.
=== Type System
The underlying techniques are not easy to understand, but there are some links:
- bidirectional type checking: https://jaked.org/blog/2021-09-15-Reconstructing-TypeScript-part-1
- type system borrowed here: https://github.com/hkust-taco/mlscript
Some tricks are taken for help reducing the complexity of code:
First, the array literals are identified as tuple type, that each cell of the array has type individually.
#let sig = $sans("sig")$
#let ags = $sans("args")$
Second, the $sig$ and the $sans("argument")$ type are reused frequently.
- the $sans("tup")$ type is notated as $(tau_1,..,tau_n)$, and the $sans("arr")$ type is a special tuple type $sans("arr") ::= sans("arr")(tau)$.
- the $sans("rec")$ type is imported from #link("https://github.com/hkust-taco/mlscript")[mlscript], notated as ${a_1=tau_1,..,a_n=tau_n}$.
- the $sig$ type consists of:
- a positional argument list, in $sans("tup")$ type.
- a named argument list, in $sans("rec")$ type.
- an optional rest argument, in $sans("arr")$ type.
- an *optional* body, in any type.
notated as $sig := sig(sans("tup")(tau_1,..,tau_n),sans("rec")(a_1=tau_(n+1),..,a_m=tau_(n+m)),..sans("arr")(tau_(n+m+1))) arrow psi$
- the $sans("argument")$ is a $sans("signature")$ without rest and body.
$ags := ags(sig(..))$
With aboving constructors, we soonly get typst's type checker.
- it checks array or dictionary literals by converting them with a corresponding $sig$ and $ags$.
- it performs the getting element operation by calls a corresponding $sig$.
- the closure is converted into a typed lambda, in $sig$ type.
- the pattern destructing are converted to array and dictionary constrains.
== Notes on implementing LSP Features
There are five basic analysis APIs:
- _lexical hierarchy_ matches crucial lexical structures in the source file.
- _def use info_ is computed based on _lexical hierarchy_\s.
- _type check info_ is computed with _def use info_.
- _find definition_ finds the definition based on _def use info_.
- _find references_ finds the references based on _def use info_.
The features are implemented based on basic analysis APIs:
- The `textDocument/documentSymbol` returns a tree of nodes converted from the _lexical hierarchy_.
- The `textDocument/foldingRange` also returns a tree of nodes converted from the _lexical hierarchy_ but with a different approach.
- The `workspace/symbol` returns an array of nodes converted from all _lexical hierarchy_\s in workspace.
- The `textDocument/definition` returns the result of _find definition_.
- The `textDocument/completion` returns a list of types of _related_ nodes according to _type check info_, matched by _AST matchers_.
- The `textDocument/hover` _finds definition_ and prints the definition with a checked type by _type check info_. Or, specific to typst, prints a set of inspected values during execution of the document.
- The `textDocument/signatureHelp` also _finds definition_ and prints the signature with union of inferred signatures by _type check info_.
- The `textDocument/prepareRename` _finds definition_ and determines whether it can be renamed.
- The `textDocument/rename` _finds defintion and references_ and renamed them all.

34
docs/pageless.typ
View file

@ -1,34 +0,0 @@
// The project function defines how your document looks.
// It takes your content and some metadata and formats it.
// Go ahead and customize it to your liking!
#let project(title: "", authors: (), mode: "paged", body) = {
assert(mode in ("paged", "pageless"), message: "invalid mode")
// Set the document's basic properties.
set document(author: authors, title: title)
set page(
height: auto,
width: 210mm,
// numbering: "1", number-align: center,
) if mode == "pageless"
set text(font: ("Linux Libertine", "Source Han Serif SC", "Source Han Sans"), size: 12pt, lang: "en")
set page(height: 297mm)
// Title row.
align(center)[
#block(text(weight: 700, 1.75em, title))
]
// Main body.
set par(justify: true)
// rules
show raw.where(block: true): set par(justify: false)
show raw.where(block: true): rect.with(width: 100%, radius: 2pt, fill: luma(240), stroke: 0pt)
show link: text.with(blue)
show link: underline
body
}

32
docs/tinymist/book.typ Normal file
View file

@ -0,0 +1,32 @@
#import "@preview/shiroa:0.1.0": *
#show: book
#book-meta(
title: "Tinymist Documentation",
description: "The documentation for tinymist servaice",
authors: ("Myriad-Dreamin",),
repository-edit: "https://github.com/Myriad-Dreamin/tinymist/edit/main/{path}",
language: "en",
summary: [
#prefix-chapter("overview.typ")[Overview]
- #chapter("principles.typ")[Principles]
- #chapter("commands.typ")[Commands System]
- #chapter("inputs.typ")[LSP Inputs]
- #chapter("type-system.typ")[Type System]
- #chapter("language-features.typ")[Language Features]
#prefix-chapter("configurations.typ")[Common Configurations]
#prefix-chapter("frontend/main.typ")[Editor Frontends]
- #chapter("frontend/vscode.typ")[VS Cod(e,ium)]
- #chapter("frontend/neovim.typ")[NeoVim]
- #chapter("frontend/helix.typ")[Helix]
- #chapter("frontend/zed.typ")[Zed]
],
)
#build-meta(dest-dir: "../../dist/tinymist")
// re-export page template
#import "/typ/templates/page.typ": project
#let book-page = project

25
docs/tinymist/commands.typ Normal file
View file

@ -0,0 +1,25 @@
#import "mod.typ": *
#show: book-page.with(title: "Tinymist Command System")
The extra features are exposed via LSP's #link("https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#workspace_executeCommand")[`workspace/executeCommand`] request, forming a command system. The commands in the system share a name convention.
- `export`#text(olive, `Fmt`). these commands perform export on some document, with a specific format (#text(olive, `Fmt`)), e.g. `exportPdf`.
- `interactCodeContext({`#text(olive, `kind`)`}[])`. The code context requests are useful for _Editor Frontends_ to extend some semantic actions. A batch of requests are sent at the same time, to get code context _atomically_.
- `getResources(`#text(olive, `"path/to/resource/"`)`, `#text(red, `opts`)`)`. The resources required by _Editor Frontends_ should be arranged in #text(olive, "paths"). A second arguments can be passed as options to request a resource. This resemebles a restful `POST` action to LSP, with a url #text(olive, "path") and a HTTP #text(red, "body"), or a RPC with a #text(olive, "method name") and #text(red, "params").
Note you can also hide some commands in list of commands in UI by putting them in `getResources` command.
- `do`#text(olive, `Xxx`). these commands are internally for _Editor Frontends_, and you'd better not to invoke them directly. You can still invoke them manually, as long as you know what would happen.
- The rest commands are public and tend to be user-friendly.
// === Stateful Commands
// Two styles are made for stateful commands.
=== Code Context
The code context requests are useful for _Editor Frontends_ to check syntax and semantic the multiple positions. For example an editor frontend can filter some completion list by acquire the code context at current position.

42
docs/tinymist/configurations.typ Normal file
View file

@ -0,0 +1,42 @@
#import "mod.typ": *
#show: book-page.with(title: "Tinymist Configurations")
#let packages = json("/editors/vscode/package.json")
#let config-type(t) = if type(t) == array {
raw(t.join(" | "))
} else {
raw(t)
}
#let config_item(key, cfg) = [
+ *#raw(key)*:
- Type: #config-type(cfg.type)
#if cfg.type == "array" [
- Items: #raw(cfg.items.type)
- Description: #eval(cfg.items.description, mode: "markup")
]
- Description: #eval(cfg.description, mode: "markup")
#if cfg.at("enum", default: none) != none [
- Valid values: #for (i, item) in cfg.enum.enumerate() [
- #raw(item): #if "enumDescriptions" in cfg { eval(cfg.enumDescriptions.at(i), mode: "markup") }
]
]
#if type(cfg.default) == str {
if cfg.default != "" [
- Default: #raw(cfg.default)
] else [
- Default: `""`
]
} else if type(cfg.default) == array [
- Default: [#cfg.default.join(",")]
] else [
- Default: #cfg.default
]
]
#for (key, cfg) in packages.contributes.configuration.properties {
config_item(key, cfg)
}

8
docs/tinymist/ebook.typ Normal file
View file

@ -0,0 +1,8 @@
#import "@preview/shiroa:0.1.0": *
#import "/typ/templates/ebook.typ"
#show: ebook.project.with(title: "tinymist", spec: "book.typ")
// set a resolver for inclusion
#ebook.resolve-inclusion(it => include it)

5
docs/tinymist/frontend/helix.typ Normal file
View file

@ -0,0 +1,5 @@
#import "mod.typ": *
#show: book-page.with(title: "Tinymist Helix Extension")
#link("https://github.com/Myriad-Dreamin/tinymist/tree/main/editors/helix")[Helix]

11
docs/tinymist/frontend/main.typ Normal file
View file

@ -0,0 +1,11 @@
#import "mod.typ": *
#show: book-page.with(title: "Tinymist Editor Frontends")
Leveraging the interface of LSP, tinymist provides frontends to each editor, located in the #link("https://github.com/Myriad-Dreamin/tinymist/tree/main/editors")[editor folders]. They are minimal, meaning that LSP should finish its main LSP features as many as possible without help of editor frontends. The editor frontends just enhances your code experience. For example, the vscode frontend takes responsibility on providing some nice editor tools. It is recommended to install these editors frontend for your editors.
Check the following chapters for uses:
- #cross-link("/frontend/vscode.typ")[VS Cod(e,ium)]
- #cross-link("/frontend/neovim.typ")[NeoVim]
- #cross-link("/frontend/helix.typ")[Helix]
- #cross-link("/frontend/zed.typ")[Zed]

1
docs/tinymist/frontend/mod.typ Normal file
View file

@ -0,0 +1 @@
#import "../mod.typ": *

5
docs/tinymist/frontend/neovim.typ Normal file
View file

@ -0,0 +1,5 @@
#import "mod.typ": *
#show: book-page.with(title: "Tinymist Neovim Extension")
#link("https://github.com/Myriad-Dreamin/tinymist/tree/main/editors/neovim")[Neovim]

5
docs/tinymist/frontend/vscode.typ Normal file
View file

@ -0,0 +1,5 @@
#import "mod.typ": *
#show: book-page.with(title: "Tinymist VS Code Extension")
#link("https://github.com/Myriad-Dreamin/tinymist/tree/main/editors/vscode")[VS Code]

5
docs/tinymist/frontend/zed.typ Normal file
View file

@ -0,0 +1,5 @@
#import "mod.typ": *
#show: book-page.with(title: "Tinymist Neovim Extension")
#link("https://github.com/WeetHet/typst.zed")[Zed]

106
docs/tinymist/inputs.typ Normal file
View file

@ -0,0 +1,106 @@
#import "mod.typ": *
#show: book-page.with(title: "Tinymist LSP Inputs")
== Prefer to Using LSP Configurations
Though tinymist doesn't refuse to keep state in your disk, it actually doesn't have any data to write to disk yet. All customized behaviors (user settings) are passed to the server by LSP configurations. This is a good practice to keep the server state clean and simple.
== Handling Compiler Input Events
The compilation triggers many side effects, but the behavior of compiler actor is still easy to predicate. This is achieved by accepting all compile inputs by events.
Let us take reading files from physical file system as example of processing compile inputs, as shown in @fig:overlay-vfs. The upper access models take precedence over the lower access models. The memory access model is updated _sequentially_ by `LspActor` receiving source change notifications, assigned with logical ticks $t_(L,n)$. The notify access model is also updated in same way by `NotifyActor`. When there is an absent access, the system access model initiates the request for the file system directly. The read contents from fs are assigned with logical access time $t_(M,n)$.
#let pg-hori-sep = 1.5
#let pg-vert-sep = 0.7
#let pg-adjust = 18pt
#figure(
align(
center,
move(
dx: pg-adjust,
diagram(
edge-stroke: 0.85pt,
node-corner-radius: 3pt,
edge-corner-radius: 4pt,
mark-scale: 80%,
node((pg-hori-sep, +pg-vert-sep), [SystemAccessModel], fill: colors.at(1)),
node((pg-hori-sep, 0), align(center)[`NotifyAccessModel`], fill: colors.at(1)),
node((pg-hori-sep, -pg-vert-sep), [MemoryAccessModel], fill: colors.at(1)),
node((0, 0), align(center)[`NotifyActor`], fill: colors.at(0)),
node((0, -pg-vert-sep), align(center)[`LspActor`], fill: colors.at(0)),
edge((0, 0), (pg-hori-sep, 0), "-}>"),
edge((0, -pg-vert-sep), (pg-hori-sep, -pg-vert-sep), "-}>"),
edge(
(-1, -pg-vert-sep),
(0, -pg-vert-sep),
"-}>",
[didChange, \ didOpen, etc.],
label-anchor: "center",
label-pos: 0,
),
edge(
(-0.8, pg-vert-sep),
(0, pg-vert-sep),
(0, 0),
"-}>",
[readFile\ readDir, etc.],
label-anchor: "center",
label-pos: 0,
),
edge((-1, pg-vert-sep), (pg-hori-sep, pg-vert-sep), "-}>"),
edge((pg-hori-sep, 0), (pg-hori-sep, pg-vert-sep), "-}>"),
edge((pg-hori-sep, -pg-vert-sep), (pg-hori-sep, 0), "-}>"),
edge(
(pg-hori-sep * 1.59, -pg-vert-sep * 1.6),
(pg-hori-sep, -pg-vert-sep * 1.6),
(pg-hori-sep, -pg-vert-sep),
"-}>",
[sourceOf(path)],
label-pos: 0.2,
),
for i in (-1, 0, 1) {
edge(
(pg-hori-sep * 1.2, i * pg-vert-sep),
(pg-hori-sep * 1.7, i * pg-vert-sep),
"-}>",
[source],
label-pos: 1,
)
},
node(
(-1.3, 0),
rotate(-90deg, rect(stroke: (bottom: (thickness: 1pt, dash: "dashed")), width: 120pt)[Input Sources]),
),
node(
(pg-hori-sep + 1.45, 0),
rotate(
90deg,
move(
dy: pg-adjust * 2,
rect(stroke: (bottom: (thickness: 1pt, dash: "dashed")), width: 120pt)[Compiler World],
),
),
),
),
),
),
caption: [The overlay virtual file system (VFS)],
) <fig:overlay-vfs>
The problem is to ensure that the compiler can read the content correctly from access models at the time.
If there is an only active input source in a _small time window_, we can know the problem is solved, as the logical ticks $t_(L,n)$ and $t_(M,n)$ keep increasing, enforced by actors. For example, if there is only `LspActor` active at the _small time window_, the memory access model receives the source changes in the order of $t_(L,n)$, i.e. the _sequential_ order of receiving notifications. The cases of two rest access models is more complicated, but are also ensured that compiler reads content in order of $t_(M,n)$.
Otherwise, the two input sources are both active in a _small time window_ on a *same file*. However, this indicates that, the file is in already the memory access model at most time. Since the precedence, the compiler reads content in order of $t_(L,n)$ at the time.
The only bad case can happen is that: When the two input sources are both active in a _small time window_ $delta$ on a *same file*:
- first `LspActor` removes the file from the memory access model, then compiler doesn't read content from file system in time $delta$.
- first `NotifyActor` inserts the file from the inotify thread, then the LSP client (editor) overlays an older content in time $delta$.
This is handled by tinymist by some tricks.
=== Record and Replay
Tinymist can record these input events with assigned the logic ticks. By replaying the events, tinymist can reproduce the server state for debugging. This technique is learnt from the well-known LSP, clangd, and the well known emulator, QEMU.

34
docs/tinymist/language-features.typ Normal file
View file

@ -0,0 +1,34 @@
#import "mod.typ": *
#show: book-page.with(title: "Tinymist Language Features")
== Base Analyses
There are five basic analysis APIs:
- _lexical hierarchy_ matches crucial lexical structures in the source file.
- _def use info_ is computed based on _lexical hierarchy_\s.
- _type check info_ is computed with _def use info_.
- _find definition_ finds the definition based on _def use info_.
- _find references_ finds the references based on _def use info_.
== Extending Language Features
Language features are implemented based on basic analysis APIs:
- The `textDocument/documentSymbol` returns a tree of nodes converted from the _lexical hierarchy_.
- The `textDocument/foldingRange` also returns a tree of nodes converted from the _lexical hierarchy_ but with a different approach.
- The `workspace/symbol` returns an array of nodes converted from all _lexical hierarchy_\s in workspace.
- The `textDocument/definition` returns the result of _find definition_.
- The `textDocument/completion` returns a list of types of _related_ nodes according to _type check info_, matched by _AST matchers_.
- The `textDocument/hover` _finds definition_ and prints the definition with a checked type by _type check info_. Or, specific to typst, prints a set of inspected values during execution of the document.
- The `textDocument/signatureHelp` also _finds definition_ and prints the signature with union of inferred signatures by _type check info_.
- The `textDocument/prepareRename` _finds definition_ and determines whether it can be renamed.
- The `textDocument/rename` _finds defintion and references_ and renamed them all.

24
docs/tinymist/mod.typ Normal file
View file

@ -0,0 +1,24 @@
#import "/docs/tinymist/book.typ": book-page, cross-link
#import "@preview/fletcher:0.4.4" as fletcher: *
/// This function is to render a text string in monospace style and function
/// color in your defining themes.
///
/// ## Examples
///
/// ```typc
/// typst-func("list.item")
/// ```
///
/// Note: it doesn't check whether input is a valid function identifier or path.
#let typst-func(it) = [
#raw(it + "()", lang: "typc") <typst-raw-func>
]
#show <typst-raw-func>: it => {
it.lines.at(0).body.children.slice(0, -2).join()
}
#let colors = (blue.lighten(10%), olive, eastern)
#import fletcher.shapes: diamond

37
docs/tinymist/overview.typ Normal file
View file

@ -0,0 +1,37 @@
#import "mod.typ": *
#show: book-page.with(title: "Tinymist")
This document gives an overview of tinymist service, which provides a single integrated language service for Typst. This document doesn't dive in details unless necessary.
== Principles
Four principles are followed, as detailed in #cross-link("/principles.typ")[Principles].
- Multiple Actors
- Multi-level Analysis
- Optional Non-LSP Features
- Minimal Editor Frontends
== Command System
The extra features are exposed via LSP's #link("https://microsoft.github.io/language-server-protocol/specifications/lsp/3.17/specification/#workspace_executeCommand")[`workspace/executeCommand`] request, forming a command system. They are detailed in #cross-link("/commands.typ")[Command System].
== Additional Concepts for Typst Language
=== AST Matchers
Many analyzers don't check AST node relationships directly. The AST matchers provide some indirect structure for analyzers.
- Most code checks the syntax object matched by `get_deref_target` or `get_check_target`.
- The folding range analyzer and def-use analyzer check the source file on the structure named _lexical hierarchy_.
- The type checker checks constraint collected by a trivial node-to-type converter.
=== Type System
Check #cross-link("/type-system.typ")[Type System] for more details.
== Notes on Implementing Language Features
Five basic analysis like _lexical hierarchy_, _def use info_ and _type check info_ are implemented first. And all rest Language features are implemented based on basic analysis. Check #cross-link("/analyses.typ")[Analyses] for more details.

109
docs/tinymist/principles.typ Normal file
View file

@ -0,0 +1,109 @@
#import "mod.typ": *
#show: book-page.with(title: "Tinymist Principles")
Four principles are followed.
== Multiple Actors
The main component, #link("https://github.com/Myriad-Dreamin/tinymist/tree/main/crates/tinymist")[tinymist], starts as a thread or process, obeying the #link("https://microsoft.github.io/language-server-protocol/")[Language Server Protocol]. tinymist will bootstrap multiple actors, each of which provides some typst feature.
The each actor holds and maintains some resources exclusively. For example, the compile server actor holds the well known ```rs trait World``` resource.
The actors communicate with each other by channels. An actor should own many receivers as its input, and many senders as output. The actor will take input from receivers _sequentially_. For example, when some LSP request or notification is coming as an LSP event, multiple actors serve the event collaboratively, as shown in @fig:actor-serve-lsp-requests.
#figure(
align(
center,
diagram(
edge-stroke: 0.85pt,
node-corner-radius: 3pt,
edge-corner-radius: 4pt,
mark-scale: 80%,
node((0, 0), [LSP Requests/\ Notifications\ (Channel)], fill: colors.at(0), shape: fletcher.shapes.hexagon),
node((2, +1), [RenderActor], fill: colors.at(1)),
node((2, 0), align(center)[`CompileServerActor`], fill: colors.at(1)),
node((2, -1), [`LspActor` (Main Thread)], fill: colors.at(1)),
node((4, 0), [LSP Responses\ (Channel)], fill: colors.at(2), shape: fletcher.shapes.hexagon),
edge((0, 0), "r,u,r", "-}>"),
edge((2, -1), "r,d,r", "-}>"),
edge((2, 0), "rr", "-}>"),
edge((2, 1), "r,u,r", "-}>"),
edge((2, 0), (2, 1), align(center)[Rendering\ Requests], "-}>"),
edge((2, -1), (2, 0), align(center)[Analysis\ Requests], "-}>"),
),
),
caption: [The IO Graph of actors serving a LSP request or notification],
) <fig:actor-serve-lsp-requests>
A _Hover_ request is taken as example of that events.
A global unique `LspActor` takes the event and _mutates_ a global server state by the event. If the event requires some additional code analysis, it is converted into an analysis request, #link("https://github.com/search?q=repo%3AMyriad-Dreamin/tinymist%20CompilerQueryRequest&type=code")[```rs struct CompilerQueryRequest```], and pushed to the actors owning compiler resources. Otherwise, `LspActor` responds to the event directly. Obviously, the _Hover_ on code request requires code analysis.
The `CompileServerActor`s are created for workspaces and main entries (files/documents) in workspaces. When a compiler query is coming, a subset of that actors will take it and give project-specific responses, combining into a final concluded LSP response. Some analysis requests even require rendering features, and those requests will be pushed to the actors owning rendering resources. If you enable the periscope feature, a `Hover` on content request requires rendering on documents.
The `RenderActor`s don't do compilations, but own project-specific rendering cache. They are designed for rendering documents in _low latency_. This is the last sink of `Hover` requests. A `RenderActor` will receive an additional compiled `Document` object, and render the compiled frames in needed. After finishing rendering, a response attached with the rendered picture is sent to the LSP response channel intermediately.
== Multi-level Analysis
he most critical features are lsp functions, built on the #link("https://github.com/Myriad-Dreamin/tinymist/tree/main/crates/tinymist-query")[tinymist-query] crate. To achieve higher concurrency, functions are classified into different levels of analysis.
// + `query_token_cache` – `TokenRequest` – locks and accesses token cache.
+ `query_source` – `SyntaxRequest` – locks and accesses a single source unit.
+ `query_world` – `SemanticRequest` – locks and accesses multiple source units.
+ `query_state` – `StatefulRequest` – acquires to accesses a specific version of compile results.
When an analysis request is coming, tinymist _upgrades_ it to a suitable level as needed, as shown in @fig:analysis-upgrading-level. A higher level requires to hold more resources and takes longer time to prepare.
#let pg-node = node.with(corner-radius: 2pt, shape: "rect");
#figure(
align(
center,
diagram(
node-stroke: 1pt,
edge-stroke: 1pt,
edge("-|>", align(center)[Analysis\ Request], label-pos: 0.1),
pg-node((1, 0), [Syntax\ Level]),
edge("-|>", []),
pg-node((3, 0), [Semantic\ Level]),
edge("-|>"),
pg-node((5, 0), [Stateful\ Level]),
edge((5, 0), (6, 0), "-|>", align(center)[Analysis\ Response], label-pos: 1),
for i in (1, 3, 5) {
edge((i, 0), (i, -0.5), (5.5, -0.5), (5.6, 0), "-|>")
},
edge(
(0.3, 0.4),
(0.3, 0),
"-|>",
align(center)[clone #typst-func("Source")],
label-anchor: "center",
label-pos: -0.5,
),
edge(
(2, 0.4),
(2, 0),
"-|>",
align(center)[snapshot ```rs trait World```],
label-anchor: "center",
label-pos: -0.5,
),
edge(
(4, 0.4),
(4, 0),
"-|>",
align(center)[acquire #typst-func("Document")],
label-anchor: "center",
label-pos: -0.5,
),
),
),
caption: [The analyzer upgrades the level to acquire necessary resources],
) <fig:analysis-upgrading-level>
== Optional Non-LSP Features
All non-LSP features in tinymist are *optional*. They are optional, as they can be disabled *totally* on compiling the tinymist binary. The significant features are enabled by default, but you can disable them with feature flags. For example, `tinymist` provides preview server features powered by `typst-preview`.
== Minimal Editor Frontends
Leveraging the interface of LSP, tinymist provides frontends to each editor, located in the #link("https://github.com/Myriad-Dreamin/tinymist/tree/main/editors")[editor folders]. They are minimal, meaning that LSP should finish its main LSP features as many as possible without help of editor frontends. The editor frontends just enhances your code experience. For example, the vscode frontend takes responsibility on providing some nice editor tools. It is recommended to install these editors frontend for your editors.

38
docs/tinymist/type-system.typ Normal file
View file

@ -0,0 +1,38 @@
#import "mod.typ": *
#show: book-page.with(title: "Tinymist Type System")
The underlying techniques are not easy to understand, but there are some links:
- bidirectional type checking: https://jaked.org/blog/2021-09-15-Reconstructing-TypeScript-part-1
- type system borrowed here: https://github.com/hkust-taco/mlscript
Some tricks are taken for help reducing the complexity of code:
First, the array literals are identified as tuple type, that each cell of the array has type individually.
#let sig = $sans("sig")$
#let ags = $sans("args")$
Second, the $sig$ and the $sans("argument")$ type are reused frequently.
- the $sans("tup")$ type is notated as $(tau_1,..,tau_n)$, and the $sans("arr")$ type is a special tuple type $sans("arr") ::= sans("arr")(tau)$.
- the $sans("rec")$ type is imported from #link("https://github.com/hkust-taco/mlscript")[mlscript], notated as ${a_1=tau_1,..,a_n=tau_n}$.
- the $sig$ type consists of:
- a positional argument list, in $sans("tup")$ type.
- a named argument list, in $sans("rec")$ type.
- an optional rest argument, in $sans("arr")$ type.
- an *optional* body, in any type.
notated as $sig := sig(sans("tup")(tau_1,..,tau_n),sans("rec")(a_1=tau_(n+1),..,a_m=tau_(n+m)),..sans("arr")(tau_(n+m+1))) arrow psi$
- the $sans("argument")$ is a $sans("signature")$ without rest and body.
$ags := ags(sig(..))$
With aboving constructors, we soonly get typst's type checker.
- it checks array or dictionary literals by converting them with a corresponding $sig$ and $ags$.
- it performs the getting element operation by calls a corresponding $sig$.
- the closure is converted into a typed lambda, in $sig$ type.
- the pattern destructing are converted to array and dictionary constrains.