Commit graph

4934 commits

Author SHA1 Message Date
Pere Diaz Bou
7068ba2908 make callbacks threadsafe with simple spin lock 2025-06-05 16:44:29 +02:00
Pekka Enberg
9f17be8162 Merge 'simulator: log the interaction about to be executed with INFO' from Jussi Saurio
I think this is useful to see, without having to drop down to DEBUG and
seeing all the other stuff.

Closes #1655
2025-06-04 10:20:54 +03:00
Pekka Enberg
7376284895 Merge 'simulator: add CREATE INDEX to interactions' from Jussi Saurio
Adds CREATE INDEX as a potential interaction.
Kudos to @alpaylan - I've been sleeping on the simulator a bit lately --
incredibly easy to add new stuff here

Closes #1656
2025-06-04 10:15:48 +03:00
Pekka Enberg
941ecdc75b Merge 'stress: Use temporary file unless one explicitly specified' from Jussi Saurio
Closes #1660
2025-06-04 10:12:31 +03:00
Jussi Saurio
7e875a44cb add limbostress.log to gitignore 2025-06-04 10:02:37 +03:00
Jussi Saurio
062469936d test/stress: use tempfile unless opts.db_file given 2025-06-04 10:02:27 +03:00
Jussi Saurio
2087393d22 Merge 'Write database header via normal pager route' from meteorgan
Closes: #1613

Closes #1634
2025-06-04 09:39:14 +03:00
Jussi Saurio
2d638e737c Merge 'sim/aws: fix sim timeout handling' from Jussi Saurio
code was trying to consume the same readablestream twice: once in the
timeouter's catch block, and a second time in the body of the try block
before postGithubIssue() could run.

Closes #1658
2025-06-04 09:38:31 +03:00
Jussi Saurio
8bc1e06964 add simulator.log to gitignore 2025-06-04 09:29:25 +03:00
Jussi Saurio
e53ac59532 Fix assertion failing on 100 != 99.99999999999999999999 2025-06-04 09:28:26 +03:00
Jussi Saurio
5680326881 sim: add CREATE INDEX to interactions 2025-06-04 09:21:59 +03:00
Jussi Saurio
ad8c9a4c15 Merge 'Fix WAL frame checksum mismatch' from Diego Reis
Closes #1622
I did an A/B test between SQLite and Limbo and they can restart the db
from each other, indicating that there isn't something very wrong with
our file format. Turns out it was with our reset logic without
truncating the file. I assumed it's safe to don't reset if we're in
`PASSIVE` mode, given the
[docs](https://www.sqlite.org/c3ref/wal_checkpoint_v2.html) and [source
code](2bd9f69d40
f447e7b5c098/src/wal.c#L2193).
It also does some small clean ups and fixes.

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #1647
2025-06-04 09:14:45 +03:00
Jussi Saurio
e29b17ccbb Merge 'simulator: options to disable certain query types' from Pedro Muniz
Some additional cli options and some adjustments to determine if a query
statement is disabled for this run. This enables us to at least run the
simulator to completion when other parts of the system cause infinite
loops almost every time. I'm looking at you UPDATE

Closes #1657
2025-06-04 09:14:17 +03:00
Jussi Saurio
4fc04c8db3 sim/aws: fix sim timeout handling
code was trying to consume the same readablestream twice: once in
the timeouter's catch block, and a second time in the body of the
try block before postGithubIssue() could run.
2025-06-04 09:08:59 +03:00
Pekka Enberg
c6ef19396d Merge 'Add support for pragma table-valued functions' from Piotr Rżysko
This PR adds support for table-valued functions for PRAGMAs (see the
[PRAGMA functions section](https://www.sqlite.org/pragma.html)).
Additionally, it introduces built-in table-valued functions. I
considered using extensions for this, but there are several reasons in
favor of a dedicated mechanism:
* It simplifies the use of internal functions, structs, etc. For
example, when implementing `json_each` and `json_tree`, direct access to
internals was necessary:
https://github.com/tursodatabase/limbo/pull/1088
* It avoids FFI overhead. [Benchmarks](https://github.com/piotrrzysko/li
mbo/blob/pragma_vtabs_bench/core/benches/pragma_benchmarks.rs) on my
hardware show that `pragma_table_info()` implemented as an extension is
2.5× slower than the built-in version.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1642
2025-06-04 09:08:10 +03:00
Pekka Enberg
d2c5c4647f Limbo supports exact vector search
Make the roadmap item more explicit that it's about indexing.
2025-06-04 09:04:13 +03:00
Pekka Enberg
025714ce94 Merge 'bindings/javascript: Add proper exec() method and raw() mode' from Diego Reis
Now we can execute a sequence of statements from exec, it's similar to
`execute_batch()` from rusqlite
EDIT: It also add support for raw mode and do a small clean up to push
if statements outside loops

Closes #1620
2025-06-04 08:13:36 +03:00
pedrocarlo
1c424818aa simulator options to disable certain query types 2025-06-03 22:21:40 -03:00
Diego Reis
09f978b239 core: Tagging some comments as TODO 2025-06-03 15:09:16 -03:00
meteorgan
1554c54f2b restore comments 2025-06-03 22:06:08 +08:00
meteorgan
ab4eda59be close conn after executing limbo in cli 2025-06-03 22:06:08 +08:00
meteorgan
f2bf6251cd write database header via normal pager route 2025-06-03 22:06:08 +08:00
Pekka Enberg
e16b12b2f0 s/modern/next/g 2025-06-03 16:36:41 +03:00
Jussi Saurio
e260a72447 sim: log the interaction about to be executed with INFO
I think this is useful to see, without having to drop down to DEBUG
and seeing all the other stuff.
2025-06-03 15:50:34 +03:00
Pekka Enberg
e3e0208360 Merge 'Set maximum open simulator-created issues' from Jussi Saurio
Closes #1653
2025-06-03 15:40:52 +03:00
Jussi Saurio
3caf10d42f Set maximum open simulator-created issues 2025-06-03 15:39:00 +03:00
Jussi Saurio
a8106a0bbd Merge 'Fix cursors not being opened for indexes in DELETE' from Jussi Saurio
Closes #1648
Closes #1628

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #1652
2025-06-03 15:37:44 +03:00
Jussi Saurio
31b37332d5 all index cursors must be opened when DELETE does an index seek too 2025-06-03 15:18:45 +03:00
Jussi Saurio
06626f72eb Fix cursors not being opened for indexes in DELETE 2025-06-03 14:45:01 +03:00
Jussi Saurio
c488c32d43 Merge 'Make cursor seek reentrant' from Pedro Muniz
Closes #1628.  Every function that calls `process_overflow_read` needs
to be reentrant. I did not change it here, but it would include
`get_prev_record` and `get_next_record`. Maybe `tablebtree_move_to` did
not need to use the state machine, but I included it as a safeguard.
Edit: Closes #1625 . When I implemented `restore_context`, I forgot to
add a `return_if_io` after calling it in `next` 🤦‍♂️
Edit: Closes #1617 . Just tested it and it also solves this bug.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1636
2025-06-03 14:24:40 +03:00
Pekka Enberg
6ffbc63ab4 Merge 'Add simulator-docker-runner for running limbo-sim in a loop on AWS' from Jussi Saurio
Closes #1645
2025-06-03 14:20:46 +03:00
Diego Reis
cf038b045d core/wal: Only reset the WAL if the file is truncated 2025-06-02 23:16:30 -03:00
Diego Reis
16c81f471b core/ondisk: Stop reading WAL file if a frame mismatch salt value with header
The salts values in the WAL header are (re)generated in every checkpoint (but in PASSIVE mode), so if we find a frame with mismatch it means it's a leftover from a previous checkpoint.
2025-06-02 23:14:51 -03:00
Diego Reis
ec4eb52734 core/wal: Refactor open_shared for readability 2025-06-02 19:21:13 -03:00
Jussi Saurio
ea301de726 Merge 'Pass input string to translate function' from Pedro Muniz
In preparation for `CREATE VIEW`, we need to have the original sql query
that was used to create the view. I'm using the scanner's offset to
slice into the original input, trimming the newlines, and passing it to
the translate function.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1621
2025-06-02 17:43:11 +03:00
Jussi Saurio
5f586b7b24 Merge 'Small tracing enhancement' from Pedro Muniz
Instrument trace_insn to debug print its the stack pc and instruction.
Also, disable rustyline logs for the CLI as it is too noisy to work
with.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1635
2025-06-02 17:41:52 +03:00
Jussi Saurio
20555bce0a Merge 'Adjust write cursors for delete to avoid opening more than once. ' from Pedro Muniz
We were opening Write cursors inside of the rewind loop, which meant
that we were opening the same cursor multiple times per iteration. Also,
in this case, we need to open all indexes before the loop starts, and
avoid opening the same cursor again. While I was trying to debug this, I
also added instrumentation to some functions, so that I could track
where the instructions were being emitted. This helped me a lot. The
`#[instrument]` macro creates a span for the particular function, which
enables us to examine the path our functions are taking when any log is
emitted. Lastly, I fixed an explain bug, where it was unwrapping a None
`CursorKey` when opening the `sqlite_schema` table.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1640
2025-06-02 17:41:19 +03:00
Jussi Saurio
d06bb70514 Add simulator-docker-runner for running limbo-sim in a loop on AWS 2025-06-02 15:51:29 +03:00
Pekka Enberg
e6cfeb9552 Merge 'simulator: add option to disable BugBase' from Jussi Saurio
we will soon run the sim on AWS where we don't need bugbase

Closes #1644
2025-06-02 12:45:15 +03:00
Jussi Saurio
51e3a04668 simulator: add option to disable BugBase 2025-06-02 12:14:35 +03:00
pedrocarlo
9b5f5f6053 do not move_to if we are already inserting in correct place 2025-06-02 02:55:32 -03:00
pedrocarlo
9dc6638313 cleaner approach for opening indexes 2025-06-02 01:13:14 -03:00
pedrocarlo
c2942a5819 small fixes 2025-06-01 12:11:03 -03:00
Piotr Rzysko
d1d8ead475 Add support for pragma table-valued functions 2025-06-01 10:25:42 +02:00
pedrocarlo
39434fd20f return_io when restoring context 2025-06-01 03:07:16 -03:00
Piotr Rzysko
4d35e36b77 Introduce virtual table types 2025-06-01 07:45:57 +02:00
Piotr Rzysko
b291179554 Extract cursor logic from VirtualTable into VirtualTableCursor 2025-06-01 07:45:57 +02:00
Piotr Rzysko
6300deb77f Move VTabOpaqueCursor to vtab module 2025-06-01 07:45:57 +02:00
Piotr Rzysko
149375b2b4 Extract VirtualTable to a separate module 2025-06-01 07:45:57 +02:00
pedrocarlo
2ddbb7eeed adjust move_to and seek functions to make them truly reentrant + adding return_if_locked_maybe_load in some places so that we read loaded pages 2025-06-01 01:01:35 -03:00