Commit graph

477 commits

Author SHA1 Message Date
Jussi Saurio
18f6e1a350 Update tests for BusySnapshot error
- Handle "snapshot is stale" in fuzz tests as retriable error
- Test that stale snapshot returns BusySnapshot, not Busy
2025-12-21 18:37:47 +02:00
pedrocarlo
32401f52a1 clippy 2025-12-19 17:20:46 -03:00
pedrocarlo
eaadaee5c3 refactor more of the core tester to use helpers 2025-12-19 17:20:46 -03:00
pedrocarlo
e986247890 convert sqlite.rs + core tester to use helpers 2025-12-19 17:20:46 -03:00
Pekka Enberg
edd45ff7b8 Improve MVCC DX by dropping --experimental-mvcc flag
The DX is right now pretty terrible:

```
penberg@vonneumann turso % cargo run -- hello.db
    Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.15s
     Running `target/debug/tursodb hello.db`
Turso v0.4.0-pre.18
Enter ".help" for usage hints.
Did you know that Turso supports live materialized views? Type .manual materialized-views to learn more.
This software is in BETA, use caution with production data and ensure you have backups.
turso> PRAGMA journal_mode = 'experimental_mvcc';
  × Invalid argument supplied: MVCC is not enabled. Enable it with `--experimental-mvcc` flag in the CLI or by setting the MVCC option in `DatabaseOpts`

turso>
```

To add insult to the injury, many SDKs don't even have a way to enable
MVCC via database options. Therefore, let's remove the flag altogether.
2025-12-19 12:59:42 +02:00
pedrocarlo
da33f421b3 add readonly checks to ensure we do not change the header 2025-12-18 10:54:58 -03:00
Jussi Saurio
c282c24d94
Merge 'clean up core tester to use conn.execute and conn.exec_rows for parsing correctly the expected values from select queries' from Pedro Muniz
## Description
The PR title. `exec_rows` also does validation of outputs automatically
which is good practice for testing
<!--
Please include a summary of the changes and the related issue.
-->
## Motivation and context
Better typing and don't have to constantly match on `turso_core::Value`
<!--
Please include relevant motivation and context.
Link relevant issues here.
-->
## AI Disclosure
Ai did most of the migration
<!--
Please disclose if any LLM's were used in the creation of this PR and to
what extent,
to help maintainers properly review.
-->

Closes #4192
2025-12-18 09:22:45 +02:00
Jussi Saurio
00d266665b
Merge 'fix coroutine panic: replace ended_coroutine Bitfield with vec' from Jussi Saurio
## Description
Closes #4146
## Motivation and context
panics are bad
## AI Disclosure
none used

Reviewed-by: Pedro Muniz (@pedrocarlo)
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #4177
2025-12-18 09:20:05 +02:00
pedrocarlo
046e6a884d use exec rows for header version test 2025-12-17 10:55:25 -03:00
pedrocarlo
33afc3015c adjust test + remove mv store after transitioning to wal mode 2025-12-17 10:55:25 -03:00
pedrocarlo
e54d3328c0 after checkpoint get header from pager to properly persist change in header 2025-12-17 10:55:25 -03:00
pedrocarlo
257dc5ad09 do not initiate a write transaction for journal mode + checkpoint before changing mode 2025-12-17 10:55:24 -03:00
pedrocarlo
277f9928b7 test changing from WAL to MVCC 2025-12-17 10:55:24 -03:00
pedrocarlo
0cbe904cef ensure DB header is flushed to DB file if header changes during DB open 2025-12-17 10:55:24 -03:00
Pekka Enberg
98308415b4 core: Don't rollback transaction when schema updated
When the SchemaUpdated error occurs during statement execution, don't
roll back the transaction, but instead re-prepare the statement.

Spotted by Whopper.
2025-12-15 13:49:21 +02:00
PThorpe92
12601af1e1
increase lantency check for flaky test in test_read_path.rs 2025-12-12 13:49:56 -05:00
pedrocarlo
60ab032e3a clean up core tester to use conn.execute instead of limbo_exec_rows and use conn.exec_rows for parsing correctly the expected values from select queries 2025-12-12 12:36:48 -03:00
Jussi Saurio
216f4d71ee cargo fmt 2025-12-11 23:37:19 +02:00
Jussi Saurio
6280ab4a7b Regression test for 4146 2025-12-11 23:37:19 +02:00
pedrocarlo
bc16588273 index and rowid fuzz should open a separate sqlite database for comparison 2025-12-10 15:38:02 -03:00
pedrocarlo
ffbbd4c270 add exec rows trait for more ergonomic testing in core_tester 2025-12-10 15:21:03 -03:00
pedrocarlo
c207eddd3f remove unused TempDatabase argument requirement for limbo_exec_rows 2025-12-10 15:21:03 -03:00
Jussi Saurio
64dba96c60
Merge 'initialize global header on bootstrap' from Pedro Muniz
On bootstrap just store the header but not flush it to disk. Only try to
flush it when we start an MVCC transaction. Also applied fix in
`OpenDup` where we should not wrap an ephemeral table with an MvCursor

Reviewed-by: Mikaël Francoeur (@LeMikaelF)
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #4151
2025-12-10 19:04:23 +02:00
Jussi Saurio
0d35366f5d
Merge 'Fix CTE scope propagation for compound SELECTs' from Martin Mauch
CTEs now work correctly when combined with UNION, UNION ALL, INTERSECT,
and EXCEPT.
**Before:**
```sql
WITH t AS (SELECT 1 as x) SELECT * FROM t UNION ALL SELECT 2 as x
-- Error: Parse error: no such table: t
```
**After:**
```sql
WITH t AS (SELECT 1 as x) SELECT * FROM t UNION ALL SELECT 2 as x
-- Works correctly, returns rows (1) and (2)
```

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #4123
2025-12-10 19:04:03 +02:00
pedrocarlo
d09b48c0e6 test page1 init 2025-12-10 12:53:25 -03:00
Nikita Sivukhin
9acf541e28 add compatibility test for multiple-columns unique constraint 2025-12-10 01:46:25 +04:00
Martin Mauch
3dc8fed204 Fix CTE scope propagation for compound SELECTs 2025-12-09 13:09:55 +01:00
Jussi Saurio
2aefb4ee8c
Merge 'fix/btree: disable move_to_rightmost optimization with triggers' from Jussi Saurio
Some checks are pending
Build & publish @tursodatabase/database / db-bindings-x86_64-pc-windows-msvc - node@20 (push) Waiting to run
Build & publish @tursodatabase/database / db-bindings-x86_64-unknown-linux-gnu - node@20 (push) Waiting to run
Build & publish @tursodatabase/database / sync-bindings-aarch64-apple-darwin - node@20 (push) Waiting to run
Build & publish @tursodatabase/database / sync-bindings-aarch64-unknown-linux-gnu - node@20 (push) Waiting to run
Build & publish @tursodatabase/database / sync-bindings-wasm32-wasip1-threads - node@20 (push) Waiting to run
Build & publish @tursodatabase/database / sync-bindings-x86_64-pc-windows-msvc - node@20 (push) Waiting to run
Build & publish @tursodatabase/database / sync-bindings-x86_64-unknown-linux-gnu - node@20 (push) Waiting to run
Build & publish @tursodatabase/database / Test DB bindings on Linux-x64-gnu - node@20 (push) Blocked by required conditions
Build & publish @tursodatabase/database / Test DB bindings on browser@20 (push) Blocked by required conditions
Build & publish @tursodatabase/database / Publish (push) Blocked by required conditions
Python / configure-strategy (push) Waiting to run
Python / test (push) Blocked by required conditions
Python / lint (push) Waiting to run
Python / linux (x86_64) (push) Waiting to run
Python / macos-arm64 (aarch64) (push) Waiting to run
Python / sdist (push) Waiting to run
Python / Release (push) Blocked by required conditions
Rust / cargo-fmt-check (push) Waiting to run
Rust / build-native (blacksmith-4vcpu-ubuntu-2404) (push) Waiting to run
Rust / build-native (macos-latest) (push) Waiting to run
Rust / build-native (windows-latest) (push) Waiting to run
Rust / clippy (push) Waiting to run
Rust / simulator (push) Waiting to run
Rust / test-limbo (push) Waiting to run
Rust / test-sqlite (push) Waiting to run
Rust Benchmarks+Nyrkiö / bench (push) Waiting to run
Rust Benchmarks+Nyrkiö / clickbench (push) Waiting to run
Rust Benchmarks+Nyrkiö / tpc-h-criterion (push) Waiting to run
Rust Benchmarks+Nyrkiö / tpc-h (push) Waiting to run
Rust Benchmarks+Nyrkiö / vfs-bench-compile (push) Waiting to run
## Closes
- Closes #4017
- Addresses #4043; this now fails with `Page cache is full` with 100k
pages, which is a separate non-corruption issue. Modifying max page
cache size to be 10 million pages makes it not finish at all. We should
modify the issue after this is merged to reflect what the new problem
is. The queries in the issue (#4043) create a WAL that is at least 1.7
GB in size
## Background
We have an optimization in the btree where if:
- We want to reach the rightmost leaf page, and
- We know the rightmost page and are already on it
Then we can skip a seek.
## Problem
The problem is this optimization should NEVER be used in cases where we
cannot be sure that the btree wasn't modified from under us e.g. by a
trigger subprogram.
## Fix
Hence, disable it when we are executing a parent program that has
triggers which will fire.
## AI Disclosure
No AI was used for this PR.

Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #4135
2025-12-09 10:02:11 +02:00
Jussi Saurio
201a7e6387 Regression test for 4017 2025-12-09 09:19:37 +02:00
Nikita Sivukhin
997a07cac9 add test with concurrent commit/rollback and insert stmt 2025-12-08 16:34:07 +04:00
Jussi Saurio
826ca4d44d chore: remove experimental_indexes feature flags 2025-12-08 13:00:37 +02:00
Nikita Sivukhin
659ef7c079 fix clippy 2025-12-05 21:39:35 +04:00
Nikita Sivukhin
487854e6d6 guard subjournal access in order to prevent concurrent operations over it within same connection 2025-12-05 21:25:13 +04:00
Jussi Saurio
eb782ce2d4 fix/mvcc: seek() must seek from both mv store and btree
for example, upon opening an existing database, all the rows are in
the btree, so if we seek only from MV store, we won't find anything.
ergo: we must look from both the mv store and the btree. if we are
iterating forwards, the smallest of the two results is where we land,
and vice versa for backwards iteration.

initially this implementation used blocking IO but was refactored to
use state machines after the rest of the Cursor methods in the MVCC cursor
module were refactored to do that too.

---

this PR was initially almost entirely written using Claude Code + Opus 4.5,
but heavily manually cleaned up as the AI made the state machine refactor
far too complicated.
2025-12-05 11:53:16 +02:00
Jussi Saurio
3b412f9b02 mvcc: commit index rows and recover them from logical log
also adds tests
2025-12-02 11:38:05 +02:00
Jussi Saurio
ea905d276c
Merge 'Add #[turso_macros::test] to automatically create tests that can run MVCC with minimal code changes' from Pedro Muniz
- added procedural macro that creates Rust tests and with just a flag,
creates a new test that runs the same with MVCC enabled
- migrated almost all tests to use this new macro and added the mvcc
flag to the tests that were not failing
- added a `TempDatabase` builder to facilitate the proc_macro to
generate the correct database options

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #3991
2025-11-27 21:53:09 +02:00
Jussi Saurio
e5937de219 mvcc: reconstruct index rows on logical log recovery
index rows are not committed to the logical log, so we derive them
after recovery and insert them to the mv store.
2025-11-27 20:35:49 +02:00
PThorpe92
9223dfdbff
Add integration test to ensure covering expr index is used 2025-11-20 12:47:50 -05:00
PThorpe92
2688f8b9c3
Add tcl tests for expr indexes and collation 2025-11-20 12:47:49 -05:00
PThorpe92
b0ac042092
Add tests for using expr indexes in query plans for ORDER BY and WHERE predicates 2025-11-20 12:47:47 -05:00
pedrocarlo
a3acec58df clippy 2025-11-19 23:42:46 -03:00
pedrocarlo
8b2caab3cc change pragma test to use new test macros 2025-11-19 23:28:23 -03:00
pedrocarlo
5aed5147c9 change trigger test to use new test macros 2025-11-19 23:28:23 -03:00
pedrocarlo
c4653ae34c change functions to use new test macros 2025-11-19 23:28:23 -03:00
pedrocarlo
479048ee99 change index_methods to use new test macros 2025-11-19 23:28:23 -03:00
pedrocarlo
4a79d4f5d4 change encryption to use new test macros 2025-11-19 14:47:13 -03:00
pedrocarlo
3bc0499f97 change test_btree to use new test macros 2025-11-19 13:46:48 -03:00
pedrocarlo
789532688e change test_ddl to use new test macros 2025-11-19 13:30:35 -03:00
pedrocarlo
c58cd32eee change test_multi_thread to use new test macros 2025-11-19 13:30:05 -03:00
pedrocarlo
5301a042d9 change test_read_path to use new test macros 2025-11-19 13:30:05 -03:00