This PR introduces few sync fixes relevant to the partial sync feature:
1. Make `try_wal_watermark_read_page` async - this is important as now
db file can be partial and require extra network IO for fetching missing
pages
2. Do not panic if requested page doesn't exist on server - this can be
valid case if db on the server has smaller size
3. Maintain clean db file size in order to avoid reads outside of the
file (this can happen, when we revert new pages allocated only in the
WAL and need to fetch their previous state if there is one)
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#4297
The DX is right now pretty terrible:
```
penberg@vonneumann turso % cargo run -- hello.db
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.15s
Running `target/debug/tursodb hello.db`
Turso v0.4.0-pre.18
Enter ".help" for usage hints.
Did you know that Turso supports live materialized views? Type .manual materialized-views to learn more.
This software is in BETA, use caution with production data and ensure you have backups.
turso> PRAGMA journal_mode = 'experimental_mvcc';
× Invalid argument supplied: MVCC is not enabled. Enable it with `--experimental-mvcc` flag in the CLI or by setting the MVCC option in `DatabaseOpts`
turso>
```
To add insult to the injury, many SDKs don't even have a way to enable
MVCC via database options. Therefore, let's remove the flag altogether.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4294
The DX is right now pretty terrible:
```
penberg@vonneumann turso % cargo run -- hello.db
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.15s
Running `target/debug/tursodb hello.db`
Turso v0.4.0-pre.18
Enter ".help" for usage hints.
Did you know that Turso supports live materialized views? Type .manual materialized-views to learn more.
This software is in BETA, use caution with production data and ensure you have backups.
turso> PRAGMA journal_mode = 'experimental_mvcc';
× Invalid argument supplied: MVCC is not enabled. Enable it with `--experimental-mvcc` flag in the CLI or by setting the MVCC option in `DatabaseOpts`
turso>
```
To add insult to the injury, many SDKs don't even have a way to enable
MVCC via database options. Therefore, let's remove the flag altogether.
This PR introduces local sync server and run tests for Go and Python
against it in CI
The local sync server implements 2 endpoints:
1. `/v2/pipeline` - subset of SQL over HTTP protocol (Hrana) to execute
logical push operations from the client
2. `/pull-updates` - endpoint which returns page updates for client to
apply locally
The implementation is based on the local database file with **disabled
checkpoint** in order to preserve whole DB history and allow server to
respond to client which can have arbitrary stale DB.
For implementation, sync server uses extra API exposed by the turso-core
under `conn-raw-api` feature which includes `wal_state` /
`wal_get_frame` methods.
Usage:
- `tursodb --sync-server 0.0.0.0:8080` - in-memory database
- `tursodb local.db --sync-server 0.0.0.0:8080` - local db file
Closes#4191
Closes#3536
# Description
This PR implements **dynamic journal mode switching** via `PRAGMA
journal_mode`, allowing users to switch between WAL and MVCC modes at
runtime.
### Key Changes
**Core Feature: Journal Mode Switching**
- Added new `JournalMode` module (`core/storage/journal_mode.rs`) to
parse and handle journal mode transitions
- Modified `op_journal_mode` to correctly parse journal modes and update
the database header
- Emit checkpoint when setting a new journal mode to ensure data
consistency
- Added MVCC checkpoint support to `Connection::checkpoint`
**Database Initialization Improvements**
- Read DB header on `Database::open` and simplified `init_pager`
- Made `Version` an enum for better comparison semantics
- Automatically convert legacy SQLite databases to WAL mode
- Ensure DB header is flushed to disk when header changes during open
- Clear page cache after header validation
**Bug Fixes**
- Fixed dirty page invalidation in pager when clearing dirty pages in
page cache
- Fixed `is_none_or` check for row version existence in DB file (handles
MvStore initialization and empty database cases)
- Added `btree_resident` field in `RowVersion` to track if
insert/deletion originated from a btree
**Testing**
- Added fuzz tests for `journal_mode` transitions with Database
operations in between
- Added integration tests for testing switching from the different modes
while checking the header version is correct
- Added some specific regression tests for delete operations lost on
mode switch
- Fixed `index_scan_compound_key_fuzz` to use separate databases for
Turso and SQLite in MVCC mode. Also had to decrease number of rows for
MVCC test, as insert was very slow.
# TODO's
- Remove sync hacks from `op_journal_mode`
- Expand fuzzer with different queries
- Add to Simulator
- Special handling for read only databases and not allow any header
changes
# Motivation and context
Facilitate our users to test MVCC and transition back and forth from it.
# AI Disclosure
Used AI to catch and fix bugs in MVCC, further my understanding with
MVCC, write tests in `tests` folder, most of the PR summary, and the
docs in the `docs/manual.md` file
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4074
- Remove wal_checkpoint_start() function
- Remove wal_checkpoint_finish() function
- Remove wal_checkpoint() function
- Add blocking_checkpoint() as a convenience wrapper around checkpoint()
- Update checkpoint_shutdown() to take sync_mode parameter and use blocking_checkpoint()
The checkpoint() function now handles all the checkpoint state machine logic
internally, so the two-phase API is no longer needed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
After flushing the WAL, the db may attempt an automatic checkpoint. Previously,
a checkpoint failure for any other reason than `Busy` rolled back the transaction;
this is incorrect because the WAL has already been written and synced, so the transaction
is actually committed and durable.
For this reason, do the following decoupling:
- Mark the transaction as committed after `SyncWal`
- Any errors beyond that point are wrapped as `LimboError::CheckpointFailed` which is
handled specially:
* Tx rollback is not attempted in `abort()` - only the WAL locks are cleaned up
and checkpoint state machines are reset
* In the simulator, the results of the query are shadowed into the sim environment
so that the sim correctly assumes that the transaction's effects were in fact
committed.
When the SchemaUpdated error occurs during statement execution, don't
roll back the transaction, but instead re-prepare the statement.
Spotted by Whopper.
`ExecRows` trait should allow us to do something like this when testing:
```rust
let rows: Vec<(String,)> = conn.exec_rows("SELECT val FROM t ORDER BY val");
```
Which just makes your life easier overall so we don't have to constantly
repeat the `.step` loop
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#4164
In my adventure of doing #3536, I had a bug where an opcode was using a
stale MvStore reference when transitioning from MvStore to Wal mode. I
did not want to special case just 1 opcode to ignore the `mv_store`
argument (in my particular case it was `Insn::Checkpoint`), so I just
edited it and forced the opcodes to get `mv_store` from the
`program.connection.mv_store()` which is always up to date.
But, if we don't want to make this change I can always just do the
special case and move on.
**AI Disclosure:**
Asked Claude to do most of refactoring, but all the changes were
manually approved by me.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4126
On database open, we store the Encryption Options and pass them onwards
to the Connection, Pager and Wal. We also have slight gain in
ergonomics, as we don't have set the Pragma's for the `cipher` and
`hexkey` on each new `Connection`.
I needed this logic, because I will need to initialize a Default Header
for empty DBs and encryption opts not being automatically propagated was
hindering me for this.
**Ai Disclosure**
Claude helped me debug and find out issues in my implementation
cc @avinassh
Reviewed-by: Avinash Sajjanshetty (@avinassh)
Closes#4100
