Closes#3536
# Description
This PR implements **dynamic journal mode switching** via `PRAGMA
journal_mode`, allowing users to switch between WAL and MVCC modes at
runtime.
### Key Changes
**Core Feature: Journal Mode Switching**
- Added new `JournalMode` module (`core/storage/journal_mode.rs`) to
parse and handle journal mode transitions
- Modified `op_journal_mode` to correctly parse journal modes and update
the database header
- Emit checkpoint when setting a new journal mode to ensure data
consistency
- Added MVCC checkpoint support to `Connection::checkpoint`
**Database Initialization Improvements**
- Read DB header on `Database::open` and simplified `init_pager`
- Made `Version` an enum for better comparison semantics
- Automatically convert legacy SQLite databases to WAL mode
- Ensure DB header is flushed to disk when header changes during open
- Clear page cache after header validation
**Bug Fixes**
- Fixed dirty page invalidation in pager when clearing dirty pages in
page cache
- Fixed `is_none_or` check for row version existence in DB file (handles
MvStore initialization and empty database cases)
- Added `btree_resident` field in `RowVersion` to track if
insert/deletion originated from a btree
**Testing**
- Added fuzz tests for `journal_mode` transitions with Database
operations in between
- Added integration tests for testing switching from the different modes
while checking the header version is correct
- Added some specific regression tests for delete operations lost on
mode switch
- Fixed `index_scan_compound_key_fuzz` to use separate databases for
Turso and SQLite in MVCC mode. Also had to decrease number of rows for
MVCC test, as insert was very slow.
# TODO's
- Remove sync hacks from `op_journal_mode`
- Expand fuzzer with different queries
- Add to Simulator
- Special handling for read only databases and not allow any header
changes
# Motivation and context
Facilitate our users to test MVCC and transition back and forth from it.
# AI Disclosure
Used AI to catch and fix bugs in MVCC, further my understanding with
MVCC, write tests in `tests` folder, most of the PR summary, and the
docs in the `docs/manual.md` file
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4074
- Remove wal_checkpoint_start() function
- Remove wal_checkpoint_finish() function
- Remove wal_checkpoint() function
- Add blocking_checkpoint() as a convenience wrapper around checkpoint()
- Update checkpoint_shutdown() to take sync_mode parameter and use blocking_checkpoint()
The checkpoint() function now handles all the checkpoint state machine logic
internally, so the two-phase API is no longer needed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
After flushing the WAL, the db may attempt an automatic checkpoint. Previously,
a checkpoint failure for any other reason than `Busy` rolled back the transaction;
this is incorrect because the WAL has already been written and synced, so the transaction
is actually committed and durable.
For this reason, do the following decoupling:
- Mark the transaction as committed after `SyncWal`
- Any errors beyond that point are wrapped as `LimboError::CheckpointFailed` which is
handled specially:
* Tx rollback is not attempted in `abort()` - only the WAL locks are cleaned up
and checkpoint state machines are reset
* In the simulator, the results of the query are shadowed into the sim environment
so that the sim correctly assumes that the transaction's effects were in fact
committed.
When the SchemaUpdated error occurs during statement execution, don't
roll back the transaction, but instead re-prepare the statement.
Spotted by Whopper.
`ExecRows` trait should allow us to do something like this when testing:
```rust
let rows: Vec<(String,)> = conn.exec_rows("SELECT val FROM t ORDER BY val");
```
Which just makes your life easier overall so we don't have to constantly
repeat the `.step` loop
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#4164
In my adventure of doing #3536, I had a bug where an opcode was using a
stale MvStore reference when transitioning from MvStore to Wal mode. I
did not want to special case just 1 opcode to ignore the `mv_store`
argument (in my particular case it was `Insn::Checkpoint`), so I just
edited it and forced the opcodes to get `mv_store` from the
`program.connection.mv_store()` which is always up to date.
But, if we don't want to make this change I can always just do the
special case and move on.
**AI Disclosure:**
Asked Claude to do most of refactoring, but all the changes were
manually approved by me.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4126
On database open, we store the Encryption Options and pass them onwards
to the Connection, Pager and Wal. We also have slight gain in
ergonomics, as we don't have set the Pragma's for the `cipher` and
`hexkey` on each new `Connection`.
I needed this logic, because I will need to initialize a Default Header
for empty DBs and encryption opts not being automatically propagated was
hindering me for this.
**Ai Disclosure**
Claude helped me debug and find out issues in my implementation
cc @avinassh
Reviewed-by: Avinash Sajjanshetty (@avinassh)
Closes#4100
## Beef
- Change logical log format to also allow index row records - this is
for simplicity during recovery and may change later.
- Checkpoint indexes and index writes to the DB file
- Fix issues related to deletes - it's possible MV store has no row
versions for a row that exists in the DB file, so we need to add a
tombstone row version in that case, and we must fetch that row's data
from the btree to be able to include the data in the row version
- fix some miscellaneous logic bugs
Closes#4067
This PR introduces program execution state in order for statement to be
aware of its state - is it terminal (Done, Failed, Interrupted) or not.
The particular problem right now is that statements like `INSERT INTO t
VALUES (1), (2), (3) RETURNING x` will execute inserts one by one and
interleave them with rows generation. This means that if statement
consumer will just read one row and then finalize the statement -
nothing will be actually committed (because transaction will be
aborted).
In order to quickly mitigate this issue - program state is introduced
which can help to decide what to do in the finalize.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4038
