## Description
Add some readonly checks in header validation and `pragma journal_mode`
. Depends on #4279 being merged first, to avoid conflicts here.
<!--
Please include a summary of the changes and the related issue.
-->
## Motivation and context
Close#4270
<!--
Please include relevant motivation and context.
Link relevant issues here.
-->
## Description of AI Usage
AI again did most of the work here, as it is pretty basic stuff and
mostly boilerplate. The main usefullness for AI here was to write the
tests to check for these edge cases.
**Prompt:**
```
I want to Make sure readonly databases cannot modify header page on Database open nor call `pragma journal mode` to update the journal mode. I need you
to implement the necessary checks to ensure we can still continue working normally and emit warnings to show that we cannot change to mvcc. Lastly add
tests in `header_version.rs` to prove your modifications works.
```
<!--
Please disclose how AI was used to help create this PR. For example, you
can share prompts,
specific tools, or ways of working that you took advantage of. You can
also share whether the
creation of the PR was mainly driven by AI, or whether it was used for
assistance.
This is a good way of sharing knowledge to other contributors about how
we can work more efficiently with
AI tools. Note that the use of AI is encouraged, but the committer is
still fully responsible for understanding
and reviewing the output.
-->
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#4280
## Description
This PR greatly simplifies the slot bitmap used to track free pages in
the arena buffer pool.
## Motivation and context
An optimization was included that would allow for allocating multiple
contiguous buffers, with the objective being that they would be
coalesced into single buffers when submitting `pwritev` calls, for
things like WAL appends. This optimization was never implemented and we
are left with a very complex bitmap with lots of unused/unnecessary
logic.
## Description of AI Usage
This was mostly codex 5.2 with the prompt:
```
This project is a SQLite rewrite in Rust, it uses a BufferPool that allocates large arenas
and tracks which slots are free using a bitmap. this bitmap is core/storage/slot_bitmap.rs..
it was originally designed in a way that would allow to request multiple buffers that
were contiguous in memory, so that they could be coalesced into a single `pwrite` operation later down the line.
However this optimization was never implemented and the bitmap has a complex 'two-pointer' algorithm that we no
longer need. please rewrite this slot_bitmap.rs to simplify and only allocate single buffers at a time, removing the need
for the two pointer hint system.
```
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#4277
## Description
Remove sync IO hacks for `op_journal_mode`
Close#4268
<!--
Please include a summary of the changes and the related issue.
-->
## Motivation and context
Remove sync io hacks so it is friendlier for WASM
<!--
Please include relevant motivation and context.
Link relevant issues here.
-->
## Description of AI Usage
Ai basically made the bulk refactoring and I made some adjustments and
trimmed down the implementation
**Prompt**:
```
if look at @core/storage/journal_mode.rs and `op_journal_mode` in `execute.rs` you will see that we have some blocking io operations with
`pager.io.block` and `program.connection.checkpoint` that also blocks. I want you refactor the code to use state machines similar in nature to how we do it
in many functions in `execute.rs`
```
<!--
Please disclose how AI was used to help create this PR. For example, you
can share prompts,
specific tools, or ways of working that you took advantage of. You can
also share whether the
creation of the PR was mainly driven by AI, or whether it was used for
assistance.
This is a good way of sharing knowledge to other contributors about how
we can work more efficiently with
AI tools. Note that the use of AI is encouraged, but the committer is
still fully responsible for understanding
and reviewing the output.
-->
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4279
## Description
We should correctly check for completion success
<!--
Please include a summary of the changes and the related issue.
-->
## Motivation and context
<!--
Please include relevant motivation and context.
Link relevant issues here.
-->
## Description of AI Usage
Ai made the tests for me
<!--
Please disclose how AI was used to help create this PR. For example, you
can share prompts,
specific tools, or ways of working that you took advantage of. You can
also share whether the
creation of the PR was mainly driven by AI, or whether it was used for
assistance.
This is a good way of sharing knowledge to other contributors about how
we can work more efficiently with
AI tools. Note that the use of AI is encouraged, but the committer is
still fully responsible for understanding
and reviewing the output.
-->
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4263
## Description
closes https://github.com/tursodatabase/turso/issues/4142
<!--
Please include a summary of the changes and the related issue.
-->
## Motivation and context
compatibility, we were wrongly rewriting table qualified cols, also
added trigger.test to all.test and expect correct values in a test
<!--
Please include relevant motivation and context.
Link relevant issues here.
-->
## AI Disclosure
None
<!--
Please disclose if any LLM's were used in the creation of this PR and to
what extent,
to help maintainers properly review.
-->
Closes#4206
## Description
Closes#4146
## Motivation and context
panics are bad
## AI Disclosure
none used
Reviewed-by: Pedro Muniz (@pedrocarlo)
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#4177
Closes#3536
# Description
This PR implements **dynamic journal mode switching** via `PRAGMA
journal_mode`, allowing users to switch between WAL and MVCC modes at
runtime.
### Key Changes
**Core Feature: Journal Mode Switching**
- Added new `JournalMode` module (`core/storage/journal_mode.rs`) to
parse and handle journal mode transitions
- Modified `op_journal_mode` to correctly parse journal modes and update
the database header
- Emit checkpoint when setting a new journal mode to ensure data
consistency
- Added MVCC checkpoint support to `Connection::checkpoint`
**Database Initialization Improvements**
- Read DB header on `Database::open` and simplified `init_pager`
- Made `Version` an enum for better comparison semantics
- Automatically convert legacy SQLite databases to WAL mode
- Ensure DB header is flushed to disk when header changes during open
- Clear page cache after header validation
**Bug Fixes**
- Fixed dirty page invalidation in pager when clearing dirty pages in
page cache
- Fixed `is_none_or` check for row version existence in DB file (handles
MvStore initialization and empty database cases)
- Added `btree_resident` field in `RowVersion` to track if
insert/deletion originated from a btree
**Testing**
- Added fuzz tests for `journal_mode` transitions with Database
operations in between
- Added integration tests for testing switching from the different modes
while checking the header version is correct
- Added some specific regression tests for delete operations lost on
mode switch
- Fixed `index_scan_compound_key_fuzz` to use separate databases for
Turso and SQLite in MVCC mode. Also had to decrease number of rows for
MVCC test, as insert was very slow.
# TODO's
- Remove sync hacks from `op_journal_mode`
- Expand fuzzer with different queries
- Add to Simulator
- Special handling for read only databases and not allow any header
changes
# Motivation and context
Facilitate our users to test MVCC and transition back and forth from it.
# AI Disclosure
Used AI to catch and fix bugs in MVCC, further my understanding with
MVCC, write tests in `tests` folder, most of the PR summary, and the
docs in the `docs/manual.md` file
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4074
This is already done in abort() but if the checkpoint error is
returned directly from commit_tx() instead of in an IO completion,
then the Tx state will be cleared and the DDL change won't be applied.
So, do it explicitly in step_end_write_txn() too.
Fixes this failing seed:
cargo run -- --memory-io --seed 12902559987470385066 --disable-heuristic-shrinking
## Description
We had a turso_stress failure happening on Antithesis. See the
associated issue for how to reproduce it. It happens only on io_uring.
This PR fixes the failure by ensuring writes to the WAL are completed
before the frame cache is updated. Without this, other threads can
retrieve a frame from the cache before the frame has been persisted.
## Motivation and context
Closes https://github.com/tursodatabase/turso/issues/4249
## Description of AI Usage
I used the reproducer in
https://github.com/tursodatabase/turso/issues/4249, and told Claude to
fix it. I then reviewed its work to make sure I understood it.
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#4252
the DB file, as if self.checkpointed_txid_max_old == None it could mean
the MvStore recently initialized or we are dealing with an empty
database. In both cases, we cannot assert the row version exists in the
db file
## Description
Does solve #4154, but I don't want to close it with this PR, because it
does not solve the Affinity issue.
We can only use an index to iterate over if the column collation in the
order by clause matches the index collation
<!--
Please include a summary of the changes and the related issue.
-->
## Motivation and context
Fix a bug in the optimizer
<!--
Please include relevant motivation and context.
Link relevant issues here.
-->
## Description of AI Usage
Used AI to write tests, fuzzers, and help me understand the optimizer
code.
Test prompt:
<details>
can you write tests in tcl that test that the correct collation sequence
is properly maintained.
```
CREATE TABLE "t1" ("c1" TEXT COLLATE RTRIM);
INSERT INTO "t1" VALUES (' ');
CREATE INDEX "i1" ON "t1" ("c1" COLLATE RTRIM DESC);
INSERT INTO "t1" VALUES (1025.1655084065987);
SELECT "c1", typeof(c1) FROM "t1" ORDER BY "c1" COLLATE BINARY DESC, rowid ASC;
```
this is an example of a query that returned incorrect results because of
this
</details>
<!--
Please disclose how AI was used to help create this PR. For example, you
can share prompts,
specific tools, or ways of working that you took advantage of. You can
also share whether the
creation of the PR was mainly driven by AI, or whether it was used for
assistance.
This is a good way of sharing knowledge to other contributors about how
we can work more efficiently with
AI tools. Note that the use of AI is encouraged, but the committer is
still fully responsible for understanding
and reviewing the output.
-->
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4248
## Beef
Centralize all checkpointing into a single state machine.
- Get rid of `wal_checkpoint_start`, `wal_checkpoint_finish`,
`wal_checkpoint`
- Rename `wal_checkpoint` to `blocking_checkpoint`
- Handle truncation and fsyncing in the single checkpoint state machine
and remove separate fsyncing steps from `commit_dirty_pages_inner` -
essentially we were fsyncing the DB file twice
## Review guide
Commit by commit is useful. I generated and cleaned up a larger single
commit with Opus first and then asked it to restructure it as a series
of atomic commits that sum up to an identical diff. The last commit is a
manual fix.
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#4255
- not all systems has cmath functions which we import from system
libraries
- let's use external implementation only in tests in order to eliminate
precision errors in the differential tests
- https://discord.com/channels/933071162680958986/933071163184283651/145
0476358005293147
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#4246
## Description
This PR fixes the JSON serialization of infinite floating-point values,
as reported in #4196. Previously, converting `f64::INFINITY` or
`f64::NEG_INFINITY` to JSON produced a parse error because the `ryu`
crate formats infinity as `"inf"`, which is not valid JSON.
The implementation now correctly serializes infinite values to match
SQLite's behavior, which varies depending on the JSON function used:
- For `json_array()` and `json_object()` with float arguments:
```
turso> create table t(a);
turso> insert into t values (1e309);
turso> insert into t values (-1e309);
turso> select json_object('k', a) from t;
┌────────────────────────┐
│ json_object ('k', t.a) │
├────────────────────────┤
│ {"k":9.0e+999} │
├────────────────────────┤
│ {"k":-9.0e+999} │
└────────────────────────┘
turso> select json_array(a) from t;
┌──────────────────┐
│ json_array (t.a) │
├──────────────────┤
│ [9.0e+999] │
├──────────────────┤
│ [-9.0e+999] │
└──────────────────┘
```
- For `json()` with bare infinity or JSON5 text containing infinity:
```
turso> select json(a) from t;
┌────────────┐
│ json (t.a) │
├────────────┤
│ 9e999 │
├────────────┤
│ -9e999 │
└────────────┘
turso> SELECT json('{x: Infinity}');
┌────────────────────────┐
│ json ('{x: Infinity}') │
├────────────────────────┤
│ {"x":9e999} │
└────────────────────────┘
```
## Motivation and context
Fixes#4196.
Currently, Turso produces a parse error when serializing infinite
values:
```
turso> create table t(a);
turso> insert into t values (1e309);
turso> insert into t values (-1e309);
turso> select json_object('k', a) from t;
× Parse error: malformed JSON
turso> select json_array(a) from t;
× Parse error: malformed JSON
turso> select json(a) from t;
× Parse error: malformed JSON
turso> SELECT json('{x: Infinity}');
┌────────────────────────┐
│ json ('{x: Infinity}') │
├────────────────────────┤
│ {"x":9e999} │
└────────────────────────┘
```
SQLite handles this differently, infinite floating-point values are
serialized using the notations `9.0e+999` and `9e999`, depending on the
JSON function:
```
sqlite> create table t(a);
sqlite> insert into t values (1e309);
sqlite> insert into t values (-1e309);
sqlite> select json_object('k', a) from t;
{"k":9.0e+999}
{"k":-9.0e+999}
sqlite> select json_array(a) from t;
[9.0e+999]
[-9.0e+999]
sqlite> select json(a) from t;
9e999
-9e999
sqlite> SELECT json('{x: Infinity}');
{"x":9e999}
```
## AI Disclosure
This PR was developed with assistance from GitHub Copilot (Claude Sonnet
4.5). The AI helped identify the root cause and assisted in writing the
unit tests.
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#4209
## Description
This PR fixes underflow issue when updating integer primary keys,
aligning with SQLite's `datatype mismatch` behavior. Previously, updates
would silently reuse the same id in release mode and panic in debug
mode. This PR also changes the `datatype mismatch` errors from parse
errors to runtime errors to match SQLite behavior.
```
turso> create table t(a integer primary key);
turso> insert into t values (-9223372036854775808);
turso> update t set a = a - 1;
× Runtime error: datatype mismatch
```
Additionally, this PR introduces a second commit to fix integer/float
equivalence in numeric string comparisons. Although this change is not
related to the PR, it is necessary for the newly introduced unit tests
in `core/util` to pass, as existing tests are failing on `master`.
## Motivation and context
Fixes#3848.
Turso's current behavior:
```
turso> create table t(a integer primary key);
turso> insert into t values (-9223372036854775808);
turso> update t set a = a - 1; -- this can be repeated many times, it's ignored
turso> select * from t;
┌──────────────────────┐
│ a │
├──────────────────────┤
│ -9223372036854775808 │
└──────────────────────┘
```
Expected behavior:
```
sqlite> create table t(a integer primary key);
sqlite> insert into t values (-9223372036854775808);
sqlite> update t set a = a - 1;
Runtime error: datatype mismatch (20)
```
## Description of AI Usage
GPT-5.1 Codex Max helped identify the underflow path.
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#4251
The clone() approach didn't work because CheckpointResult contains
CheckpointGuard which holds locks. Cloning creates two guards that both
think they own the locks, and when one is dropped it releases them.
Instead, we restructure CheckpointState:
- CheckpointState is now a struct with phase and result fields
- CheckpointPhase is an enum that doesn't contain CheckpointResult
- We can safely clone the phase to check state
- The result stays in place and is only taken at the end
Also adds PartialEq derive to CheckpointMode.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove wal_checkpoint_start() function
- Remove wal_checkpoint_finish() function
- Remove wal_checkpoint() function
- Add blocking_checkpoint() as a convenience wrapper around checkpoint()
- Update checkpoint_shutdown() to take sync_mode parameter and use blocking_checkpoint()
The checkpoint() function now handles all the checkpoint state machine logic
internally, so the two-phase API is no longer needed.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
- Remove OpCheckpointState enum (no longer needed)
- Remove op_checkpoint_inner function
- Simplify op_checkpoint to call checkpoint() directly
- Remove op_checkpoint_state field from ProgramState
- Fix MVCC path to properly return after completing (was using break)
The checkpoint() function now handles all the state machine logic internally,
so the VDBE op just needs to call it and handle the result.
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
Now that checkpoint() handles sync and truncation internally, we can
simplify commit_dirty_pages_inner:
- Rename CommitState::Checkpoint to CommitState::AutoCheckpoint
- Remove CommitState::SyncDbFile and CommitState::WaitSyncDbFile states
- Remove CommitInfo::result field (no longer needed)
- Simplify the AutoCheckpoint state to just call checkpoint() and return
🤖 Generated with [Claude Code](https://claude.com/claude-code)
Co-Authored-By: Claude Opus 4.5 <noreply@anthropic.com>
