## Description
In certain edge cases, Into::into type inference fails and causes
library to fail to build in downstream crates
## Motivation and context
See https://github.com/ranile/turso-compile-fail
In this case, patching chrono to work with `ic-cdk` fails compilation of
turso. `ic-cdk` adds a dependency on `candid`, which provides some trait
implementations that make Into::into type inference break.
<details><summary>Error message</summary>
```
error[E0283]: type annotations needed
--> /Users/me/.cargo/git/checkouts/turso-455557cd4a2364c7/2d78b53/core/mvcc/database/checkpoint_state_machine.rs:216:71
|
216 | .is_some_and(|txid_max_old| b <= txid_max_old.into())
| -- ^^^^
| |
| type must be known at this point
|
= note: multiple `impl`s satisfying `u64: PartialOrd<_>` found in the following crates: `candid`, `core`:
- impl PartialOrd for u64;
- impl PartialOrd<candid::types::number::Int> for u64;
- impl PartialOrd<candid::types::number::Nat> for u64;
help: try using a fully qualified path to specify the expected types
|
216 - .is_some_and(|txid_max_old| b <= txid_max_old.into())
216 + .is_some_and(|txid_max_old| b <= <std::num::NonZero<u64> as Into<T>>::into(txid_max_old))
|
error[E0283]: type annotations needed
--> /Users/me/.cargo/git/checkouts/turso-455557cd4a2364c7/2d78b53/core/mvcc/database/checkpoint_state_machine.rs:226:67
|
226 | .is_some_and(|txid_max_old| e <= txid_max_old.into())
| -- ^^^^
| |
| type must be known at this point
|
= note: multiple `impl`s satisfying `u64: PartialOrd<_>` found in the following crates: `candid`, `core`:
- impl PartialOrd for u64;
- impl PartialOrd<candid::types::number::Int> for u64;
- impl PartialOrd<candid::types::number::Nat> for u64;
help: try using a fully qualified path to specify the expected types
|
226 - .is_some_and(|txid_max_old| e <= txid_max_old.into())
226 + .is_some_and(|txid_max_old| e <= <std::num::NonZero<u64> as Into<T>>::into(txid_max_old))
|
error[E0283]: type annotations needed
--> /Users/me/.cargo/git/checkouts/turso-455557cd4a2364c7/2d78b53/core/mvcc/database/checkpoint_state_machine.rs:242:90
|
242 | .is_none_or(|txid_max_old| begin_ts.is_some_and(|b| b > txid_max_old.into()));
| - ^^^^
| |
| type must be known at this point
|
= note: multiple `impl`s satisfying `u64: PartialOrd<_>` found in the following crates: `candid`, `core`:
- impl PartialOrd for u64;
- impl PartialOrd<candid::types::number::Int> for u64;
- impl PartialOrd<candid::types::number::Nat> for u64;
help: try using a fully qualified path to specify the expected types
|
242 - .is_none_or(|txid_max_old| begin_ts.is_some_and(|b| b > txid_max_old.into()));
242 + .is_none_or(|txid_max_old| begin_ts.is_some_and(|b| b > <std::num::NonZero<u64> as Into<T>>::into(txid_max_old)));
|
For more information about this error, try `rustc --explain E0283`.
error: could not compile `turso_core` (lib) due to 3 previous errors
```
</details>
## Description of AI Usage
All code is hand-written
Reviewed-by: Nikita Sivukhin (@sivukhin)
Closes#4293
## Description
in `op_new_rowid` we already have code logic that encodes how to get the
last rowid correctly, this PR uses advantage of it in MVCC too but with
a few `lock` guards in place to not collide rowids
## Motivation and context
It is hard to maintain two ways of getting a new rowid so this tries to
fold mvcc with btree
## Description of AI Usage
None
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4228
The DX is right now pretty terrible:
```
penberg@vonneumann turso % cargo run -- hello.db
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.15s
Running `target/debug/tursodb hello.db`
Turso v0.4.0-pre.18
Enter ".help" for usage hints.
Did you know that Turso supports live materialized views? Type .manual materialized-views to learn more.
This software is in BETA, use caution with production data and ensure you have backups.
turso> PRAGMA journal_mode = 'experimental_mvcc';
× Invalid argument supplied: MVCC is not enabled. Enable it with `--experimental-mvcc` flag in the CLI or by setting the MVCC option in `DatabaseOpts`
turso>
```
To add insult to the injury, many SDKs don't even have a way to enable
MVCC via database options. Therefore, let's remove the flag altogether.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4294
The DX is right now pretty terrible:
```
penberg@vonneumann turso % cargo run -- hello.db
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.15s
Running `target/debug/tursodb hello.db`
Turso v0.4.0-pre.18
Enter ".help" for usage hints.
Did you know that Turso supports live materialized views? Type .manual materialized-views to learn more.
This software is in BETA, use caution with production data and ensure you have backups.
turso> PRAGMA journal_mode = 'experimental_mvcc';
× Invalid argument supplied: MVCC is not enabled. Enable it with `--experimental-mvcc` flag in the CLI or by setting the MVCC option in `DatabaseOpts`
turso>
```
To add insult to the injury, many SDKs don't even have a way to enable
MVCC via database options. Therefore, let's remove the flag altogether.
This PR introduces local sync server and run tests for Go and Python
against it in CI
The local sync server implements 2 endpoints:
1. `/v2/pipeline` - subset of SQL over HTTP protocol (Hrana) to execute
logical push operations from the client
2. `/pull-updates` - endpoint which returns page updates for client to
apply locally
The implementation is based on the local database file with **disabled
checkpoint** in order to preserve whole DB history and allow server to
respond to client which can have arbitrary stale DB.
For implementation, sync server uses extra API exposed by the turso-core
under `conn-raw-api` feature which includes `wal_state` /
`wal_get_frame` methods.
Usage:
- `tursodb --sync-server 0.0.0.0:8080` - in-memory database
- `tursodb local.db --sync-server 0.0.0.0:8080` - local db file
Closes#4191
RowidAllocator is a centralized lock protected rowid allocator that is
used to ask for a new rowid. The idea is to have single atomic i64 that
we can increment when we get asked to allocate a new rowid.
## Description
Add some readonly checks in header validation and `pragma journal_mode`
. Depends on #4279 being merged first, to avoid conflicts here.
<!--
Please include a summary of the changes and the related issue.
-->
## Motivation and context
Close#4270
<!--
Please include relevant motivation and context.
Link relevant issues here.
-->
## Description of AI Usage
AI again did most of the work here, as it is pretty basic stuff and
mostly boilerplate. The main usefullness for AI here was to write the
tests to check for these edge cases.
**Prompt:**
```
I want to Make sure readonly databases cannot modify header page on Database open nor call `pragma journal mode` to update the journal mode. I need you
to implement the necessary checks to ensure we can still continue working normally and emit warnings to show that we cannot change to mvcc. Lastly add
tests in `header_version.rs` to prove your modifications works.
```
<!--
Please disclose how AI was used to help create this PR. For example, you
can share prompts,
specific tools, or ways of working that you took advantage of. You can
also share whether the
creation of the PR was mainly driven by AI, or whether it was used for
assistance.
This is a good way of sharing knowledge to other contributors about how
we can work more efficiently with
AI tools. Note that the use of AI is encouraged, but the committer is
still fully responsible for understanding
and reviewing the output.
-->
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#4280
## Description
This PR greatly simplifies the slot bitmap used to track free pages in
the arena buffer pool.
## Motivation and context
An optimization was included that would allow for allocating multiple
contiguous buffers, with the objective being that they would be
coalesced into single buffers when submitting `pwritev` calls, for
things like WAL appends. This optimization was never implemented and we
are left with a very complex bitmap with lots of unused/unnecessary
logic.
## Description of AI Usage
This was mostly codex 5.2 with the prompt:
```
This project is a SQLite rewrite in Rust, it uses a BufferPool that allocates large arenas
and tracks which slots are free using a bitmap. this bitmap is core/storage/slot_bitmap.rs..
it was originally designed in a way that would allow to request multiple buffers that
were contiguous in memory, so that they could be coalesced into a single `pwrite` operation later down the line.
However this optimization was never implemented and the bitmap has a complex 'two-pointer' algorithm that we no
longer need. please rewrite this slot_bitmap.rs to simplify and only allocate single buffers at a time, removing the need
for the two pointer hint system.
```
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#4277
## Description
Remove sync IO hacks for `op_journal_mode`
Close#4268
<!--
Please include a summary of the changes and the related issue.
-->
## Motivation and context
Remove sync io hacks so it is friendlier for WASM
<!--
Please include relevant motivation and context.
Link relevant issues here.
-->
## Description of AI Usage
Ai basically made the bulk refactoring and I made some adjustments and
trimmed down the implementation
**Prompt**:
```
if look at @core/storage/journal_mode.rs and `op_journal_mode` in `execute.rs` you will see that we have some blocking io operations with
`pager.io.block` and `program.connection.checkpoint` that also blocks. I want you refactor the code to use state machines similar in nature to how we do it
in many functions in `execute.rs`
```
<!--
Please disclose how AI was used to help create this PR. For example, you
can share prompts,
specific tools, or ways of working that you took advantage of. You can
also share whether the
creation of the PR was mainly driven by AI, or whether it was used for
assistance.
This is a good way of sharing knowledge to other contributors about how
we can work more efficiently with
AI tools. Note that the use of AI is encouraged, but the committer is
still fully responsible for understanding
and reviewing the output.
-->
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4279
## Description
We should correctly check for completion success
<!--
Please include a summary of the changes and the related issue.
-->
## Motivation and context
<!--
Please include relevant motivation and context.
Link relevant issues here.
-->
## Description of AI Usage
Ai made the tests for me
<!--
Please disclose how AI was used to help create this PR. For example, you
can share prompts,
specific tools, or ways of working that you took advantage of. You can
also share whether the
creation of the PR was mainly driven by AI, or whether it was used for
assistance.
This is a good way of sharing knowledge to other contributors about how
we can work more efficiently with
AI tools. Note that the use of AI is encouraged, but the committer is
still fully responsible for understanding
and reviewing the output.
-->
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4263
## Description
closes https://github.com/tursodatabase/turso/issues/4142
<!--
Please include a summary of the changes and the related issue.
-->
## Motivation and context
compatibility, we were wrongly rewriting table qualified cols, also
added trigger.test to all.test and expect correct values in a test
<!--
Please include relevant motivation and context.
Link relevant issues here.
-->
## AI Disclosure
None
<!--
Please disclose if any LLM's were used in the creation of this PR and to
what extent,
to help maintainers properly review.
-->
Closes#4206
## Description
Closes#4146
## Motivation and context
panics are bad
## AI Disclosure
none used
Reviewed-by: Pedro Muniz (@pedrocarlo)
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#4177
Closes#3536
# Description
This PR implements **dynamic journal mode switching** via `PRAGMA
journal_mode`, allowing users to switch between WAL and MVCC modes at
runtime.
### Key Changes
**Core Feature: Journal Mode Switching**
- Added new `JournalMode` module (`core/storage/journal_mode.rs`) to
parse and handle journal mode transitions
- Modified `op_journal_mode` to correctly parse journal modes and update
the database header
- Emit checkpoint when setting a new journal mode to ensure data
consistency
- Added MVCC checkpoint support to `Connection::checkpoint`
**Database Initialization Improvements**
- Read DB header on `Database::open` and simplified `init_pager`
- Made `Version` an enum for better comparison semantics
- Automatically convert legacy SQLite databases to WAL mode
- Ensure DB header is flushed to disk when header changes during open
- Clear page cache after header validation
**Bug Fixes**
- Fixed dirty page invalidation in pager when clearing dirty pages in
page cache
- Fixed `is_none_or` check for row version existence in DB file (handles
MvStore initialization and empty database cases)
- Added `btree_resident` field in `RowVersion` to track if
insert/deletion originated from a btree
**Testing**
- Added fuzz tests for `journal_mode` transitions with Database
operations in between
- Added integration tests for testing switching from the different modes
while checking the header version is correct
- Added some specific regression tests for delete operations lost on
mode switch
- Fixed `index_scan_compound_key_fuzz` to use separate databases for
Turso and SQLite in MVCC mode. Also had to decrease number of rows for
MVCC test, as insert was very slow.
# TODO's
- Remove sync hacks from `op_journal_mode`
- Expand fuzzer with different queries
- Add to Simulator
- Special handling for read only databases and not allow any header
changes
# Motivation and context
Facilitate our users to test MVCC and transition back and forth from it.
# AI Disclosure
Used AI to catch and fix bugs in MVCC, further my understanding with
MVCC, write tests in `tests` folder, most of the PR summary, and the
docs in the `docs/manual.md` file
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4074
This is already done in abort() but if the checkpoint error is
returned directly from commit_tx() instead of in an IO completion,
then the Tx state will be cleared and the DDL change won't be applied.
So, do it explicitly in step_end_write_txn() too.
Fixes this failing seed:
cargo run -- --memory-io --seed 12902559987470385066 --disable-heuristic-shrinking
## Description
We had a turso_stress failure happening on Antithesis. See the
associated issue for how to reproduce it. It happens only on io_uring.
This PR fixes the failure by ensuring writes to the WAL are completed
before the frame cache is updated. Without this, other threads can
retrieve a frame from the cache before the frame has been persisted.
## Motivation and context
Closes https://github.com/tursodatabase/turso/issues/4249
## Description of AI Usage
I used the reproducer in
https://github.com/tursodatabase/turso/issues/4249, and told Claude to
fix it. I then reviewed its work to make sure I understood it.
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#4252
the DB file, as if self.checkpointed_txid_max_old == None it could mean
the MvStore recently initialized or we are dealing with an empty
database. In both cases, we cannot assert the row version exists in the
db file
