## Description
In certain edge cases, Into::into type inference fails and causes
library to fail to build in downstream crates
## Motivation and context
See https://github.com/ranile/turso-compile-fail
In this case, patching chrono to work with `ic-cdk` fails compilation of
turso. `ic-cdk` adds a dependency on `candid`, which provides some trait
implementations that make Into::into type inference break.
<details><summary>Error message</summary>
```
error[E0283]: type annotations needed
--> /Users/me/.cargo/git/checkouts/turso-455557cd4a2364c7/2d78b53/core/mvcc/database/checkpoint_state_machine.rs:216:71
|
216 | .is_some_and(|txid_max_old| b <= txid_max_old.into())
| -- ^^^^
| |
| type must be known at this point
|
= note: multiple `impl`s satisfying `u64: PartialOrd<_>` found in the following crates: `candid`, `core`:
- impl PartialOrd for u64;
- impl PartialOrd<candid::types::number::Int> for u64;
- impl PartialOrd<candid::types::number::Nat> for u64;
help: try using a fully qualified path to specify the expected types
|
216 - .is_some_and(|txid_max_old| b <= txid_max_old.into())
216 + .is_some_and(|txid_max_old| b <= <std::num::NonZero<u64> as Into<T>>::into(txid_max_old))
|
error[E0283]: type annotations needed
--> /Users/me/.cargo/git/checkouts/turso-455557cd4a2364c7/2d78b53/core/mvcc/database/checkpoint_state_machine.rs:226:67
|
226 | .is_some_and(|txid_max_old| e <= txid_max_old.into())
| -- ^^^^
| |
| type must be known at this point
|
= note: multiple `impl`s satisfying `u64: PartialOrd<_>` found in the following crates: `candid`, `core`:
- impl PartialOrd for u64;
- impl PartialOrd<candid::types::number::Int> for u64;
- impl PartialOrd<candid::types::number::Nat> for u64;
help: try using a fully qualified path to specify the expected types
|
226 - .is_some_and(|txid_max_old| e <= txid_max_old.into())
226 + .is_some_and(|txid_max_old| e <= <std::num::NonZero<u64> as Into<T>>::into(txid_max_old))
|
error[E0283]: type annotations needed
--> /Users/me/.cargo/git/checkouts/turso-455557cd4a2364c7/2d78b53/core/mvcc/database/checkpoint_state_machine.rs:242:90
|
242 | .is_none_or(|txid_max_old| begin_ts.is_some_and(|b| b > txid_max_old.into()));
| - ^^^^
| |
| type must be known at this point
|
= note: multiple `impl`s satisfying `u64: PartialOrd<_>` found in the following crates: `candid`, `core`:
- impl PartialOrd for u64;
- impl PartialOrd<candid::types::number::Int> for u64;
- impl PartialOrd<candid::types::number::Nat> for u64;
help: try using a fully qualified path to specify the expected types
|
242 - .is_none_or(|txid_max_old| begin_ts.is_some_and(|b| b > txid_max_old.into()));
242 + .is_none_or(|txid_max_old| begin_ts.is_some_and(|b| b > <std::num::NonZero<u64> as Into<T>>::into(txid_max_old)));
|
For more information about this error, try `rustc --explain E0283`.
error: could not compile `turso_core` (lib) due to 3 previous errors
```
</details>
## Description of AI Usage
All code is hand-written
Reviewed-by: Nikita Sivukhin (@sivukhin)
Closes#4293
## Description
in `op_new_rowid` we already have code logic that encodes how to get the
last rowid correctly, this PR uses advantage of it in MVCC too but with
a few `lock` guards in place to not collide rowids
## Motivation and context
It is hard to maintain two ways of getting a new rowid so this tries to
fold mvcc with btree
## Description of AI Usage
None
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4228
The DX is right now pretty terrible:
```
penberg@vonneumann turso % cargo run -- hello.db
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.15s
Running `target/debug/tursodb hello.db`
Turso v0.4.0-pre.18
Enter ".help" for usage hints.
Did you know that Turso supports live materialized views? Type .manual materialized-views to learn more.
This software is in BETA, use caution with production data and ensure you have backups.
turso> PRAGMA journal_mode = 'experimental_mvcc';
× Invalid argument supplied: MVCC is not enabled. Enable it with `--experimental-mvcc` flag in the CLI or by setting the MVCC option in `DatabaseOpts`
turso>
```
To add insult to the injury, many SDKs don't even have a way to enable
MVCC via database options. Therefore, let's remove the flag altogether.
RowidAllocator is a centralized lock protected rowid allocator that is
used to ask for a new rowid. The idea is to have single atomic i64 that
we can increment when we get asked to allocate a new rowid.
the DB file, as if self.checkpointed_txid_max_old == None it could mean
the MvStore recently initialized or we are dealing with an empty
database. In both cases, we cannot assert the row version exists in the
db file
<!-- CURSOR_SUMMARY -->
> [!NOTE]
> Clears `state` in `MvccLazyCursor` when `next()` reaches `End` and
when `prev()` reaches `BeforeFirst`, preventing stale iteration state.
>
> - **MVCC Cursor (`core/mvcc/cursor.rs`)**:
> - `next()`: when reaching `CursorPosition::End`, now resets
`self.state` before returning.
> - `prev()`: when reaching `CursorPosition::BeforeFirst`, now resets
`self.state` before returning.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
dc8ae2c2e4. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Closes#4161
## Description
If we use `btree()`, it creates a clone of the value inside the `Table`,
which means the number of ref counts > 1. This means that if we you try
to use `Arc::make_mut`, it will just clone the Btree table, and never
change the actual schema
<!--
Please include a summary of the changes and the related issue.
-->
## Motivation and context
My fuzzer was failing in #4074 with negative rootpages
<!--
Please include relevant motivation and context.
Link relevant issues here.
-->
## AI Disclosure
None
<!--
Please disclose if any LLM's were used in the creation of this PR and to
what extent,
to help maintainers properly review.
-->
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4181
Sadly, due to how we use dual cursors, we cannot use optimization under
btree cursor to count rows without first checking if the row in btree is
valid. So this is a slow count implementation.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Adds a state-driven `count()` implementation that iterates via dual
cursors, validating B-Tree keys and tallying visible rows.
>
> - **Core (MVCC Cursor)**:
> - **Counting**:
> - Implement `count()` using a small state machine (`CountState`)
to iterate (`rewind` → `next`) and tally rows, ensuring B-Tree keys are
validated via existing dual-cursor logic.
> - **State Management**:
> - Add `CountState` enum and `count_state` field to
`MvccLazyCursor` to keep count logic isolated from other cursor states.
> - Initialize `count_state` in `new()`.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
356ea0869d. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4160
## Closes
- Closes#4017
- Addresses #4043; this now fails with `Page cache is full` with 100k
pages, which is a separate non-corruption issue. Modifying max page
cache size to be 10 million pages makes it not finish at all. We should
modify the issue after this is merged to reflect what the new problem
is. The queries in the issue (#4043) create a WAL that is at least 1.7
GB in size
## Background
We have an optimization in the btree where if:
- We want to reach the rightmost leaf page, and
- We know the rightmost page and are already on it
Then we can skip a seek.
## Problem
The problem is this optimization should NEVER be used in cases where we
cannot be sure that the btree wasn't modified from under us e.g. by a
trigger subprogram.
## Fix
Hence, disable it when we are executing a parent program that has
triggers which will fire.
## AI Disclosure
No AI was used for this PR.
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#4135
The idea is to have a custom `all-mvcc.test` so we can add `.test` files
that we expect to work with MVCC. In cases where files are not enough we
have `is_turso_mvcc` to check if we want to run a test.
For example we skip partial index tests like this:
```
if {![is_turso_mvcc]} {
do_execsql_test_on_specific_db {:memory:} autoinc-conflict-on-nothing {
CREATE TABLE t (id INTEGER PRIMARY KEY AUTOINCREMENT, k TEXT);
CREATE UNIQUE INDEX idx_k_partial ON t(k) WHERE id > 1;
INSERT INTO t (k) VALUES ('a');
INSERT INTO t (k) VALUES ('a');
INSERT INTO t (k) VALUES ('a') ON CONFLICT DO NOTHING;
INSERT INTO t (k) VALUES ('b');
SELECT * FROM t ORDER BY id;
} {1|a 2|a 4|b}
}
```
`test-mvcc-compat` is not run under CI for now as we need to fix every
test anyways so no point in making every PR fail for now.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4139
We have an optimization in the btree where if:
- We want to reach the rightmost leaf page, and
- We know the rightmost page and are already on it
Then we can skip a seek.
The problem is this optimization should NEVER be used in cases where we cannot be sure that the btree wasn't modified from under us e.g. by a trigger subprogram.
Hence, disable it when we are executing a parent program that has triggers which
will fire.
No AI was used for this PR.
for example, upon opening an existing database, all the rows are in
the btree, so if we seek only from MV store, we won't find anything.
ergo: we must look from both the mv store and the btree. if we are
iterating forwards, the smallest of the two results is where we land,
and vice versa for backwards iteration.
initially this implementation used blocking IO but was refactored to
use state machines after the rest of the Cursor methods in the MVCC cursor
module were refactored to do that too.
---
this PR was initially almost entirely written using Claude Code + Opus 4.5,
but heavily manually cleaned up as the AI made the state machine refactor
far too complicated.
existing bootstrap had a half-baked implementation for getting
root pages from the DB file via a Statement, but it's unnecessary
because we have already parsed the schema at that point, so we can
just use it directly.
- Do not assume all cursors are mvcc cursors in `op_new_rowid`
- Fix bug in logical log reader where it would ignore already buffered
bytes and try to read too much past the end of the file
- Add safeguard to the same reader logic where it issues multiple reads
if it gets a short read for some reason
Both of these issues were found using `cd simulator && cargo run --
--profile simple_mvcc`, using `num_connections=1`
Closes#4077
