## Description
BTreeCursor sets null flag to false once `seek` is called. This PR does
the same for MVCC
## Motivation and context
join.test failed with some cases due to this bug
## Description of AI Usage
I asked AI to find the issue but I ended showing the agent why he did
things wrong and that he should be ashamed
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4296
## Description
This PR adds missing affinity conversion to hash joins by applying
affinity conversion to build and probe keys before hashing.
```
turso> CREATE TABLE x(a INTEGER);
turso> CREATE TABLE y(b TEXT);
turso> INSERT INTO x VALUES (2),(3);
turso> INSERT INTO y VALUES ('02'),('2'),('2.0'),('3x'),('3.5');
turso> SELECT a, b
FROM x JOIN y ON a = b
ORDER BY a, b;
┌───┬─────┐
│ a │ b │
├───┼─────┤
│ 2 │ 02 │
├───┼─────┤
│ 2 │ 2 │
├───┼─────┤
│ 2 │ 2.0 │
└───┴─────┘
```
## Motivation and context
Fixes#3482.
Currently, Turso returns an empty result set:
```
turso> CREATE TABLE x(a INTEGER);
turso> CREATE TABLE y(b TEXT);
turso> INSERT INTO x VALUES (2),(3);
turso> INSERT INTO y VALUES ('02'),('2'),('2.0'),('3x'),('3.5');
turso> SELECT a, b
FROM x JOIN y ON a = b
ORDER BY a, b;
turso>
```
Expected behavior:
```
sqlite> CREATE TABLE x(a INTEGER);
sqlite> CREATE TABLE y(b TEXT);
sqlite> INSERT INTO x VALUES (2),(3);
sqlite> INSERT INTO y VALUES ('02'),('2'),('2.0'),('3x'),('3.5');
sqlite> SELECT a, b
...> FROM x JOIN y ON a = b
...> ORDER BY a, b;
2|02
2|2
2|2.0
```
## Description of AI Usage
This PR was developed with assistance from Claude Sonnet 4.5 through
code completions.
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#4317
using shared_mut() is unnecessary because the locks use atomics, and
in fact using shared_mut() can cause a deadlock, example:
- Thread 1 calls with_shared_mut() at in try_begin_read_tx
to claim/update a read mark slot. This waits for readers to release
their shared read locks.
- Thread 2 tries to use with_shared_mut() in end_read_tx() and also
can't proceed
Extract try_begin_read_tx with TryBeginReadResult::Retry for transient
conditions. begin_read_tx now retries with SQLite's quadratic backoff:
immediate retries for first 5 attempts, yield for 6-9, then quadratic
microsecond delays matching SQLite's formula.
BusySnapshot indicates the transaction's snapshot is permanently stale
and must be rolled back. Unlike Busy, retrying with busy_timeout will
never help - the caller must rollback and restart the transaction.
This PR fixes incorrect conversion from TEXT to INTEGER when text is a
number followed by a trailing non-breaking space.
This happens because `str::trim()` trims non-breaking space and unicode
whitespace while SQLite only trims ASCII whitespace.
Closes: https://github.com/tursodatabase/turso/issues/3679
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#3878
This PR introduces few sync fixes relevant to the partial sync feature:
1. Make `try_wal_watermark_read_page` async - this is important as now
db file can be partial and require extra network IO for fetching missing
pages
2. Do not panic if requested page doesn't exist on server - this can be
valid case if db on the server has smaller size
3. Maintain clean db file size in order to avoid reads outside of the
file (this can happen, when we revert new pages allocated only in the
WAL and need to fetch their previous state if there is one)
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#4297
## Description
In certain edge cases, Into::into type inference fails and causes
library to fail to build in downstream crates
## Motivation and context
See https://github.com/ranile/turso-compile-fail
In this case, patching chrono to work with `ic-cdk` fails compilation of
turso. `ic-cdk` adds a dependency on `candid`, which provides some trait
implementations that make Into::into type inference break.
<details><summary>Error message</summary>
```
error[E0283]: type annotations needed
--> /Users/me/.cargo/git/checkouts/turso-455557cd4a2364c7/2d78b53/core/mvcc/database/checkpoint_state_machine.rs:216:71
|
216 | .is_some_and(|txid_max_old| b <= txid_max_old.into())
| -- ^^^^
| |
| type must be known at this point
|
= note: multiple `impl`s satisfying `u64: PartialOrd<_>` found in the following crates: `candid`, `core`:
- impl PartialOrd for u64;
- impl PartialOrd<candid::types::number::Int> for u64;
- impl PartialOrd<candid::types::number::Nat> for u64;
help: try using a fully qualified path to specify the expected types
|
216 - .is_some_and(|txid_max_old| b <= txid_max_old.into())
216 + .is_some_and(|txid_max_old| b <= <std::num::NonZero<u64> as Into<T>>::into(txid_max_old))
|
error[E0283]: type annotations needed
--> /Users/me/.cargo/git/checkouts/turso-455557cd4a2364c7/2d78b53/core/mvcc/database/checkpoint_state_machine.rs:226:67
|
226 | .is_some_and(|txid_max_old| e <= txid_max_old.into())
| -- ^^^^
| |
| type must be known at this point
|
= note: multiple `impl`s satisfying `u64: PartialOrd<_>` found in the following crates: `candid`, `core`:
- impl PartialOrd for u64;
- impl PartialOrd<candid::types::number::Int> for u64;
- impl PartialOrd<candid::types::number::Nat> for u64;
help: try using a fully qualified path to specify the expected types
|
226 - .is_some_and(|txid_max_old| e <= txid_max_old.into())
226 + .is_some_and(|txid_max_old| e <= <std::num::NonZero<u64> as Into<T>>::into(txid_max_old))
|
error[E0283]: type annotations needed
--> /Users/me/.cargo/git/checkouts/turso-455557cd4a2364c7/2d78b53/core/mvcc/database/checkpoint_state_machine.rs:242:90
|
242 | .is_none_or(|txid_max_old| begin_ts.is_some_and(|b| b > txid_max_old.into()));
| - ^^^^
| |
| type must be known at this point
|
= note: multiple `impl`s satisfying `u64: PartialOrd<_>` found in the following crates: `candid`, `core`:
- impl PartialOrd for u64;
- impl PartialOrd<candid::types::number::Int> for u64;
- impl PartialOrd<candid::types::number::Nat> for u64;
help: try using a fully qualified path to specify the expected types
|
242 - .is_none_or(|txid_max_old| begin_ts.is_some_and(|b| b > txid_max_old.into()));
242 + .is_none_or(|txid_max_old| begin_ts.is_some_and(|b| b > <std::num::NonZero<u64> as Into<T>>::into(txid_max_old)));
|
For more information about this error, try `rustc --explain E0283`.
error: could not compile `turso_core` (lib) due to 3 previous errors
```
</details>
## Description of AI Usage
All code is hand-written
Reviewed-by: Nikita Sivukhin (@sivukhin)
Closes#4293
## Description
in `op_new_rowid` we already have code logic that encodes how to get the
last rowid correctly, this PR uses advantage of it in MVCC too but with
a few `lock` guards in place to not collide rowids
## Motivation and context
It is hard to maintain two ways of getting a new rowid so this tries to
fold mvcc with btree
## Description of AI Usage
None
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4228
The DX is right now pretty terrible:
```
penberg@vonneumann turso % cargo run -- hello.db
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.15s
Running `target/debug/tursodb hello.db`
Turso v0.4.0-pre.18
Enter ".help" for usage hints.
Did you know that Turso supports live materialized views? Type .manual materialized-views to learn more.
This software is in BETA, use caution with production data and ensure you have backups.
turso> PRAGMA journal_mode = 'experimental_mvcc';
× Invalid argument supplied: MVCC is not enabled. Enable it with `--experimental-mvcc` flag in the CLI or by setting the MVCC option in `DatabaseOpts`
turso>
```
To add insult to the injury, many SDKs don't even have a way to enable
MVCC via database options. Therefore, let's remove the flag altogether.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4294
The DX is right now pretty terrible:
```
penberg@vonneumann turso % cargo run -- hello.db
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.15s
Running `target/debug/tursodb hello.db`
Turso v0.4.0-pre.18
Enter ".help" for usage hints.
Did you know that Turso supports live materialized views? Type .manual materialized-views to learn more.
This software is in BETA, use caution with production data and ensure you have backups.
turso> PRAGMA journal_mode = 'experimental_mvcc';
× Invalid argument supplied: MVCC is not enabled. Enable it with `--experimental-mvcc` flag in the CLI or by setting the MVCC option in `DatabaseOpts`
turso>
```
To add insult to the injury, many SDKs don't even have a way to enable
MVCC via database options. Therefore, let's remove the flag altogether.
This PR introduces local sync server and run tests for Go and Python
against it in CI
The local sync server implements 2 endpoints:
1. `/v2/pipeline` - subset of SQL over HTTP protocol (Hrana) to execute
logical push operations from the client
2. `/pull-updates` - endpoint which returns page updates for client to
apply locally
The implementation is based on the local database file with **disabled
checkpoint** in order to preserve whole DB history and allow server to
respond to client which can have arbitrary stale DB.
For implementation, sync server uses extra API exposed by the turso-core
under `conn-raw-api` feature which includes `wal_state` /
`wal_get_frame` methods.
Usage:
- `tursodb --sync-server 0.0.0.0:8080` - in-memory database
- `tursodb local.db --sync-server 0.0.0.0:8080` - local db file
Closes#4191
RowidAllocator is a centralized lock protected rowid allocator that is
used to ask for a new rowid. The idea is to have single atomic i64 that
we can increment when we get asked to allocate a new rowid.
## Description
Add some readonly checks in header validation and `pragma journal_mode`
. Depends on #4279 being merged first, to avoid conflicts here.
<!--
Please include a summary of the changes and the related issue.
-->
## Motivation and context
Close#4270
<!--
Please include relevant motivation and context.
Link relevant issues here.
-->
## Description of AI Usage
AI again did most of the work here, as it is pretty basic stuff and
mostly boilerplate. The main usefullness for AI here was to write the
tests to check for these edge cases.
**Prompt:**
```
I want to Make sure readonly databases cannot modify header page on Database open nor call `pragma journal mode` to update the journal mode. I need you
to implement the necessary checks to ensure we can still continue working normally and emit warnings to show that we cannot change to mvcc. Lastly add
tests in `header_version.rs` to prove your modifications works.
```
<!--
Please disclose how AI was used to help create this PR. For example, you
can share prompts,
specific tools, or ways of working that you took advantage of. You can
also share whether the
creation of the PR was mainly driven by AI, or whether it was used for
assistance.
This is a good way of sharing knowledge to other contributors about how
we can work more efficiently with
AI tools. Note that the use of AI is encouraged, but the committer is
still fully responsible for understanding
and reviewing the output.
-->
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Reviewed-by: Preston Thorpe <preston@turso.tech>
Closes#4280
## Description
This PR greatly simplifies the slot bitmap used to track free pages in
the arena buffer pool.
## Motivation and context
An optimization was included that would allow for allocating multiple
contiguous buffers, with the objective being that they would be
coalesced into single buffers when submitting `pwritev` calls, for
things like WAL appends. This optimization was never implemented and we
are left with a very complex bitmap with lots of unused/unnecessary
logic.
## Description of AI Usage
This was mostly codex 5.2 with the prompt:
```
This project is a SQLite rewrite in Rust, it uses a BufferPool that allocates large arenas
and tracks which slots are free using a bitmap. this bitmap is core/storage/slot_bitmap.rs..
it was originally designed in a way that would allow to request multiple buffers that
were contiguous in memory, so that they could be coalesced into a single `pwrite` operation later down the line.
However this optimization was never implemented and the bitmap has a complex 'two-pointer' algorithm that we no
longer need. please rewrite this slot_bitmap.rs to simplify and only allocate single buffers at a time, removing the need
for the two pointer hint system.
```
Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>
Closes#4277
