Improve errors messages for sync engine:
- Error for unexpected metadata file format
```
sync engine operation failed: deserialization error: unexpected metadata file format, 'version' field must be present and have string type
```
- Error from the server:
```
sync engine operation failed: database sync engine error: remote server returned an error: status=401, body={"error":"Unauthorized: `unauthorized access attempt on database: empty JWT token`"}
```
Closes#4155
Sadly, due to how we use dual cursors, we cannot use optimization under
btree cursor to count rows without first checking if the row in btree is
valid. So this is a slow count implementation.
<!-- CURSOR_SUMMARY -->
---
> [!NOTE]
> Adds a state-driven `count()` implementation that iterates via dual
cursors, validating B-Tree keys and tallying visible rows.
>
> - **Core (MVCC Cursor)**:
> - **Counting**:
> - Implement `count()` using a small state machine (`CountState`)
to iterate (`rewind` → `next`) and tally rows, ensuring B-Tree keys are
validated via existing dual-cursor logic.
> - **State Management**:
> - Add `CountState` enum and `count_state` field to
`MvccLazyCursor` to keep count logic isolated from other cursor states.
> - Initialize `count_state` in `new()`.
>
> <sup>Written by [Cursor
Bugbot](https://cursor.com/dashboard?tab=bugbot) for commit
356ea0869d. This will update automatically
on new commits. Configure
[here](https://cursor.com/dashboard?tab=bugbot).</sup>
<!-- /CURSOR_SUMMARY -->
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4160
turso cloud generate urls with `libsql://` protocol by default
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Reviewed-by: Pedro Muniz (@pedrocarlo)
Closes#4159
On bootstrap just store the header but not flush it to disk. Only try to
flush it when we start an MVCC transaction. Also applied fix in
`OpenDup` where we should not wrap an ephemeral table with an MvCursor
Reviewed-by: Mikaël Francoeur (@LeMikaelF)
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4151
CTEs now work correctly when combined with UNION, UNION ALL, INTERSECT,
and EXCEPT.
**Before:**
```sql
WITH t AS (SELECT 1 as x) SELECT * FROM t UNION ALL SELECT 2 as x
-- Error: Parse error: no such table: t
```
**After:**
```sql
WITH t AS (SELECT 1 as x) SELECT * FROM t UNION ALL SELECT 2 as x
-- Works correctly, returns rows (1) and (2)
```
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4123
This significantly simplifies using the Rust binding (and adds a more
complete example).
By adding prepare to Transaction (that just passes it onto the
Transactions connection) we can write simpler application code that is
passed a transaction. This allows multiple database update methods to be
flexibly grouped into transactions, even when they use prepared
statements.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4113
This PR introduces fuzz test which maintains 2 database files and
periodically switch the "engine" which process operations over this
files between sqlite and turso. The purpose of this fuzz test is to
ensure compatibility with on-disk data format and logic depending on it
between sqlite and turso.
With this test, few bugs were identified:
1. automatic indices order were incompatible with sqlite in turso
because we processed constraints first and columns later (so, for cases
like `CREATE TABLE t (x UNIQUE, y, UNIQUE (y))` the order of indices
were `y` and `x` while sqlite expected that first autoindex is over `x`
and second is over `y`
2. after collecting `unique_sets` we need to process them in exactly
same order for the same reason describe above
3. now all write queries require subtransaction journal because without
this (with previous extra condition `if
!self.table_references.is_empty()`) we incorrectly processed `CREATE
UNIQUE INDEX` queries in case when they failed with `CONSTRAINT` error
(because column is not unique)
4. Table access method decision fixed in order to not use index with
collation different from table column collation
We still have at least one more bug which can be identified by
uncommenting fuzz test configuration (see issue
https://github.com/tursodatabase/turso/issues/4154):
```
// temporary disable this action - because right now we still have bug with affinity for insertion to the indices
// 35..=55 => Action::InsertData, // ~20%
```
## AI disclosure
The fuzz test was mainly written by `openai/gpt-5` model with the
following prompt
<details>
[Code output="mod.rs" model="openai/gpt-5" language="rust"]
Write fuzz test which will simulate different data layout in SQLite and
switch connections between each other periodically.
The purpose of the test is to identify potential incompatibilities in
data layou between sqlite3 and turso.
From the layout perspective you MUST randomize following things:
1. UNIQUE constraint placement:
* Mix 3 different options:
```sql
CREATE TABLE t (x, y UNIQUE);
CREATE TABLE t (x, y, UNIQUE (y));
CREATE UNIQUE INDEX t_idx ON t (y);
```
2. UNIQUE constraint column order - it must be randomized and multiple
column constraints must be fuzzed
3. Create INDEX column order
4. Column collation
5. Sort order for index columns
6. Amount of columns
7. AUTOINCREMENT columns (they are affecting data on disk as they force
creation of sqlite_sequence table)
Test also must execute read queries in order to be able to capture
potential layout mismatch:
Use SELECT with random subset of columns from the table for that
purpose.
The test should be structured like this:
```rs
struct FuzzTestState {
tables: Vec<FuzzTestTable>,
indices: Vec<FuzzTestIndex>,
}
#[test]
pub fn test_data_layout_compatibility() {
const OUTER: usize = 100;
const INNER: usize = 100;
let (mut rng, seed) = rng_from_time_or_env();
tracing::info!("test_data_layout_compatibility seed: {}", seed);
let left = NamedTempFile::new().unwrap();
let right = NamedTempFile::new().unwrap();
let state = FuzzTestState::new();
for i in 0..OUTER {
let turso_path = if i % 2 == 0 { left.path() } else { right.path() };
let sqlite_path = if i % 2 == 1 { left.path() } else { right.path() };
let turso = TempDatabase::builder().with_db_path(turso_path).build();
let sqlite = TempDatabase::builder().with_db_path(sqlite_path).build();
for _ in 0..INNER {
// use state to pick random valid action for a database among the list:
// 1. create table
// 2. create index
// 3. select data from existing table
let sql = "...";
// execute sql against turso and sqlite and compare the generated rows
}
}
}
```
Use following snippet to understand style of tests and availble methods:
[Shell cmd="cat ./mod.rs | head -n 1000" /]
[/Code]
</details>
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4153
## FIxes to simulator transactions
1. Rollback all sim transactions when `REOPEN_DATABASE` happens
2. Handle recoverable errors: `WriteWriteConflict` causes a rollback,
`TxError` (e.g. no transaction is active when trying to `COMMIT`) does
not -- in either case, don't shadow the results of the query but don't
fail the sim either
3. The issue with `apply_snapshot` in the current sim was that the sim
was executing queries on commit, instead of applying their effects. For
example, if transaction T1 did a `DELETE FROM t WHERE TRUE` it would
delete all the rows even if another mvcc transaction T2 had inserted
rows concurrently that should not be visible to T1. This was not a
problem in non-MVCC due to single writer, but it doesn't work for MVCC.
Instead, record the operations and then apply them in order on commit.
## Fixes to property / assertion handling
- e.g. in "ReadYourUpdatesBack", we first do an UPDATE and then a SELECT
to verify that those updates were actually made. Problem is the SELECT
assertion was made even if the UPDATE failed. Fix: change interaction
handling so that if a precondition in a property fails, the entire
property is abandoned.
## AI usage
Pretty much all of the PR is again generated by prompting Claude Code +
Opus 4.5 by giving it failing simulator seeds and CLI flags.
Reviewed-by: Pedro Muniz (@pedrocarlo)
Closes#4145
Currently the simulator running on AWS will skip posting issues that are
too similar to existing ones, to avoid spamming the issue tracker.
However, this loses some information about how frequent a given failure
is, so a better solution is to comment on the existing issue whenever a
similar failure occurs.
Changes:
- Track issue numbers alongside titles in openIssues
- Add commentOnIssue() to post comments on duplicate issues
- Extract shared createFaultDetails() from issue/comment body creation
## AI usage
Whole PR written using Opus 4.5. Prompt:
> @simulator-docker-runner/docker-entrypoint.simulator.ts @simulator-
docker-runner/github.ts Let's change this so that if it finds a
duplicate issue, it comments on the existing issue.
https://github.com/octokit/octokit.js
and
> Can we share code between createCommentBody and createIssueBody? They
should be able to be very similar
Reviewed-by: Pekka Enberg <penberg@iki.fi>
Closes#4143
This PR fixes the sqlite3 compatibility bug as turso right now always
sort unique constraint columns in the order of **table** columns
definition.
This makes tursodb incompatible with sqlite3 - because sqlite3 expect
unique autoindex to have columns in the order of **constraint**
definition while turso will reorder them.
Consider following simple example:
```
$> sqlite3 no-compat.db
sqlite> create table t(a, b, UNIQUE (b, a));
sqlite> insert into t values (1, 2);
sqlite> select a, b from t;
1|2
$> tursodb no-compat.db
turso> select a, b from t;
2|1
```
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#4149
e.g. in "ReadYourUpdatesBack", we first do an UPDATE and then a
SELECT to verify that those updates were actually made.
Problem is the SELECT assertion was made even if the UPDATE failed.
Fix: change interaction handling so that if a precondition in a
property fails, the entire property is abandoned.
This PR adds asyncio support to the python driver
We still ship single package but user can choose the configuration by
using different submodules:
```py
import turso # blocking driver for turso database
import turso.sync # blocking driver for syncrhonization engine on top of the turso database
import turso.aio # non-blocking driver for turso database
import turso.aio.sync # non-blocking driver for syncrhonization engine on top of the turso database
```
Closes#4137
