Commit graph

11998 commits

Author SHA1 Message Date
Preston Thorpe
fa7b2ef2d9
Merge 'remove unneeded Result in exec unixepoch' from Juan V. García
Some checks failed
Run long fuzz tests and stress test / run-fuzz-tests (push) Has been cancelled
Run long fuzz tests and stress test / run-long-fuzz-tests (push) Has been cancelled
Run long fuzz tests and stress test / simple-stress-test (push) Has been cancelled
Build & Publish Go Driver / Build Rust and run Go tests (push) Has been cancelled
Java Tests / test (push) Has been cancelled
Build & publish @tursodatabase/database / db-bindings-aarch64-apple-darwin - node@20 (push) Has been cancelled
Build & publish @tursodatabase/database / db-bindings-aarch64-unknown-linux-gnu - node@20 (push) Has been cancelled
Build & publish @tursodatabase/database / db-bindings-wasm32-wasip1-threads - node@20 (push) Has been cancelled
Build & publish @tursodatabase/database / db-bindings-x86_64-pc-windows-msvc - node@20 (push) Has been cancelled
Build & publish @tursodatabase/database / db-bindings-x86_64-unknown-linux-gnu - node@20 (push) Has been cancelled
Build & publish @tursodatabase/database / sync-bindings-aarch64-apple-darwin - node@20 (push) Has been cancelled
Build & publish @tursodatabase/database / sync-bindings-aarch64-unknown-linux-gnu - node@20 (push) Has been cancelled
Build & publish @tursodatabase/database / sync-bindings-wasm32-wasip1-threads - node@20 (push) Has been cancelled
Build & publish @tursodatabase/database / sync-bindings-x86_64-pc-windows-msvc - node@20 (push) Has been cancelled
Build & publish @tursodatabase/database / sync-bindings-x86_64-unknown-linux-gnu - node@20 (push) Has been cancelled
Python / lint (push) Has been cancelled
Python / linux (x86_64) (push) Has been cancelled
Python / macos-arm64 (aarch64) (push) Has been cancelled
Python / sdist (push) Has been cancelled
Rust / test-limbo (push) Has been cancelled
Rust / test-sqlite (push) Has been cancelled
Rust Benchmarks+Nyrkiö / bench (push) Has been cancelled
Rust Benchmarks+Nyrkiö / clickbench (push) Has been cancelled
Rust Benchmarks+Nyrkiö / tpc-h-criterion (push) Has been cancelled
Build & Publish Go Driver / Publish Go driver (push) Has been cancelled
Build & publish @tursodatabase/database / Test DB bindings on browser@20 (push) Has been cancelled
Build & publish @tursodatabase/database / Publish (push) Has been cancelled
Build & publish @tursodatabase/database / Test DB bindings on Linux-x64-gnu - node@20 (push) Has been cancelled
Python / test (push) Has been cancelled
Python / Release (push) Has been cancelled
## Description
remove unneeded Result in exec unixepoch
## Motivation and context
I can't commit a lot of time to the project but I'm slowly trying to get
into the code and I'm currently taking a look at understanding enough to
implement https://github.com/tursodatabase/turso/issues/602 at least
partially.
While exploring the code for `unixepoch` I found what seems to be like
an unneeded `Result`, since the underlying APIs don't fail themselves.
## Description of AI Usage
No AI Usage, I'm trying to get into the code base the old-fashioned way
as a way to keep up my Rust skills more or less sharp (had to change
jobs and I no longer work with Rust as much :( )

Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #4292
2025-12-22 16:04:15 -05:00
Preston Thorpe
21e8fc93f1
Merge 'Fix squeue overflow issue in io_uring' from Preston Thorpe
## Description
The existing squeue overflow handling logic improperly error'd when
`squeue full` instead of best-effort pushing any overflow SQE's when
`flush_overflow` was called and then preserved others for later retry.
Also fixed bug where `empty` didn't take overflow SQE's into
consideration.
## Motivation and context
fixing 2043523817/j
ob/58714851849?pr=4326
## Description of AI Usage
nada

Reviewed-by: Pedro Muniz (@pedrocarlo)
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #4329
2025-12-22 16:03:38 -05:00
Juan V. García
2486b2a167
Merge branch 'tursodatabase:main' into main 2025-12-22 19:34:14 +01:00
Preston Thorpe
167f838e12
Merge 'Implement busy handlers/callbacks' from Preston Thorpe
## Description
Implement SQLite3 Busy Handler callback functions:
https://sqlite.org/c3ref/busy_handler.html
## Description of AI Usage
opus4.5 helped writing the tests and comments

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #4320
2025-12-22 12:35:14 -05:00
PThorpe92
e4c61aaf65
Add tests for busy handlers for sqlite3 C api 2025-12-22 12:23:16 -05:00
PThorpe92
e472cc7aa3
Add more unit tests for core/busy.rs 2025-12-22 12:23:16 -05:00
PThorpe92
d796eee5a6
Add busy handler plumbing to sqlite3 C api 2025-12-22 12:23:16 -05:00
PThorpe92
e1fbb7d619
Introduce/incorporate BusyHandler 2025-12-22 12:23:14 -05:00
PThorpe92
dd9692c83a
Add core/busy.rs with busy handler/busy timeout logic 2025-12-22 12:19:40 -05:00
Preston Thorpe
e19c89e77f
Merge 'Lexer/Parser Optimization and refactoring' from Khashayar Fereidani
Hey great project! In this PR I did some improvements for optimization
and code readability. please run the benchmarks yourself to see if you
like these changes. for me it yields 5-8% better performance specially
on long queries.
Note: I removed some `unwrap` from the `parser.rs`, please review those
parts, tests are running fine, but I'm unsure about soundness of what I
did there, If we need those unwraps it is easy to add unwrap to
`TokenType`.
## Description
I improved lexer in following ways:
1. Used `.get(self.offset..).unwrap_or(&[])` to eliminate `panic` code
handling generation.
2. Improved `eat_while` implementation to be faster and also easier to
work with.
For the second commit:
1. I moved `None` to `TokenType` itself, as it is slower to process
`Option<TokenType>`.
General improvement:
I marked some parts of code with `cold()` as they are unlikely branches.
I added `new` function to `Token` and it is much easier to read the
code.
## Motivation and context
Improving lexer and parser performance and also code readability.
## Description of AI Usage
None, I have copilot installed, I tried to use it for `new` function
refactoring parts it generated too many bugs that I rewrote those parts
myself again.

Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #4327
2025-12-22 12:19:04 -05:00
PThorpe92
d3b6df5e07
Fix squeue overflow issue in io_uring 2025-12-22 11:43:05 -05:00
Khashayar Fereidani
9bf05c9522 fix clippy warnings 2025-12-22 19:51:11 +03:30
Khashayar Fereidani
dcd7bd2570 Add TK_NONE variant to TokenType and utility methods 2025-12-22 19:22:49 +03:30
Preston Thorpe
b507c884ae
Merge 'Affinity fixes' from Pedro Muniz
Some checks are pending
Build & publish @tursodatabase/database / db-bindings-wasm32-wasip1-threads - node@20 (push) Waiting to run
Build & publish @tursodatabase/database / db-bindings-x86_64-pc-windows-msvc - node@20 (push) Waiting to run
Build & publish @tursodatabase/database / db-bindings-x86_64-unknown-linux-gnu - node@20 (push) Waiting to run
Build & publish @tursodatabase/database / sync-bindings-aarch64-apple-darwin - node@20 (push) Waiting to run
Build & publish @tursodatabase/database / sync-bindings-aarch64-unknown-linux-gnu - node@20 (push) Waiting to run
Build & publish @tursodatabase/database / sync-bindings-wasm32-wasip1-threads - node@20 (push) Waiting to run
Build & publish @tursodatabase/database / sync-bindings-x86_64-pc-windows-msvc - node@20 (push) Waiting to run
Build & publish @tursodatabase/database / sync-bindings-x86_64-unknown-linux-gnu - node@20 (push) Waiting to run
Rust / build-native (macos-latest) (push) Waiting to run
Build & publish @tursodatabase/database / Test DB bindings on Linux-x64-gnu - node@20 (push) Blocked by required conditions
Build & publish @tursodatabase/database / Test DB bindings on browser@20 (push) Blocked by required conditions
Python / sdist (push) Waiting to run
Python / configure-strategy (push) Waiting to run
Python / test (push) Blocked by required conditions
Python / lint (push) Waiting to run
Python / linux (x86_64) (push) Waiting to run
Python / macos-arm64 (aarch64) (push) Waiting to run
Python / Release (push) Blocked by required conditions
Rust / build-native (blacksmith-4vcpu-ubuntu-2404) (push) Waiting to run
Rust / build-native (windows-latest) (push) Waiting to run
Rust / clippy (push) Waiting to run
Rust / simulator (push) Waiting to run
Rust / cargo-fmt-check (push) Waiting to run
Rust / test-limbo (push) Waiting to run
Rust / test-sqlite (push) Waiting to run
Rust Benchmarks+Nyrkiö / tpc-h (push) Waiting to run
Rust Benchmarks+Nyrkiö / vfs-bench-compile (push) Waiting to run
Rust Benchmarks+Nyrkiö / bench (push) Waiting to run
Rust Benchmarks+Nyrkiö / clickbench (push) Waiting to run
Rust Benchmarks+Nyrkiö / tpc-h-criterion (push) Waiting to run
## Description
Emits Affinity instruction in places SQLite also emits affinity
instructions.
<!--
Please include a summary of the changes and the related issue.
-->
## Motivation and context
Closes https://github.com/tursodatabase/turso/issues/4154
Closes #3481
Closes https://github.com/tursodatabase/turso/issues/3477
<!--
Please include relevant motivation and context.
Link relevant issues here.
-->
## Description of AI Usage
Ai generated with my prompting and guidance.
https://opncd.ai/share/EC7FRkwT
<!--
Please disclose how AI was used to help create this PR. For example, you
can share prompts,
specific tools, or ways of working that you took advantage of. You can
also share whether the
creation of the PR was mainly driven by AI, or whether it was used for
assistance.
This is a good way of sharing knowledge to other contributors about how
we can work more efficiently with
AI tools. Note that the use of AI is encouraged, but the committer is
still fully responsible for understanding
and reviewing the output.
-->

Closes #4300
2025-12-22 10:21:17 -05:00
Preston Thorpe
2aca809511
Merge 'Read only fixes' from Pedro Muniz
This is take 2 of this PR. With most of the description from the other
PR intact.
https://github.com/tursodatabase/turso/pull/4208
If you want to test the change try running `cargo run --bin tursodb
test.db --readonly` in two separate shells in main and in this PR.
## Description
Avoid file locking in read only mode, so that multiple processes can
read from it. We don't really support multi processing yet, but this
allows us to run many databases in read-only mode in multiple processes.
Includes some fixes in unnecessarily opening WAL files and just
simplifying the WAL opening logic.
**EDIT:**
I mistankely thought that for some reason this was fixed after I closed
this PR, but I noticed that I was just dumb and forgot to rebase my test
runner PR to remove these commits. So I thought the issue was fixed in
main, when it was not.
<!--
Please include a summary of the changes and the related issue.
-->
## Motivation and context
I'm building a custom test runner and for parallelism, I need to run
some Databases in different processes in read-only mode.
<!--
Please include relevant motivation and context.
Link relevant issues here.
-->
## AI Disclosure
Used AI for a bug I was having regarding opening a WAL in read-only
mode, when it did not exist. It was great excuse to test opencode and
its share feature
https://opncd.ai/share/fDj8QAsz
<!--
Please disclose if any LLM's were used in the creation of this PR and to
what extent,
to help maintainers properly review.
-->

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #4309
2025-12-22 09:57:19 -05:00
pedrocarlo
6416d00f72 simplify affinity checking and avoid allocating string if it will not be used later 2025-12-22 11:14:24 -03:00
pedrocarlo
69eed773a2 clippy 2025-12-22 11:14:24 -03:00
pedrocarlo
3c782897b4 add tcl tests 2025-12-22 11:14:24 -03:00
pedrocarlo
af919d10a3 add affinity conversion in update + upsert 2025-12-22 11:14:24 -03:00
pedrocarlo
6f93eba58a add affinity emission code for in list 2025-12-22 11:14:23 -03:00
pedrocarlo
1159e561dd simplify wal creation and do not open the wal file 2 times in header validation. open_shared_if_exists will open a new file if needed 2025-12-22 11:13:35 -03:00
pedrocarlo
2a7b06ea79 allow read only databases to open without wal if wal is empty 2025-12-22 11:13:35 -03:00
pedrocarlo
5e691a74c8 delete the correct wal in turso cli test 2025-12-22 11:13:35 -03:00
pedrocarlo
9fcc49fe63 do not call checkpoint_shutdown on read only database + small open flags fixes 2025-12-22 11:13:35 -03:00
pedrocarlo
7f0315caf8 do not call lock_file when OpenFlags::ReadOnly 2025-12-22 11:13:35 -03:00
Preston Thorpe
37e8cf176a
Merge 'slightly adjust fixed unstable test' from Nikita Sivukhin
Follow up after 2257614423
dbef65c907312afe8e437dde763c4b

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #4325
2025-12-22 09:02:34 -05:00
fereidani
b2113add93 fix/perf: improve handling of input boundaries and simplify token parsing logic 2025-12-22 17:21:03 +03:30
Jussi Saurio
92e78466c9
Merge 'tcl: run PRAGMA journal_mode=experimental_mvcc with mvcc' from Pere Diaz Bou
## Description
run PRAGMA journal_mode=experimental_mvcc with mvcc
## Motivation and context
After https://github.com/tursodatabase/turso/pull/4294 we need a way run
with mvcc mode.
## Description of AI Usage
it wrote the remove first line

Closes #4298
2025-12-22 15:08:28 +02:00
Nikita Sivukhin
9923fdd2f7 fix clippy 2025-12-22 16:36:01 +04:00
Nikita Sivukhin
b6a63be40e slightly adjust fixed unstable test 2025-12-22 16:32:48 +04:00
Jussi Saurio
1085e78124 Merge 'CI test setup fixes + fix GroupCompletion bug' from Pedro Muniz
## CI speed improvements
Let's use Sccache for compiling our code faster, cargo nextest to run
tester faster, run all tests in CI, automatically terminate workflows if
you push a new commit (so we don't waste more money running code that we
don't care anymore).
Also https://github.com/useblacksmith/rust-cache is deprecated, so we
should be using `Swatinem/rust-cache@v2`
## CI test setup fixes
We were ignoring entire suites of tests in CI. Re-enable them and fix
the failures. Most of them were stale test setups, apart from https://gi
thub.com/tursodatabase/turso/pull/4311/commits/a67eb0006d8ac23dd85918309
303701959fdc988 which appears to be a real bug.
Also fix linking sqlite not working on windows for compat tests
## Runtime code fixes
a67eb0006d
dd85918309303701959fdc988 appears to be a real bug in GroupCompletion
code - `succeeded()` requires that the result be set to `Some(None)`
## Motivation and context
Closes https://github.com/tursodatabase/turso/issues/4266
<!--
Please include relevant motivation and context.
Link relevant issues here.
-->
## Description of AI Usage
Ai just edited the files for me, I was the one asked it do the changes
in the workflows
<!--
Please disclose how AI was used to help create this PR. For example, you
can share prompts,
specific tools, or ways of working that you took advantage of. You can
also share whether the
creation of the PR was mainly driven by AI, or whether it was used for
assistance.
This is a good way of sharing knowledge to other contributors about how
we can work more efficiently with
AI tools. Note that the use of AI is encouraged, but the committer is
still fully responsible for understanding
and reviewing the output.
-->

Closes #4311
2025-12-22 14:24:55 +02:00
Jussi Saurio
596d8e4abb Ensure mvcc and non-mvcc test cant share the same directory 2025-12-22 13:49:33 +02:00
Jussi Saurio
75a1dbea58 Enable conn_raw_api feature in core_tester 2025-12-22 13:48:01 +02:00
Jussi Saurio
2257614423 make test_db_concurrent_use() more robust 2025-12-22 12:52:50 +02:00
Jussi Saurio
84fb3d6f55 Dont compare against real sqlite on windows in c compat tests due to complexity 2025-12-22 12:52:50 +02:00
Jussi Saurio
9517c1290f don't test macro comments - theyre just an illustration 2025-12-22 12:24:06 +02:00
Jussi Saurio
29aed49ec2 fix completions: result must be Some for succeeded() be true 2025-12-22 12:24:06 +02:00
Jussi Saurio
08805d6315 fix test_pager_setup: page1 should be initialized properly 2025-12-22 12:24:06 +02:00
Jussi Saurio
206a42ef7e fix timediff test: comparing eq times should return zeros, not null 2025-12-22 12:24:06 +02:00
Jussi Saurio
b52661c28e fix setup_test_env(): initialize page 1 2025-12-22 12:24:06 +02:00
Jussi Saurio
a0ba3691dc fix/sdk-kit: fix whitespace difference in assertion 2025-12-22 12:24:06 +02:00
Jussi Saurio
00d7b5d8db fix page_cache tests not to use page id 1 which is never evictable 2025-12-22 12:24:06 +02:00
Jussi Saurio
e20bec6167 Fix incorrect error expectation in test_wal_stale_snapshot 2025-12-22 12:24:06 +02:00
pedrocarlo
9e08ffc04c use cargo nextest to run all tests and not only integration tests + remove sccache show stats 2025-12-22 12:24:06 +02:00
pedrocarlo
591cf05fbd try claude suggestions 2025-12-22 12:24:06 +02:00
Pekka Enberg
b4a331da38
Merge 'SDK tweaks' from Nikita Sivukhin
Few small tweaks in SDKs:
1. Export `ConnectionSync` type from Python SDK
2. Remove dependency on `conn_raw_api` and corresponding methods from
rust bindings - no one use them so let's drop support for this feature
completely
3. Add Waker dependency in the sdk-kit in order to integrate it later to
rust bindings
4. Add io_uring feature to the sdk-kit

Closes #4284
2025-12-22 11:59:02 +02:00
Pekka Enberg
f598dfa13d
Merge 'core/mvcc: set_null_flag(false) when seek is called' from Pere Diaz Bou
Some checks are pending
Build & publish @tursodatabase/database / db-bindings-x86_64-pc-windows-msvc - node@20 (push) Waiting to run
Build & publish @tursodatabase/database / db-bindings-x86_64-unknown-linux-gnu - node@20 (push) Waiting to run
Build & publish @tursodatabase/database / sync-bindings-aarch64-apple-darwin - node@20 (push) Waiting to run
Build & publish @tursodatabase/database / sync-bindings-aarch64-unknown-linux-gnu - node@20 (push) Waiting to run
Build & publish @tursodatabase/database / sync-bindings-wasm32-wasip1-threads - node@20 (push) Waiting to run
Build & publish @tursodatabase/database / sync-bindings-x86_64-pc-windows-msvc - node@20 (push) Waiting to run
Build & publish @tursodatabase/database / sync-bindings-x86_64-unknown-linux-gnu - node@20 (push) Waiting to run
Build & publish @tursodatabase/database / Test DB bindings on Linux-x64-gnu - node@20 (push) Blocked by required conditions
Build & publish @tursodatabase/database / Test DB bindings on browser@20 (push) Blocked by required conditions
Build & publish @tursodatabase/database / Publish (push) Blocked by required conditions
Python / configure-strategy (push) Waiting to run
Python / test (push) Blocked by required conditions
Python / lint (push) Waiting to run
Python / linux (x86_64) (push) Waiting to run
Python / macos-arm64 (aarch64) (push) Waiting to run
Python / sdist (push) Waiting to run
Python / Release (push) Blocked by required conditions
Rust / cargo-fmt-check (push) Waiting to run
Rust / build-native (blacksmith-4vcpu-ubuntu-2404) (push) Waiting to run
Rust / build-native (macos-latest) (push) Waiting to run
Rust / build-native (windows-latest) (push) Waiting to run
Rust / clippy (push) Waiting to run
Rust / simulator (push) Waiting to run
Rust / test-limbo (push) Waiting to run
Rust / test-sqlite (push) Waiting to run
Rust Benchmarks+Nyrkiö / tpc-h (push) Waiting to run
Rust Benchmarks+Nyrkiö / vfs-bench-compile (push) Waiting to run
Rust Benchmarks+Nyrkiö / bench (push) Waiting to run
Rust Benchmarks+Nyrkiö / clickbench (push) Waiting to run
Rust Benchmarks+Nyrkiö / tpc-h-criterion (push) Waiting to run
## Description
BTreeCursor sets null flag to false once `seek` is called. This PR does
the same for MVCC
## Motivation and context
join.test failed with some cases due to this bug
## Description of AI Usage
I asked AI to find the issue but I ended showing the agent why he did
things wrong and that he should be ashamed

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #4296
2025-12-22 09:15:24 +02:00
Pekka Enberg
6fcf66be3d
Merge 'pyturso: fix panic' from Nikita Sivukhin
`resume` was not re-entrant and panicked in case when was called after
operation completion.
```
thread '<unnamed>' panicked at sync/sdk-kit/src/turso_async_operation.rs:59:20:
`async fn` resumed after completion
...
```
This PR makes `resume` method for `PyTursoAsyncOperation` re-entrant
(like in the c-api) and also convert exception type in the sync module

Closes #4315
2025-12-22 09:09:37 +02:00
Pekka Enberg
d3714f4120
Merge 'Mark triggers as experimental' from Jussi Saurio
Closes #4022

Closes #4318
2025-12-22 09:08:54 +02:00
Preston Thorpe
d6ceda8dcb
Merge 'fix(core/translate): apply affinity conversion to hash join build and probe keys' from Nuno Gonçalves
## Description
This PR adds missing affinity conversion to hash joins by applying
affinity conversion to build and probe keys before hashing.
```
turso> CREATE TABLE x(a INTEGER);
turso> CREATE TABLE y(b TEXT);
turso> INSERT INTO x VALUES (2),(3);
turso> INSERT INTO y VALUES ('02'),('2'),('2.0'),('3x'),('3.5');
turso> SELECT a, b
  FROM x JOIN y ON a = b
  ORDER BY a, b;
┌───┬─────┐
│ a │ b   │
├───┼─────┤
│ 2 │ 02  │
├───┼─────┤
│ 2 │ 2   │
├───┼─────┤
│ 2 │ 2.0 │
└───┴─────┘
```
## Motivation and context
Fixes #3482.
Currently, Turso returns an empty result set:
```
turso> CREATE TABLE x(a INTEGER);
turso> CREATE TABLE y(b TEXT);
turso> INSERT INTO x VALUES (2),(3);
turso> INSERT INTO y VALUES ('02'),('2'),('2.0'),('3x'),('3.5');
turso> SELECT a, b
  FROM x JOIN y ON a = b
  ORDER BY a, b;
turso>
```
Expected behavior:
```
sqlite> CREATE TABLE x(a INTEGER);
sqlite> CREATE TABLE y(b TEXT);
sqlite> INSERT INTO x VALUES (2),(3);
sqlite> INSERT INTO y VALUES ('02'),('2'),('2.0'),('3x'),('3.5');
sqlite> SELECT a, b
   ...>   FROM x JOIN y ON a = b
   ...>   ORDER BY a, b;
2|02
2|2
2|2.0
```
## Description of AI Usage
This PR was developed with assistance from Claude Sonnet 4.5 through
code completions.

Reviewed-by: Preston Thorpe <preston@turso.tech>

Closes #4317
2025-12-21 21:08:29 -05:00