Commit graph

661 commits

Author SHA1 Message Date
Pere Diaz Bou
cb4efdbc19 Merge 'LimboRwLock write and read lock fixes' from Pere Diaz Bou
Some checks are pending
Python / linux (x86_64) (push) Waiting to run
Python / macos-x86_64 (x86_64) (push) Waiting to run
Python / macos-arm64 (aarch64) (push) Waiting to run
Python / sdist (push) Waiting to run
Python / Release (push) Blocked by required conditions
Rust / cargo-fmt-check (push) Waiting to run
Rust / build-native (blacksmith-4vcpu-ubuntu-2404) (push) Waiting to run
Rust / build-native (macos-latest) (push) Waiting to run
Rust / build-native (windows-latest) (push) Waiting to run
Rust / clippy (push) Waiting to run
Rust / build-wasm (push) Waiting to run
Rust / test-limbo (push) Waiting to run
Rust / test-sqlite (push) Waiting to run
JavaScript / stable - x86_64-apple-darwin - node@20 (push) Waiting to run
JavaScript / stable - x86_64-pc-windows-msvc - node@20 (push) Waiting to run
JavaScript / stable - x86_64-unknown-linux-gnu - node@20 (push) Waiting to run
JavaScript / Test bindings on x86_64-apple-darwin - node@18 (push) Blocked by required conditions
JavaScript / Test bindings on x86_64-apple-darwin - node@20 (push) Blocked by required conditions
JavaScript / Test bindings on Linux-x64-gnu - node@18 (push) Blocked by required conditions
JavaScript / Test bindings on Linux-x64-gnu - node@20 (push) Blocked by required conditions
JavaScript / Build universal macOS binary (push) Blocked by required conditions
JavaScript / Publish (push) Blocked by required conditions
Python / check-requirements (push) Waiting to run
Python / lint (push) Waiting to run
Python / configure-strategy (push) Waiting to run
Python / test (push) Blocked by required conditions
Rust Benchmarks+Nyrkiö / bench (push) Waiting to run
Rust Benchmarks+Nyrkiö / clickbench (push) Waiting to run
Rust Benchmarks+Nyrkiö / tpc-h-criterion (push) Waiting to run
Rust Benchmarks+Nyrkiö / tpc-h (push) Waiting to run
* `write` was returning `true` even though it shouldn't because it
should return `false` in case it was already acquired.
* `read` theoritically can increase `nread` after another thread calls
`unlock` between first lock load and increase of `nread`. So it looks
something like this:
                  1. THREAD  1: read()
                  2. THREAD 2: get lock = `SHARED_LOCK`
                  3. THREAD 1: unlock -> lock is now `NO_LOCK`
                  4. THREAD 2: increase nread, return true.
    This is obviously wrong because `nreads` will be ` > 0 ` but `lock`
is `NO_LOCK`

Closes #1676
2025-06-09 16:15:46 +02:00
Zaid Humayun
e994adfb40 Persisting database header and pointer map page to cache
This commit ensures that the metadata in the database header and the pointer map pages allocated are correctly persisted to the page cache. This was not being done earlier.
2025-06-06 23:14:25 +05:30
Zaid Humayun
e7d09edf09 fix clippy warnings 2025-06-06 23:14:25 +05:30
Zaid Humayun
1f5025541c addresses comment https://github.com/tursodatabase/limbo/pull/1600#discussion_r2115796655 by @jussisaurio
this commit ensures that ptrmap operations return a CursorResult so operation can be suspended & later retried
2025-06-06 23:14:25 +05:30
Zaid Humayun
8efbce5980 Fixes tests after asserting MIN_PAGE_SIZE
this commit fixes tests that broke after addding the MIN_PAGE_SIZE assertion
2025-06-06 23:14:25 +05:30
Zaid Humayun
6e87761a34 Addresses comment https://github.com/tursodatabase/limbo/pull/1600#discussion_r2115842804 by @jussisaurio
this commit adds an assert into ptrmap_page_cycle_length
2025-06-06 23:14:25 +05:30
Zaid Humayun
20e20b8a32 Larger compiler directive. Addresses https://github.com/tursodatabase/limbo/pull/1600#discussion_r2115820195
this commit places all pointer map functionality within a module called ptrmap guarded by a compiler directive
2025-06-06 23:14:25 +05:30
Zaid Humayun
33fc60232c removed unnecessary comment 2025-06-06 23:14:25 +05:30
Zaid Humayun
8e6fb1d28f addresses comments by @jussisaurio 2025-06-06 23:14:25 +05:30
Zaid Humayun
0b5338ec7e fixed off by one error 2025-06-06 23:14:25 +05:30
Zaid Humayun
51a1b01ed9 removed comment 2025-06-06 23:14:25 +05:30
Zaid Humayun
7c83086eed removed inferred types 2025-06-06 23:14:25 +05:30
Zaid Humayun
275548e991 adds comment 2025-06-06 23:14:25 +05:30
Zaid Humayun
b9ae5e7ce9 removed omit_autovacuum from flags and fixed tests 2025-06-06 23:14:25 +05:30
Zaid Humayun
4b9ab35012 fixed clippy warnings 2025-06-06 23:14:25 +05:30
Zaid Humayun
5827a33517 Beginnings of AUTOVACUUM
This commit introduces AUTOVACUUM to Limbo. It introduces the concept of ptrmap pages and also adds some additional instructions that are required to make AUTOVACUUM PRAGMA work
2025-06-06 23:14:22 +05:30
Pere Diaz Bou
914c1a4407 fix race condition with read lock in between unlock 2025-06-06 12:16:41 +02:00
Pere Diaz Bou
e67089b377 fix false double acquire on write lock 2025-06-06 12:16:41 +02:00
meteorgan
a242bac340 Fix: ensure PRAGMA cache_size changes persist only for current session 2025-06-05 16:55:41 +08:00
Jussi Saurio
2087393d22 Merge 'Write database header via normal pager route' from meteorgan
Closes: #1613

Closes #1634
2025-06-04 09:39:14 +03:00
Jussi Saurio
ad8c9a4c15 Merge 'Fix WAL frame checksum mismatch' from Diego Reis
Closes #1622
I did an A/B test between SQLite and Limbo and they can restart the db
from each other, indicating that there isn't something very wrong with
our file format. Turns out it was with our reset logic without
truncating the file. I assumed it's safe to don't reset if we're in
`PASSIVE` mode, given the
[docs](https://www.sqlite.org/c3ref/wal_checkpoint_v2.html) and [source
code](2bd9f69d40
f447e7b5c098/src/wal.c#L2193).
It also does some small clean ups and fixes.

Reviewed-by: Pere Diaz Bou <pere-altea@homail.com>

Closes #1647
2025-06-04 09:14:45 +03:00
Diego Reis
09f978b239 core: Tagging some comments as TODO 2025-06-03 15:09:16 -03:00
meteorgan
1554c54f2b restore comments 2025-06-03 22:06:08 +08:00
meteorgan
f2bf6251cd write database header via normal pager route 2025-06-03 22:06:08 +08:00
Diego Reis
cf038b045d core/wal: Only reset the WAL if the file is truncated 2025-06-02 23:16:30 -03:00
Diego Reis
16c81f471b core/ondisk: Stop reading WAL file if a frame mismatch salt value with header
The salts values in the WAL header are (re)generated in every checkpoint (but in PASSIVE mode), so if we find a frame with mismatch it means it's a leftover from a previous checkpoint.
2025-06-02 23:14:51 -03:00
Diego Reis
ec4eb52734 core/wal: Refactor open_shared for readability 2025-06-02 19:21:13 -03:00
pedrocarlo
9b5f5f6053 do not move_to if we are already inserting in correct place 2025-06-02 02:55:32 -03:00
pedrocarlo
c2942a5819 small fixes 2025-06-01 12:11:03 -03:00
pedrocarlo
39434fd20f return_io when restoring context 2025-06-01 03:07:16 -03:00
pedrocarlo
2ddbb7eeed adjust move_to and seek functions to make them truly reentrant + adding return_if_locked_maybe_load in some places so that we read loaded pages 2025-06-01 01:01:35 -03:00
pedrocarlo
d688cfd547 make find_cell and process_overflow_page reentrant 2025-05-31 23:31:59 -03:00
pedrocarlo
dae58be071 make move_to reentrant 2025-05-31 02:56:30 -03:00
pedrocarlo
33480540f1 make cursor seek reentrant 2025-05-30 13:25:46 -03:00
Pere Diaz Bou
da4190a23e Convert u64 rowid to i64
Rowids can be negative, therefore let's swap to i64
2025-05-30 13:07:31 +02:00
Pere Diaz Bou
dd15b7df7f remove dumb comment from pagecachekey 2025-05-29 14:12:16 +02:00
Pere Diaz Bou
93161e9fce remove lru size > 0 check on page cache fuzz 2025-05-29 14:12:16 +02:00
Pere Diaz Bou
37e834b092 remove unnecessary test 2025-05-29 14:12:16 +02:00
Pere Diaz Bou
44007075d9 remove frame_id from PageCacheKey
After reading sqlite a bit, it isn't needed because we have RWlock for
each table in the database file.
2025-05-29 14:12:16 +02:00
Jussi Saurio
85316d8419 Merge 'clear page cache on transaction failure' from Pere Diaz Bou
This is the first step towards rollback, since we still don't spill
pages with WAL, we can simply invalidate page cache in case of failure.

Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>

Closes #1599
2025-05-28 23:14:44 +03:00
Pere Diaz Bou
28bd24b7d4 clear page cache on transaction failure
This is the first step towards rollback, since we still don't spill
pages with WAL, we can simply invalidate page cache in case of failure.
2025-05-28 15:54:28 +02:00
Jussi Saurio
dad1e6293b Btree: fix cursor record state not being updated in insert_into_page()
overwrite_cell() requires that the cursor state is pointing to a valid
record, but this was not currently set properly.
2025-05-28 16:54:00 +03:00
Pekka Enberg
59d28eac93 core: Switch Completion "is_completed" to use Cell
Suggested by Jussi
2025-05-27 14:05:07 +03:00
Pekka Enberg
3250560eb8 sqlite3: Add libsql_wal_get_frame() API 2025-05-27 13:47:40 +03:00
Pekka Enberg
05df548b10 core/io: Add wait_for_completion() to I/O dispatcher 2025-05-27 13:47:40 +03:00
Pekka Enberg
eca9a5b703 core/io: Switch to Arc<Completion> 2025-05-27 11:28:49 +03:00
Pekka Enberg
21535018aa core: Don't pass page to begin_read_wal_frame()
Make `begin_read_wal_frame()` a bit more generic by not requiring a page
to be passed.
2025-05-27 10:02:36 +03:00
Jussi Saurio
3ba9f2ab97 Small cleanups to pager/wal/vdbe - mostly naming
- Instead of using a confusing CheckpointStatus for many different things,
  introduce the following statuses:
    * PagerCacheflushStatus - cacheflush can result in either:
      - the WAL being written to disk and fsynced
      - but also a checkpoint to the main BD file, and fsyncing the main DB file

      Reflect this in the type.
    * WalFsyncStatus - previously CheckpointStatus was also used for this, even
      though fsyncing the WAL doesn't checkpoint.
    * CheckpointStatus/CheckpointResult is now used only for actual checkpointing.

- Rename HaltState to CommitState (program.halt_state -> program.commit_state)
- Make WAL a non-optional property in Pager
  * This gets rid of a lot of if let Some(...) boilerplate
  * For ephemeral indexes, provide a DummyWAL implementation that does nothing.
- Rename program.halt() to program.commit_txn()
- Add some documentation comments to structs and functions
2025-05-26 10:37:34 +03:00
Jussi Saurio
b5ac095716 Fix off-by-one error in max_frame after WAL load 2025-05-25 19:34:51 +03:00
Jussi Saurio
385c0d8987 clippy stfu part 2: electric boogaloo 2025-05-25 10:32:23 +03:00