mirrors/limbo
1
0
Fork 0
mirror of https://github.com/tursodatabase/limbo.git synced 2025-07-15 16:35:00 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 16
5298 commits 55 branches 54 tags 63 MiB
Commit graph

151 commits

Author SHA1 Message Date
Piotr Sarna
ddbcd9be79 database: bring back dropping unused row versions 2023-06-06 12:47:40 +02:00
Piotr Sarna
625394000e unignore test_dirty_read_deleted 2023-06-06 11:34:34 +02:00
Piotr Sarna
47eb149214 database: drop the mutex 
Without a critical section, we naturally hit a few unimplemented
paths when handling concurrent transactions, which is great news!
Visiting previously impossible paths already proves that lock-free
is able to handle concurrency > 1.
Now, the easy part - fixing all the unimplemented paths and making
the Hekaton implementation 100% foolproof.
 2023-06-06 11:28:27 +02:00
Piotr Sarna
a8faffa9f5 database: migrate txs to SkipMap 2023-06-06 09:43:00 +02:00
Piotr Sarna
fdfc4fd5b4 database: drop RefCell from SkipMap 
not needed, the structure is already Send&Sync
 2023-06-05 13:58:24 +02:00
Piotr Sarna
74a7628a0a mvcc-rs: move database tests to a separate file 
That makes the file more human-readable
 2023-06-05 12:50:35 +02:00
Piotr Sarna
77e88d3f04 fix clippy 2023-06-05 11:47:53 +02:00
Pekka Enberg
51f33919d3 Switch to concurrent SkipMap for row versions (#49) 
Let's switch to concurrent SkipMap as the first small step towards
lockless index...
 2023-05-17 15:31:16 +02:00
Piotr Sarna
ae4cc872b6 treewide: overhaul the API to be sync again 
We dropped all occurrences of Tokio to avoid the cost of allocations
induced by async runtimes.
The only async part of the code is now S3 storage, which is just
wrapped in a futures::executor::block_on()
 2023-05-17 12:39:08 +02:00
Piotr Sarna
a28c41f919 bindings: run recovery on MVCCDatabaseOpen 2023-05-16 14:55:16 +02:00
Piotr Sarna
34d21b0eb7 storage: add S3 storage 
It's now possible to run with S3 persistent storage
if bindings are built with `s3_storage` feature.
Regular S3 credentials need to be set up locally,
and all customary env variables like AWS_SECRET_ACCESS_KEY
and AWS_ACCESS_KEY_ID work just fine.

For local development, one can set MVCCRS_ENDPOINT env variable.
For testing with MinIO, the following setup customarily works:
MVCCRS_ENDPOINT=http://localhost:9000 \
  AWS_SECRET_ACCESS_KEY=minioadmin \
  AWS_ACCESS_KEY_ID=minioadmin \
    ./libsql /tmp/testme.db
 2023-05-16 14:36:58 +02:00
Piotr Sarna
5da87739fa bindings: split transcation begin from insert/read 
libSQL expects to be able to begin/commit a transaction
independently of reading or inserting data.
 2023-05-15 14:33:25 +02:00
Piotr Sarna
8b1ef20c08 treewide: drop storage trait 
We're good with an enum, and async_trait has a runtime cost
we don't like.
 2023-05-15 10:50:47 +02:00
Pekka Enberg
7772c65f8d Switch to Tokio's mutex (#40) 
The async trait wrapper is very expensive because it's doing pin boxing
in the hot path. Switch to Tokio's mutex to get back performance.

Fixes #39
 2023-05-12 19:25:08 +02:00
Piotr Sarna
3236421f77 mvcc: switch to (table_id, row_id) for row ids 
With that, we're able to easily distinguish rows from different
tables.
 2023-05-12 12:15:19 +02:00
Piotr Sarna
a782ae5a0a cursor: add is_empty 2023-05-12 10:09:18 +02:00
Piotr Sarna
d5eec5d528 cursor: add MVCCScanCursorPosition 2023-05-11 14:11:10 +02:00
Piotr Sarna
582bf14934 database: keep rows in BTreeMap 
We need ordering libSQL-side.
 2023-05-11 13:38:38 +02:00
Piotr Sarna
e8bdfc8e7a cursor, read: update pointers to *mut u8 2023-05-11 13:35:19 +02:00
Piotr Sarna
54ee330912 cursor: add closing cursor 
... which also terminates the read transaction open for scanning.
 2023-05-11 12:13:27 +02:00
Piotr Sarna
5bdcfc9924 cursor: handle current() graciously when there's no data 2023-05-11 11:08:43 +02:00
Piotr Sarna
ef097362ff mvcc, bindings: expose a scan cursor 2023-05-10 15:48:46 +02:00
Pekka Enberg
57b5919637 Merge pull request #35 from penberg/recovery 
Initial support for recovery
 2023-05-10 13:28:16 +03:00
Pekka Enberg
0cf7b787fd Merge pull request #36 from penberg/readimpl 
bindings: expose reading from the database
 2023-05-10 13:22:57 +03:00
Piotr Sarna
d047a24a32 bindings: expose reading from the database 
The results are returned as a CString for now.
 2023-05-10 12:09:06 +02:00
Pekka Enberg
08adf2118c Rename mutations.png to transactions.png 2023-05-10 13:01:33 +03:00
Pekka Enberg
ec336d5578 Initial support for recovery 
We have the transaction log in persistent storage so let's implement
recovery by replaying that.
 2023-05-10 12:56:19 +03:00
Pekka Enberg
8c56b381c0 Rename mutations to log records 
The Hekaton paper talks about "log records" so let's just run with that
terminology to avoid confusion.
 2023-05-09 21:32:36 +03:00
Pekka Enberg
cf31de5d41 Fix MVCCDatabaseInsert() type signature 
Values are opaque blobs so use "const void *" for C callers.
 2023-05-09 16:14:28 +03:00
Pekka Enberg
3d0c8a415e Add stdint.h include to mvcc.h 2023-05-09 13:49:31 +03:00
Piotr Sarna
d4da54b10b mvcc: build static library for C bindings 
That's how it's currently consumed by libSQL.
 2023-05-09 10:28:26 +02:00
Pekka Enberg
f47eea1a72 Move DESIGN.md to docs 2023-05-09 11:00:00 +03:00
Pekka Enberg
b9575c4375 Rename database directory to mvcc-rs 2023-05-09 10:56:46 +03:00
Pekka Enberg
41bed41544 Improve logging 2023-05-09 10:51:14 +03:00
Pekka Enberg
5ce2bc41f9 cargo fmt 2023-05-09 10:47:43 +03:00
Pekka Enberg
b2f46e156b Improve C binding error reporting 2023-05-09 10:47:40 +03:00
Pekka Enberg
3ecb0fb2a9 Mark MVCCDatabaseOpen() as unsafe 2023-05-09 10:44:06 +03:00
Pekka Enberg
db1e313aca Silence clippy 2023-05-09 10:43:42 +03:00
Pekka Enberg
1fc99181f1 Update README.md 2023-05-09 10:42:05 +03:00
Pekka Enberg
c94561a646 Add mvcc.h to the tree 2023-05-09 10:41:33 +03:00
Pekka Enberg
779ad3066a Add error codes to C bindings 2023-05-09 10:41:13 +03:00
Pekka Enberg
34a4f1a269 Improve generated C bindings 
Before:

```c
typedef LocalClock Clock;

typedef JsonOnDisk Storage;

typedef DatabaseInner<Clock, Storage> Inner;

typedef Database<Clock, Storage, Mutex<Inner>> Db;

typedef struct {
  Db db;
  Runtime runtime;
} DbContext;

extern "C" {

DbContext *mvccrs_new_database(const char *path);

void mvccrs_free_database(Db *db);

int32_t mvccrs_insert(DbContext *db, uint64_t id, const uint8_t *value_ptr, uintptr_t value_len);

} // extern "C"

```

After:

```c

typedef struct DbContext DbContext;

typedef const DbContext *MVCCDatabaseRef;

extern "C" {

MVCCDatabaseRef MVCCDatabaseOpen(const char *path);

void MVCCDatabaseClose(MVCCDatabaseRef db);

int32_t MVCCDatabaseInsert(MVCCDatabaseRef db, uint64_t id, const uint8_t *value_ptr, uintptr_t value_len);

} // extern "C"

```
 2023-05-09 10:34:11 +03:00
Pekka Enberg
124446f17c Generate C header file on cargo build 2023-05-09 10:34:11 +03:00
Pekka Enberg
d5b96d5edf Move C bindings to separate crate 2023-05-09 10:34:11 +03:00
Piotr Sarna
488201cab2 add c_bindings feature for exposing functions to C 
The library can now be compiled to a static or dynamic
native lib, with very basic functionality exposed.
 2023-05-08 14:20:34 +02:00
Pekka Enberg
aac835fce9 Document persistent storage design (#24) 2023-05-06 20:15:08 +02:00
Piotr Sarna
fb6ce70993 database: add dropping unused row versions 
When a row version is not visible by any transactions,
active or future ones, it should be dropped.
 2023-04-20 15:47:36 +02:00
Piotr Sarna
2a018ea9a3 fixup: move DatabaseError under a feature 2023-04-20 15:47:36 +02:00
Piotr Sarna
b6b36a0d94 fixup: implement Stream for JsonOnDiskStream under a feature 2023-04-20 14:46:14 +02:00
Piotr Sarna
04a78f73fb treewide: add persistent storage trait 
This draft adds a persistent storage trait that can be used
to store transaction logs and read the log for recovery purposes.
Work in heavy progress, because ideally the design should also
allow reading versions from the storage, so that data can be
spilled from memory to disk if there's not enough RAM available.
 2023-04-17 14:46:39 +02:00