Commit graph

71 commits

Author SHA1 Message Date
Piotr Rzysko
4d35e36b77 Introduce virtual table types 2025-06-01 07:45:57 +02:00
meteorgan
2f82762ca2 add function parse_signed_number 2025-05-28 00:33:41 +08:00
Jussi Saurio
7c07c09300 Add stable internal_id property to TableReference
Currently our "table id"/"table no"/"table idx" references always
use the direct index of the `TableReference` in the plan, e.g. in
`SelectPlan::table_references`. For example:

```rust
Expr::Column { table: 0, column: 3, .. }
```

refers to the 0'th table in the `table_references` list.

This is a fragile approach because it assumes the table_references
list is stable for the lifetime of the query processing. This has so
far been the case, but there exist certain query transformations,
e.g. subquery unnesting, that may fold new table references from
a subquery (which has its own table ref list) into the table reference
list of the parent.

If such a transformation is made, then potentially all of the Expr::Column
references to tables will become invalid. Consider this example:

```sql
-- Assume tables: users(id, age), orders(user_id, amount)

-- Get total amount spent per user on orders over $100
SELECT u.id, sub.total
FROM users u JOIN
     (SELECT user_id, SUM(amount) as total
      FROM orders o
      WHERE o.amount > 100
      GROUP BY o.user_id) sub
WHERE u.id = sub.user_id

-- Before subquery unnesting:
-- Main query table_references: [users, sub]
-- u.id refers to table 0, column 0
-- sub.total refers to table 1, column 1
--
-- Subquery table_references: [orders]
-- o.user_id refers to table 0, column 0
-- o.amount refers to table 0, column 1
--
-- After unnesting and folding subquery tables into main query,
-- the query might look like this:

SELECT u.id, SUM(o.amount) as total
FROM users u JOIN orders o ON u.id = o.user_id
WHERE o.amount > 100
GROUP BY u.id;

-- Main query table_references: [users, orders]
-- u.id refers to table index 0 (correct)
-- o.amount refers to table index 0 (incorrect, should be 1)
-- o.user_id refers to table index 0 (incorrect, should be 1)
```

We could ofc traverse every expression in the subquery and rewrite
the table indexes to be correct, but if we instead use stable identifiers
for each table reference, then all the column references will continue
to be correct.

Hence, this PR introduces a `TableInternalId` used in `TableReference`
as well as `Expr::Column` and `Expr::Rowid` so that this kind of query
transformations can happen with less pain.
2025-05-25 20:26:17 +03:00
Jussi Saurio
c18c6a00fa refactor: use walk_expr() in resolving vtab constraints 2025-05-23 16:28:56 +03:00
Piotr Rzysko
6b454ea36f Normalize column names when creating virtual tables
Ensures consistent handling of column names between virtual and regular
tables and allows the use of quoted column names.
2025-05-21 08:30:26 +02:00
pedrocarlo
f8854f180a Added collation to create table columns 2025-05-19 15:22:14 -03:00
Pekka Enberg
e3f71259d8 Rename OwnedValue -> Value
We have not had enough merge conflicts for a while so let's do a
tree-wide rename.
2025-05-15 09:59:46 +03:00
pedrocarlo
3526a206e4 support Unique properly by creating a vec of auto indices 2025-05-14 11:34:39 -03:00
pedrocarlo
e4ca1bb55e modify automatic index creation to account for unique columns 2025-05-14 11:34:11 -03:00
pedrocarlo
bb158a5433 add unique field to Column 2025-05-14 11:34:11 -03:00
meteorgan
261adb5ed7 fix cargo fmt 2025-05-08 22:26:50 +08:00
meteorgan
a1f981a973 handle int64 overflow by f64 2025-05-08 22:22:55 +08:00
meteorgan
ef3f004e30 refactor numeric literal 2025-05-08 18:37:17 +08:00
Jussi Saurio
bf2e198a57 Merge 'Fix out of bounds access on parse_numeric_str' from Levy A.
Fixes #1361.

Closes #1362
2025-04-18 15:24:37 +03:00
Levy A.
5fd2ed0bae fix: handle empty case 2025-04-17 20:20:57 -03:00
Levy A.
32d59b8c78 refactor+fix: using a more robust pattern matching approach 2025-04-17 20:08:05 -03:00
PThorpe92
a25a02efe1
Improve xBestIndex call site and allow for proper handling of join and where constraints 2025-04-17 14:01:45 -04:00
PThorpe92
e17fd7edc4
Add comments and address PR review 2025-04-17 14:01:44 -04:00
PThorpe92
7d271edf8a
Remove unused function in core/util.rs 2025-04-17 14:01:44 -04:00
PThorpe92
0f34a813ff
Add can_pushdown_predicate fn to evaluate ast expressions for constness 2025-04-17 13:53:28 -04:00
Jussi Saurio
198aedb042 Refactor: add 'pos_in_table' to IndexColumn for easier lookup 2025-04-15 14:47:49 +03:00
Diego Reis
ff7a4e8297 core: Change always falseness of equivalence between variables expressions to be only on anonymous variables
Named variables are compared by name
2025-04-12 18:04:15 -03:00
Diego Reis
73764e198e core: Fix equivalence between variable expressions to be always false
Since until the bind to a value they are treated as NULL. https://sqlite.org/lang_expr.html#varparam
2025-04-12 16:34:39 -03:00
PThorpe92
41ac91f14f
Add tests for parsing vtab creation sql in ParseSchema 2025-04-08 20:10:49 -04:00
PThorpe92
3a7f1e4056
Add comments explaining flow of reloading vtabs from schema tbl 2025-04-08 20:10:49 -04:00
PThorpe92
6b5ec1f07b
Remove mut borrow from sym table in parse schema fn 2025-04-08 20:10:49 -04:00
PThorpe92
c15035caf8
Add module and vtab to schema after table is reopened with proper ext 2025-04-08 20:10:48 -04:00
PThorpe92
4b9b6c969b
Parse schema rows after extensions are loaded 2025-04-08 20:10:47 -04:00
PThorpe92
3ad7d194cb
Prevent panic on loading non-existent vtab module 2025-04-08 20:09:27 -04:00
PThorpe92
ae2be30204
Move init label to proper place in create vtab translation 2025-04-03 20:22:14 -04:00
Diego Reis
f499f756fb core/util: Fix invalid numeric parsing
To see details: https://github.com/tursodatabase/limbo/issues/1157
2025-03-24 20:21:09 -03:00
Diego Reis
5dba4999a7 core/util: Add unit tests for parse_numeric_str and fix whitespace handling 2025-03-24 19:56:13 -03:00
Pekka Enberg
bf3163c7fe core: Fix parse_schema() to use existing MVCC TX 2025-03-06 10:16:42 +02:00
Pere Diaz Bou
8daf7666d1 Make database Sync + Send 2025-03-05 14:07:48 +01:00
Pekka Enberg
fe440b7b34 Merge 'Fix casting text to integer to match SQLite' from Preston Thorpe
```console
thread 'fuzz::tests::logical_expression_fuzz_run' panicked at tests\integration\fuzz\mod.rs:818:13:
assertion `left == right` failed: query: SELECT  ( ( 3622873 || -8851250 ) * ( ( ( -124 ) + ( -5792536 ) ) ) ) = ( 179434259456392 < 65481085924370 ), limbo: [[Integer(1)]], sqlite: [[Integer(0)]]
  left: [[Integer(1)]]
 right: [[Integer(0)]]
```
This and a few other failing fuzzing tests were due to incorrectly
parsing numerics from strings. Some of our casting was done properly,
but it wasn't being applied to all cases where the behavior was needed.
It was also attempting to parse a string[0..N] N times until
`string[0..N].parse()` would no longer succeed. This searches for the
index of the first illegal character and parses the resulting slice
once.
Tests were added for some of the edgecases that were previously failing.
This PR also adds a macro in vdbe/insn.rs that allows for a bit of
cleanup and reduces some matching.

Closes #1053
2025-02-25 15:44:37 +02:00
Pekka Enberg
7f2525ac27 Merge 'Implement create virtual table using vtab modules, more work on virtual tables' from Preston Thorpe
This PR started out as one to improve the API of extensions but I ended
up building on top of this quite a bit and it just kept going. Sorry
this one is so large but there wasn't really a good stopping point, as
it kept leaving stuff in broken states.
**VCreate**: Support for `CREATE VIRTUAL TABLE t USING vtab_module`
**VUpdate**: Support for `INSERT` and `DELETE` methods on virtual
tables.
Sqlite uses `xUpdate` function with the `VUpdate` opcode to handle all
insert/update/delete functionality in virtual tables..
have to just document that:
```
if args[0] == NULL:  INSERT args[1] the values in args[2..]

if args[1] == NULL: DELETE args[0]

if args[0] != NULL && len(args) > 2: Update values=args[2..]  rowid=args[0]
```
I know I asked @jussisaurio on discord about this already, but it just
sucked so bad that I added some internal translation so we could expose
a [nice API](https://github.com/tursodatabase/limbo/pull/996/files#diff-
3e8f8a660b11786745b48b528222d11671e9f19fa00a032a4eefb5412e8200d1R54) and
handle the logic ourselves while keeping with sqlite's opcodes.
I'll change it back if I have to, I just thought it was genuinely awful
to have to rely on comments to explain all that to extension authors.
The included extension is not meant to be a legitimately useful one, it
is there for testing purposes. I did something similar in #960 using a
test extension, so I figure when they are both merged, I will go back
and combine them into one since you can do many kinds at once, and that
way it will reduce the amount of crates and therefore compile time.
1. Remaining opcodes.
2. `UPDATE` (when we support the syntax)
3. `xConnect` - expose API for a DB connection to a vtab so it can
perform arbitrary queries.

Closes #996
2025-02-25 15:31:12 +02:00
PThorpe92
b31363aecb
More improvements/cleanups to vdbe around casting 2025-02-24 21:31:26 -05:00
PThorpe92
6d55cdba3b
Remove allocations from numeric text casting, cleanups 2025-02-24 12:30:38 -05:00
Pekka Enberg
eb6019b453 cargo fmt 2025-02-24 17:39:21 +02:00
Pekka Enberg
4cefb222db Merge 'Fix cast_text_to_number compatibility' from Pedro Muniz
Modified  `cast_text_to_number` to be more compatible with SQLite. When
I was running some fuzz tests, I would eventually get errors due to
incorrect casting of text to `INTEGER` or `REAL`. Previously in code
there were 2 implementations of `cast_text_to_number`: one in
`core/vdbe/insn.rs` and one in `core/vdbe/mod.rs`. I consolidated the
casting to only one function. Previously, the `mod.rs` function was just
calling `checked_cast_text_to_numeric`, which was used in `MustBeInt`
opcode.  Hopefully this fixes some of the CI testing issues we are
having. This was the query that prompted me to do this: `SELECT  ( ( (
878352367 ) <> ( 29 ) ) ) = ( ( ( -4309097 ) / ( -37 || -149680985265412
) ) - 755066415 );`

Closes #1038
2025-02-24 11:20:14 +02:00
pedrocarlo
2e38aa1d6b remove dbg 2025-02-20 16:09:39 -03:00
pedrocarlo
13639899a5 more adjustments to parser to handle edge cases 2025-02-20 16:05:50 -03:00
pedrocarlo
033d0116d6 rewrote parsing from text to integer and real 2025-02-20 02:16:30 -03:00
PThorpe92
e86f00cb81
Add normalizing windows paths to sqlite spec 2025-02-18 22:41:35 -05:00
PThorpe92
42a0c18574
Add parsing sqlite URI to prep for vfs 2025-02-18 21:02:48 -05:00
PThorpe92
e63436dc47
Fix sqlite_schema and remove explicit vtables 2025-02-17 20:44:45 -05:00
PThorpe92
8b5772fe1c
Implement VUpdate (insert/delete for virtual tables 2025-02-17 20:44:44 -05:00
PThorpe92
9c8083231c
Implement create virtual table and VUpdate opcode 2025-02-17 20:44:44 -05:00
Pekka Enberg
ac54c35f92 Switch to workspace dependencies
...makes it easier to specify a version, which is needed for `cargo publish`.
2025-02-12 17:28:04 +02:00
Aarni Koskela
eaea02c567 Fix a handful of typos 2025-02-09 18:08:29 +02:00