### The problem:
Sqlite displays the column names of the underlying vtab module when
displaying the `.schema`
Previously limbo omitted this, which makes it difficult for the user to
see what/how many columns the module's table has.
This matches sqlite's behavior by fetching the module's schema when the
schema entry is being inserted in translation.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#1168
we had an incorrect optimization in `eliminate_orderby_like_groupby()`
where it could remove e.g. the first term of the ORDER BY if it matched
the first GROUP BY term and the result set was naturally ordered by that
term. this is invalid. see e.g.:
```sql
main branch - BAD: removes the `ORDER BY id` term because the results are naturally ordered by id.
However, this results in sorting the entire thing by last name only!
limbo> select id, last_name, count(1) from users GROUP BY 1,2 order by id, last_name desc limit 3;
┌──────┬───────────┬───────────┐
│ id │ last_name │ count (1) │
├──────┼───────────┼───────────┤
│ 6235 │ Zuniga │ 1 │
├──────┼───────────┼───────────┤
│ 8043 │ Zuniga │ 1 │
├──────┼───────────┼───────────┤
│ 944 │ Zimmerman │ 1 │
└──────┴───────────┴───────────┘
after fix - GOOD:
limbo> select id, last_name, count(1) from users GROUP BY 1,2 order by id, last_name desc limit 3;
┌────┬───────────┬───────────┐
│ id │ last_name │ count (1) │
├────┼───────────┼───────────┤
│ 1 │ Foster │ 1 │
├────┼───────────┼───────────┤
│ 2 │ Salazar │ 1 │
├────┼───────────┼───────────┤
│ 3 │ Perry │ 1 │
└────┴───────────┴───────────┘
I also refactored sorters to always use the ast `SortOrder` instead of boolean vectors, and use the `compare_immutable()` utility we use inside btrees too.
Closes#1365
The previous version of `julian_day-converter` had precision issues,
potentially causing loss of precision when converting between
`julianday` and `datetime`
Reviewed-by: Diego Reis (@diegoreis42)
Closes#1344
In left joins, even if the join condition is not matched, the system
must emit a row for every row of the outer table:
-- this must return t1.count() rows, with NULLs for all columns of t2
SELECT * FROM t1 LEFT JOIN t2 ON FALSE;
Our logic for clearing the null flag was to do it in Next/Prev. However,
this is problematic for a few reasons:
- If the inner table of the left join is using SeekRowid, then Next/Prev
is never called on its cursor, so the null flag doesn't get cleared.
- If the inner table of the left join is using a non-covering index seek,
i.e. it iterates its rows using an index, but seeks to the main table
to fetch data, then Next/Prev is never called on the main table, and the
main table's null flag doesn't get cleared.
What this results in is NULL values incorrectly being emitted for the
inner table after the first correct NULL row, since the null flag is
correctly set to true, but never cleared.
This PR fixes the issue by clearing the null flag whenever seek() is
invoked on the cursor. Hence, the null flag is now cleared on:
- next()
- prev()
- seek()
First attempt at closing #1212. Also with this PR, I added the option of
using `with` syntax for `TestLimboShell`. It automatically closes the
shell on error, and facilitates error handling overall. If this is
merged, I can update the other python tests to use `with` as well.
Reviewed-by: Preston Thorpe (@PThorpe92)
Closes#1230
When the python tests fail, they will sometimes truncate the output if
it is smaller than the `PIPE_BUFF` size. With this fix we can now
properly print the backtrace, when the program panics.
# Before
This is the problematic CI output from #1331 that led me to fix this.
In this case, it was already truncating the output of the `assert`
prints.
```
./testing/cli_tests/extensions.py
Extension ./target/debug/liblimbo_regexp loaded successfully.
Testing: uuid functions are registered properly with ext loaded
Testing: scalar alias's are registered properly
Testing: median agg function returns null when ext not loaded
Testing: median agg function works
Testing: median agg function works with odd number of elements
Testing: test aggregate percentile function with 2 arguments works
Testing: test aggregate percentile function with 1 argument works
Testing: crypto_blake3 returns null when ext not loaded
Testing: blake3 should encrypt correctly
Testing: md5 should encrypt correctly
Testing: sha1 should encrypt correctly
Testing: sha256 should encrypt correctly
Testing: sha384 should encrypt correctly
Testing: sha512 should encrypt correctly
Testing: base32 should encode correctly
Testing: base32 should decode correctly
Testing: base64 should encode correctly
Testing: base64 should decode correctly
Testing: base85 should encode correctly
Testing: base85 should decode correctly
Testing: hex should encode correctly
Testing: hex should decode correctly
Testing: url should encode correctly
Testing: url should decode correctly
Testing: ipfamily function returns null when ext not loaded
Testing: ipfamily function returns 4 for IPv4
Testing: ipfamily function returns 6 for IPv6
Testing: ipcontains function returns 1 for IPv4
Testing: ipcontains function returns 0 for IPv4
Testing: iphost function returns the host for IPv4
Testing: iphost function returns the host for IPv6
Testing: ipmasklen function returns the mask length for IPv4
Testing: ipmasklen function returns the mask length for IPv6
Testing: ipnetwork function returns the flattened CIDR for IPv4
Testing: ipnetwork function returns the network for IPv6
Testing: testvfs not loaded
Testing: testvfs extension loaded
thread 'main' panicked at core/storage/pager.rs:61:38:
called `Option::unwrap()` on a `None` value
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Test FAILED: Error encountered in Limbo shell.
make: *** [Makefile:70: test-extensions] Error 1
```
# After
```
with-env {RUST_BACKTRACE:1} {make test-extensions}
cargo build
Compiling limbo_regexp v0.0.19-pre.4 (/Users/pedro/Projects/limbo/extensions/regexp)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.51s
cargo build --package limbo_regexp
Compiling limbo_regexp v0.0.19-pre.4 (/Users/pedro/Projects/limbo/extensions/regexp)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.19s
./testing/cli_tests/extensions.py
Extension ./target/debug/liblimbo_regexp loaded successfully.
Testing: uuid functions are registered properly with ext loaded
Testing: scalar alias's are registered properly
Testing: median agg function returns null when ext not loaded
Testing: median agg function works
Testing: median agg function works with odd number of elements
Testing: test aggregate percentile function with 2 arguments works
Testing: test aggregate percentile function with 1 argument works
Testing: crypto_blake3 returns null when ext not loaded
Testing: blake3 should encrypt correctly
Testing: md5 should encrypt correctly
Testing: sha1 should encrypt correctly
Testing: sha256 should encrypt correctly
Testing: sha384 should encrypt correctly
Testing: sha512 should encrypt correctly
Testing: base32 should encode correctly
Testing: base32 should decode correctly
Testing: base64 should encode correctly
Testing: base64 should decode correctly
Testing: base85 should encode correctly
Testing: base85 should decode correctly
Testing: hex should encode correctly
Testing: hex should decode correctly
Testing: url should encode correctly
Testing: url should decode correctly
Testing: ipfamily function returns null when ext not loaded
Testing: ipfamily function returns 4 for IPv4
Testing: ipfamily function returns 6 for IPv6
Testing: ipcontains function returns 1 for IPv4
Testing: ipcontains function returns 0 for IPv4
Testing: iphost function returns the host for IPv4
Testing: iphost function returns the host for IPv6
Testing: ipmasklen function returns the mask length for IPv4
Testing: ipmasklen function returns the mask length for IPv6
Testing: ipnetwork function returns the flattened CIDR for IPv4
Testing: ipnetwork function returns the network for IPv6
Testing: testvfs not loaded
Testing: testvfs extension loaded
thread 'main' panicked at core/storage/pager.rs:61:38:
called `Option::unwrap()` on a `None` value
stack backtrace:
0: rust_begin_unwind
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:665:5
1: core::panicking::panic_fmt
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/panicking.rs:74:14
2: core::panicking::panic
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/panicking.rs:148:5
3: core::option::unwrap_failed
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/option.rs:2012:5
4: core::option::Option<T>::unwrap
at /Users/pedro/.rustup/toolchains/1.83.0-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/option.rs:972:21
5: limbo_core::storage:📟:Page::get_contents
at ./core/storage/pager.rs:61:9
6: limbo_core::storage::btree::BTreeCursor::balance_non_root
at ./core/storage/btree.rs:1723:40
7: limbo_core::storage::btree::BTreeCursor::balance
at ./core/storage/btree.rs:1570:35
8: limbo_core::storage::btree::BTreeCursor::insert_into_page
at ./core/storage/btree.rs:1512:35
9: limbo_core::storage::btree::BTreeCursor::insert
at ./core/storage/btree.rs:3024:31
10: limbo_core::vdbe::execute::op_insert
at ./core/vdbe/execute.rs:3654:23
11: limbo_core::vdbe::Program::step
at ./core/vdbe/mod.rs:379:23
12: limbo_core::Statement::step
at ./core/lib.rs:582:9
13: limbo::app::Limbo::print_query_result
at ./cli/app.rs:657:27
14: limbo::app::Limbo::run_query
at ./cli/app.rs:420:20
15: limbo::app::Limbo::handle_input_line
at ./cli/app.rs:527:13
16: limbo::main
at ./cli/main.rs:29:31
17: core::ops::function::FnOnce::call_once
at /Users/pedro/.rustup/toolchains/1.83.0-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Testing: Tested large write to testfs
Test FAILED: Test failed
SQL: SELECT count(*) FROM test;
Actual:
None
make: *** [test-extensions] Error 1
```
Closes#1346
Sqlite reference implementation: https://github.com/sqlite/sqlite/blob/8
37dc09bce7de8971c7488b70cf5da93c60fbed0/src/vdbe.c#L2558
We did not support blobs before and our ordering of the match statements
were incorrect when one of the arguments was NULL
Closes#1337
Fixes#1298
- Fixes Limbo trying to use an index using a WHERE clause constraint
that refers to the same table on both sides, e.g. `WHERE t.x = t.x`
- Fixes not using indexes when the relevant expression is paren wrapped,
e.g.
- `SELECT * FROM t WHERE (indexcol) > 5`
- `SELECT * FROM t WHERE (indexcol > 5)`
- Changes existing table logical expr fuzz test to have primary keys
(which helped me find both issues above)
Closes#1300
