This PR adds support for table-valued functions for PRAGMAs (see the
[PRAGMA functions section](https://www.sqlite.org/pragma.html)).
Additionally, it introduces built-in table-valued functions. I
considered using extensions for this, but there are several reasons in
favor of a dedicated mechanism:
* It simplifies the use of internal functions, structs, etc. For
example, when implementing `json_each` and `json_tree`, direct access to
internals was necessary:
https://github.com/tursodatabase/limbo/pull/1088
* It avoids FFI overhead. [Benchmarks](https://github.com/piotrrzysko/li
mbo/blob/pragma_vtabs_bench/core/benches/pragma_benchmarks.rs) on my
hardware show that `pragma_table_info()` implemented as an extension is
2.5× slower than the built-in version.
Reviewed-by: Jussi Saurio <jussi.saurio@gmail.com>
Closes#1642
Fixes a runtime panic caused by failing to halt the virtual machine
after executing CREATE VIRTUAL TABLE IF NOT EXISTS.
Previously resulted in:
thread 'main' panicked at core/vdbe/mod.rs:408:52:
index out of bounds: the len is 3 but the index is 3
When the python tests fail, they will sometimes truncate the output if
it is smaller than the `PIPE_BUFF` size. With this fix we can now
properly print the backtrace, when the program panics.
# Before
This is the problematic CI output from #1331 that led me to fix this.
In this case, it was already truncating the output of the `assert`
prints.
```
./testing/cli_tests/extensions.py
Extension ./target/debug/liblimbo_regexp loaded successfully.
Testing: uuid functions are registered properly with ext loaded
Testing: scalar alias's are registered properly
Testing: median agg function returns null when ext not loaded
Testing: median agg function works
Testing: median agg function works with odd number of elements
Testing: test aggregate percentile function with 2 arguments works
Testing: test aggregate percentile function with 1 argument works
Testing: crypto_blake3 returns null when ext not loaded
Testing: blake3 should encrypt correctly
Testing: md5 should encrypt correctly
Testing: sha1 should encrypt correctly
Testing: sha256 should encrypt correctly
Testing: sha384 should encrypt correctly
Testing: sha512 should encrypt correctly
Testing: base32 should encode correctly
Testing: base32 should decode correctly
Testing: base64 should encode correctly
Testing: base64 should decode correctly
Testing: base85 should encode correctly
Testing: base85 should decode correctly
Testing: hex should encode correctly
Testing: hex should decode correctly
Testing: url should encode correctly
Testing: url should decode correctly
Testing: ipfamily function returns null when ext not loaded
Testing: ipfamily function returns 4 for IPv4
Testing: ipfamily function returns 6 for IPv6
Testing: ipcontains function returns 1 for IPv4
Testing: ipcontains function returns 0 for IPv4
Testing: iphost function returns the host for IPv4
Testing: iphost function returns the host for IPv6
Testing: ipmasklen function returns the mask length for IPv4
Testing: ipmasklen function returns the mask length for IPv6
Testing: ipnetwork function returns the flattened CIDR for IPv4
Testing: ipnetwork function returns the network for IPv6
Testing: testvfs not loaded
Testing: testvfs extension loaded
thread 'main' panicked at core/storage/pager.rs:61:38:
called `Option::unwrap()` on a `None` value
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Test FAILED: Error encountered in Limbo shell.
make: *** [Makefile:70: test-extensions] Error 1
```
# After
```
with-env {RUST_BACKTRACE:1} {make test-extensions}
cargo build
Compiling limbo_regexp v0.0.19-pre.4 (/Users/pedro/Projects/limbo/extensions/regexp)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.51s
cargo build --package limbo_regexp
Compiling limbo_regexp v0.0.19-pre.4 (/Users/pedro/Projects/limbo/extensions/regexp)
Finished `dev` profile [unoptimized + debuginfo] target(s) in 0.19s
./testing/cli_tests/extensions.py
Extension ./target/debug/liblimbo_regexp loaded successfully.
Testing: uuid functions are registered properly with ext loaded
Testing: scalar alias's are registered properly
Testing: median agg function returns null when ext not loaded
Testing: median agg function works
Testing: median agg function works with odd number of elements
Testing: test aggregate percentile function with 2 arguments works
Testing: test aggregate percentile function with 1 argument works
Testing: crypto_blake3 returns null when ext not loaded
Testing: blake3 should encrypt correctly
Testing: md5 should encrypt correctly
Testing: sha1 should encrypt correctly
Testing: sha256 should encrypt correctly
Testing: sha384 should encrypt correctly
Testing: sha512 should encrypt correctly
Testing: base32 should encode correctly
Testing: base32 should decode correctly
Testing: base64 should encode correctly
Testing: base64 should decode correctly
Testing: base85 should encode correctly
Testing: base85 should decode correctly
Testing: hex should encode correctly
Testing: hex should decode correctly
Testing: url should encode correctly
Testing: url should decode correctly
Testing: ipfamily function returns null when ext not loaded
Testing: ipfamily function returns 4 for IPv4
Testing: ipfamily function returns 6 for IPv6
Testing: ipcontains function returns 1 for IPv4
Testing: ipcontains function returns 0 for IPv4
Testing: iphost function returns the host for IPv4
Testing: iphost function returns the host for IPv6
Testing: ipmasklen function returns the mask length for IPv4
Testing: ipmasklen function returns the mask length for IPv6
Testing: ipnetwork function returns the flattened CIDR for IPv4
Testing: ipnetwork function returns the network for IPv6
Testing: testvfs not loaded
Testing: testvfs extension loaded
thread 'main' panicked at core/storage/pager.rs:61:38:
called `Option::unwrap()` on a `None` value
stack backtrace:
0: rust_begin_unwind
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/std/src/panicking.rs:665:5
1: core::panicking::panic_fmt
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/panicking.rs:74:14
2: core::panicking::panic
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/panicking.rs:148:5
3: core::option::unwrap_failed
at /rustc/90b35a6239c3d8bdabc530a6a0816f7ff89a0aaf/library/core/src/option.rs:2012:5
4: core::option::Option<T>::unwrap
at /Users/pedro/.rustup/toolchains/1.83.0-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/option.rs:972:21
5: limbo_core::storage:📟:Page::get_contents
at ./core/storage/pager.rs:61:9
6: limbo_core::storage::btree::BTreeCursor::balance_non_root
at ./core/storage/btree.rs:1723:40
7: limbo_core::storage::btree::BTreeCursor::balance
at ./core/storage/btree.rs:1570:35
8: limbo_core::storage::btree::BTreeCursor::insert_into_page
at ./core/storage/btree.rs:1512:35
9: limbo_core::storage::btree::BTreeCursor::insert
at ./core/storage/btree.rs:3024:31
10: limbo_core::vdbe::execute::op_insert
at ./core/vdbe/execute.rs:3654:23
11: limbo_core::vdbe::Program::step
at ./core/vdbe/mod.rs:379:23
12: limbo_core::Statement::step
at ./core/lib.rs:582:9
13: limbo::app::Limbo::print_query_result
at ./cli/app.rs:657:27
14: limbo::app::Limbo::run_query
at ./cli/app.rs:420:20
15: limbo::app::Limbo::handle_input_line
at ./cli/app.rs:527:13
16: limbo::main
at ./cli/main.rs:29:31
17: core::ops::function::FnOnce::call_once
at /Users/pedro/.rustup/toolchains/1.83.0-aarch64-apple-darwin/lib/rustlib/src/rust/library/core/src/ops/function.rs:250:5
note: Some details are omitted, run with `RUST_BACKTRACE=full` for a verbose backtrace.
Testing: Tested large write to testfs
Test FAILED: Test failed
SQL: SELECT count(*) FROM test;
Actual:
None
make: *** [test-extensions] Error 1
```
Closes#1346
My PR #972 always keeps breaking when there are changes in how Btree
works lol. I'm extracting the memory tests I created there here, so we
can track problems earlier.
Closes#1228
