Internal improvements to Parser::next_token/prev_token
This reduces the number of helper functions used by next_token()/prev_token() while slightly improving performance and reducing the chances of coding errors when using prev_token() after hitting end-of-file.
Before this `next_token()` would only increment the index when returning
`Some(token)`. This means that the caller wishing to rewind must be
careful not to call `prev_token()` on EOF (`None`), while not forgetting
to call it for `Some`. Not doing this resulted in bugs in the
undertested code that does error handling.
After making this mistake several times, I'm changing `next_token()` /
`prev_token()` so that calling `next_token(); prev_token(); next_token()`
returns the same token in the first and the last invocation.
- Avoid cloning whitespace tokens in `peek_nth_token()` by using a
&Token from `tokens.get()` instead of a cloned `Token` from `token_at()`
- Similarly avoid cloning in `next_token_no_skip`, and clone the
non-whitespace tokens in `next_token` instead.
- Remove `token_at`, which was only used in `peek_token` and
`peek_nth_token`
- Fold `prev_token_no_skip()` into `prev_token()` and make `prev_token`
return nothing, as the return value isn't used anyway.
* Rewrite parsing of `ALTER TABLE ADD CONSTRAINT`
* Support constraints in CREATE TABLE
* Change `Value::Long()` to be unsigned, use u64 consistently
* Allow trailing comma in CREATE TABLE
The tokenizer emits a separate Token for +/- signs, so the value of
Value::Long() (as well as of parse_literal_int()) may never be negative.
Also we have been using both u64 and usize to represent a parsed
unsigned number. Change to using u64 for consistency.
NOT LIKE has the same precedence as the LIKE operator. The parser was
previously assigning it the precedence of the unary NOT operator. NOT
BETWEEN and NOT IN are treated similarly, as they are equivalent, from a
precedence perspective, to NOT LIKE.
The fix for this requires associating precedences with sequences of
tokens, rather than single tokens, so that "NOT LIKE" and "NOT <expr>"
can have different preferences. Perhaps surprisingly, this change is not
very invasive.
An alternative I considered involved adjusting the tokenizer to lex
NOT, NOT LIKE, NOT BETWEEN, and NOT IN as separate tokens. This broke
symmetry in strange ways, though, as NotLike, NotBetween, and NotIn
gained dedicated tokens, while LIKE, BETWEEN, and IN remained as
stringly identifiers.
Fixes#81.
`BETWEEN <thing> AND <thing>` allows <thing> to be any expr that doesn't
contain boolean operators. (Allowing boolean operators would wreak
havoc, because of the repurposing of AND as both a boolean operation
and part of the syntax of BETWEEN.)
Roughly the <character factor> production - https://jakewheat.github.io/sql-overview/sql-2011-foundation-grammar.html#character-factor
If an expression is followed by the keyword `COLLATE`, it must be
followed by the collation name, which is optionally schema-qualified
identifier.
The `COLLATE` keyword is not a regular binary operator in that it can't
be "nested": `foo COLLATE bar COLLATE baz` is not valid. If you prefer
to think of it as an operator, you might say it has the highest
precedence (judging from the spec), i.e. it binds to the smallest valid
expression to the left of it (so in `foo < bar COLLATE c`, the COLLATE
is applied first).
