The Ident type was previously an alias for a String. Turn it into a full
fledged struct, so that the parser can preserve the distinction between
identifier value and quote style already made by the tokenizer's Word
structure.
The SQL standard requires that numeric literals with a decimal point,
like 1.23, are represented exactly, up to some precision. That means
that parsing these literals into f64s is invalid, as it is impossible
to represent many decimal numbers exactly in binary floating point (for
example, 0.3).
This commit parses all numeric literals into a new `Value` variant
`Number(String)`, removing the old `Long(u64)` and `Double(f64)`
variants. This is slightly less convenient for downstream consumers, but
far more flexible, as numbers that do not fit into a u64 and f64 are now
representable.
The rationale here is the same as the last commit: since this crate
exclusively parses SQL, there's no need to restate that in every type
name. (The prefix seems to be an artifact of this crate's history as a
submodule of Datafusion, where it was useful to explicitly call out
which types were related to SQL parsing.)
This commit has the additional benefit of making all type names
consistent; over type we'd added some types which were not prefixed with
"SQL".
The ASTNode enum was confusingly named. In the past, the name made
sense, as the enum contained nearly all of the nodes in the AST, but
over time, pieces have been split into different structs, like
SQLStatement and SQLQuery. The ASTNode enum now contains only contains
expression nodes, so Expr is a better name.
Also rename the UnnamedExpression and ExpressionWithAlias variants
of SQLSelectItem to UnnamedExpr and ExprWithAlias, respectively, to
match the new shorthand for the word "expression".
Standardize the license header, removing the Grove Enterprise copyright
notice where it exists per #58. Also add a CI check to ensure that files
without license headers don't get merged.
Fix#58.
The `@@version` test is MS' dialect of SQL, it seems, so test it with
its own dialect.
Update the rules for identifiers in Postresql dialect per documentation,
while we're at it. The current identifier rules in Postgresql dialect
were introduced in this commit - as a copy of generic rules, it seems:
810cd8e6cf (diff-2808df0fba0aed85f9d35c167bd6a5f1L138)
1) Removed unused date/time parsing methods from `Parser`
I don't see how the token-based parsing code would ever be used: the
date/time literals are usually quoted, like `DATE 'yyyy-mm-dd'` or
simply `'YYYYMMDD'`, so the date will be a single token.
2) Removed unused date/time related variants from `Value` and the
dependency on `chrono`.
We don't support parsing date/time literals at the moment and when we
do I think we should store the exact String to let the consumer parse
it as they see fit.
3) Removed `parse_timestamps_example` and
`parse_timestamps_with_millis_example` tests. They parsed as
`number(2016) minus number(02) minus number(15) <END OF EXPRESSION>`
(leaving the time part unparsed) as it makes no sense to try parsing
a yyyy-mm-dd value as an SQL expression.
`parse_example_value` parses as compound identifier, which makes no
sense ("SARAH"."LEWISE@sakilacustomer"."org")
`parse_function_now` is unnecessary since we already test the parsing
of functions in `parse_scalar_function_in_projection`
...to match the name of the recently introduced `SQLObjectName` struct
and to avoid any reservations about using it with multi-part names of
objects other than tables (as in the `type_name` case).
(To store "A name of a table, view, custom type, etc., possibly
multi-part, i.e. db.schema.obj".)
Before this change
- some places used `String` for this (these are updated in this commit)
- while others (notably SQLStatement::SQLDelete::relation, which is
the reason for this series of commits) relied on
ASTNode::SQLCompoundIdentifier (which is also backed by a
Vec<SQLIdent>, but, as a variant of ASTNode enum, is not convenient
to use when you know you need that specific variant).
Before this commit there was a single `parse_expr(u8)` method, which
was called both
1) from within the expression parser (to parse subexpression consisting
of operators with higher priority than the current one), and
2) from the top-down parser both
a) to parse true expressions (such as an item of the SELECT list or
the condition after WHERE or after ON), and
b) to parse sequences which are not exactly "expressions".
This starts cleaning this up by renaming the `parse_expr(u8)` method to
`parse_subexpr()` and using it only for (1) - i.e. usually providing a
non-zero precedence parameter.
The non-intuitively called `parse()` method is renamed to `parse_expr()`,
which became available and is used for (2a).
While reviewing the existing callers of `parse_expr`, four points to
follow up on were identified (marked "TBD (#)" in the commit):
1) Do not lose parens (e.g. `(1+2)*3`) when roundtripping
String->AST->String by using SQLNested.
2) Incorrect precedence of the NOT unary
3) `parse_table_factor` accepts any expression where a SELECT subquery
is expected.
4) parse_delete uses parse_expr() to retrieve a table name
These are dealt with in the commits to follow.
Parser::parse_sql() can now parse a semicolon-separated list of
statements, returning them in a Vec<SQLStatement>.
To support this we:
- Move handling of inter-statement tokens from the end of individual
statement parsers (`parse_select` and `parse_delete`; this was not
implemented for other top-level statements) to the common
statement-list parsing code (`parse_sql`);
- Change the "Unexpected token at end of ..." error, which didn't have
tests and prevented us from parsing successive statements ->
"Expected end of statement" (i.e. a delimiter - currently only ";" -
or the EOF);
- Add PartialEq on ParserError to be able to assert_eq!() that parsing
statements that do not terminate properly returns an expected error.
Continuing from https://github.com/andygrove/sqlparser-rs/pull/33#issuecomment-453060427
This stops the parser from accepting (and the AST from being able to
represent) SQL look-alike code that makes no sense, e.g.
SELECT ... FROM (CREATE TABLE ...) foo
SELECT ... FROM (1+CAST(...)) foo
Generally this makes the AST less "partially typed": meaning certain
parts are strongly typed (e.g. SELECT can only contain projections,
relations, etc.), while everything that didn't get its own type is
dumped into ASTNode, effectively untyped. After a few more fixes (yet
to be implemented), `ASTNode` could become an `SQLExpression`. The
Pratt-style expression parser (returning an SQLExpression) would be
invoked from the top-down parser in places where a generic expression
is expected (e.g. after SELECT <...>, WHERE <...>, etc.), while things
like select's `projection` and `relation` could be more appropriately
(narrowly) typed.
Since the diff is quite large due to necessarily large number of
mechanical changes, here's an overview:
1) Interface changes:
- A new AST enum - `SQLStatement` - is split out of ASTNode:
- The variants of the ASTNode enum, which _only_ make sense as a top
level statement (INSERT, UPDATE, DELETE, CREATE, ALTER, COPY) are
_moved_ to the new enum, with no other changes.
- SQLSelect is _duplicated_: now available both as a variant in
SQLStatement::SQLSelect (top-level SELECT) and ASTNode:: (subquery).
- The main entry point (Parser::parse_sql) now expects an SQL statement
as input, and returns an `SQLStatement`.
2) Parser changes: instead of detecting the top-level constructs deep
down in the precedence parser (`parse_prefix`) we are able to do it
just right after setting up the parser in the `parse_sql` entry point
(SELECT, again, is kept in the expression parser to demonstrate how
subqueries could be implemented).
The rest of parser changes are mechanical ASTNode -> SQLStatement
replacements resulting from the AST change.
3) Testing changes: for every test - depending on whether the input was
a complete statement or an expresssion - I used an appropriate helper
function:
- `verified` (parses SQL, checks that it round-trips, and returns
the AST) - was replaced by `verified_stmt` or `verified_expr`.
- `parse_sql` (which returned AST without checking it round-tripped)
was replaced by:
- `parse_sql_expr` (same function, for expressions)
- `one_statement_parses_to` (formerly `parses_to`), extended to
deal with statements that are not expected to round-trip.
The weird name is to reduce further churn when implementing
multi-statement parsing.
- `verified_stmt` (in 4 testcases that actually round-tripped)
Fold Token::{Keyword, Identifier, DoubleQuotedString} into one
Token::SQLWord, which has the necessary information (was it a
known keyword and/or was it quoted).
This lets the parser easily accept DoubleQuotedString (a quoted
identifier) everywhere it expects an Identifier in the same match
arm. (To complete support of quoted identifiers, or "delimited
identifiers" as the spec calls them, a TODO in parse_tablename()
ought to be addressed.)
As an aside, per <https://en.wikibooks.org/wiki/SQL_Dialects_Reference/Data_structure_definition/Delimited_identifiers>
sqlite seems to be the only one supporting 'identifier'
(which is rather hairy, since it can also be a string
literal), and `identifier` seems only to be supported by
MySQL. I didn't implement either one.
This also allows the use of `parse`/`expect_keyword` machinery
for non-reserved keywords: previously they relied on the keyword
being a Token::Keyword, which wasn't a Token::Identifier, and so
wasn't accepted as one.
Now whether a keyword can be used as an identifier can be decided
by the parser. (I didn't add a blacklist of "reserved" keywords,
so that any keyword which doesn't have a special meaning in the
parser could be used as an identifier. The list of keywords in
the dialect could be re-used for that purpose at a later stage.)
...as this syntax is not specific to the PostgreSQL dialect.
Also use verified() to assert that parsing + serializing results in the
original SQL string.
Mainly by replacing `assert_eq!(sql, ast.to_string())` with a call to
the recently introduced `verified()` helper or using `parses_to()` where
the expected serialization differs from the original SQL string.
There was one case (parse_implicit_join), where the inputs were different:
let sql = "SELECT * FROM t1,t2";
//vs
let sql = "SELECT * FROM t1, t2";
and since we don't test the whitespace handling in other tests, I just
used the canonical representation as input.
