mirrors/RustPython-Parser
1
0
Fork 0
mirror of https://github.com/RustPython/Parser.git synced 2025-08-31 15:48:24 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions
369 commits 5 branches 3 tags 4.4 MiB
Commit graph

46 commits

Author SHA1 Message Date
Steve Shi
a2e3209c42
Replace num-bigint with malachite-bigint (#18) 
Co-authored-by: Jeong YunWon <jeong@youknowone.org>
 2023-06-02 17:06:18 +09:00
Jeong, YunWon
4de0cb1827
Parse Trait (#71) 2023-05-28 21:03:27 +09:00
Jeong, YunWon
735c06d5f4
Fix full-lexer feature (#38) 2023-05-16 15:45:03 +09:00
Charlie Marsh
10dda125ff
Always emit non-logical newlines for 'empty' lines (#27) 2023-05-15 14:13:05 -04:00
Jeong, YunWon
27e3873dc2
Add full-lexer feature (#36) 2023-05-16 02:21:34 +09:00
Jeong YunWon
1d366d52ab Let located only for python located stuff 2023-05-10 14:35:38 +09:00
Jeong YunWon
a3d9d8cb14 numerous refactoring 
- Split parser core and compiler core. Fix #14
- AST int type to `u32`
- Updated asdl_rs.py and update_asdl.sh fix #6
- Use `ruff_python_ast::SourceLocation` for Python source location. Deleted our own Location.
- Renamed ast::Located to ast::Attributed to distinguish terms for TextSize and SourceLocation
- `ast::<Node>`s for TextSize located ast. `ast::located::<Node>` for Python source located ast.
- And also strictly renaming `located` to refer only python location related interfaces.
- `SourceLocator` to convert locations.
- New `source-code` features of to disable python locations when unnecessary.
- Also including fully merging https://github.com/astral-sh/RustPython/pull/4 closes #9
 2023-05-10 14:35:38 +09:00
Jeong YunWon
a14e43e03a Separate byteoffset ast and located ast 2023-05-09 00:21:52 +09:00
Jeong YunWon
f47dfca4e3 Rename compiler Location to TextSize 2023-05-08 03:38:10 +09:00
Micha Reiser
58c35ab458 Replace row/column based Location with byte-offsets. 2023-05-08 03:38:10 +09:00
Jeong YunWon
f9b5469642 Update cspell for compiler 2023-03-16 22:39:09 +09:00
Jeong YunWon
97a08ee77b remove #[macro_use] 2023-02-22 20:41:27 +09:00
Jeong YunWon
cb8c6fb78d Flatten rustpython_parser interface 2023-02-22 20:32:31 +09:00
Jeong YunWon
8580e4ebb5 make_tokenizer -> lex to integrate terms 
we don't distinguish scanner or tokenizer from lexer
 2023-02-22 20:28:15 +09:00
Jeong YunWon
1511b6631b Break down rustpython_parser::error module 
because it doesn't share any common errors but specific error for each sub module
 2023-02-22 20:28:01 +09:00
Charlie Marsh
2a8aa6f308 Always wrap in SoftKeywordTransformer 2023-02-21 19:18:42 -05:00
Dimitris Fasarakis Hilliard
659f4dd8bf Document lexer. 2023-02-07 21:43:57 +02:00
Dimitris Fasarakis Hilliard
bd158089e0 Move NewLineHandler inline, don't check each character twice. 2023-02-07 20:58:53 +02:00
Dimitris Fasarakis Hilliard
1468fe46ab Hint that the unwrap should always succeed. 2023-02-01 12:17:33 +02:00
Dimitris Fasarakis Hilliard
c9364718b4 Eat for comma. 2023-01-31 12:26:05 +02:00
Dimitris Fasarakis Hilliard
38cf933bcb Add initial capacities, use u32s for indents/spaces. 2023-01-31 12:26:05 +02:00
Dimitris Fasarakis Hilliard
838990ae15 Don't call is_emoji_presentation for each invocation of consume_normal 2023-01-31 12:26:05 +02:00
Dimitris Fasarakis Hilliard
aa0290bbfc Match on ascii start/continuation characters before calling functions. 2023-01-31 12:26:05 +02:00
Ran Benita
674eeec29c Add NonLogicalNewline token 
This token is completely ignored by the parser, but it's useful for
other users of the lexer, such as the Ruff linter. For example, the
token is helpful for a "trailing comma" lint.

The same idea exists in Python's `tokenize` module - there is a NEWLINE
token (logical newline), and a NL token (non-logical newline).

Fixes #4385.
 2023-01-12 16:47:12 +02:00
harupy
a4a5366504 Include comment text in token 2023-01-06 23:29:20 +09:00
harupy
2dfd053bed Implement Default for Location 2023-01-05 22:48:47 +09:00
harupy
7f552e4594 Address comments 2023-01-05 18:24:54 +09:00
harupy
9efa872023 Use try_from 2023-01-05 01:18:30 +09:00
harupy
fd8468c5eb Simplify string check 2023-01-04 23:57:50 +09:00
harupy
33a62789f7 Address comment 2023-01-04 00:03:18 +09:00
harupy
300710f7db Improve CharWindow 2023-01-03 17:27:35 +09:00
Jeong YunWon
f8787a9377 Move (c)format basic implementations to rustpython-common 2023-01-02 20:21:36 +09:00
harupy
439298e735 Fix FormattedValue location 2022-12-30 21:39:29 +09:00
harupy
c21d0d9283 Fix make_tokenizer_located 2022-12-27 21:54:18 +09:00
yt2b
bd0c15d34e Fix comment 2022-12-26 09:30:12 +09:00
yt2b
ce0be73841 Add BOM check 2022-12-25 11:15:29 +09:00
Jeong YunWon
28785784b2 Fix nightly clippy warnings 2022-12-05 12:18:16 +09:00
yt2b
e8200ab674 use bool.then 2022-11-24 09:23:20 +09:00
yt2b
492f09298f restore if expression 2022-11-23 11:30:54 +09:00
yt2b
35eea0b8ec Refactor lexer functions 2022-11-22 21:10:19 +09:00
yt2b
1eeddb521e Add test 2022-11-21 22:36:05 +09:00
Bongjun Jang
4e5626dfd5 Refactor lexer struct (#4257) 2022-11-19 21:43:34 +09:00
Charlie Marsh
b6c230f3ca Implement Tok::Comment 2022-11-07 10:33:55 -05:00
Charlie Marsh
519718e65d Start simple string at quote mark 2022-10-16 11:25:46 -04:00
Charlie Marsh
3397737a76 Start string location at kind or quote prefix 2022-10-15 11:03:50 -04:00
Jeong YunWon
53c48bf6b9 reorganize compiler crates 2022-08-22 04:42:42 +09:00