language-servers/ruff
1
0
Fork 0
mirror of https://github.com/astral-sh/ruff.git synced 2025-08-16 00:20:38 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 3

Add support for help end IPython escape commands (#6358)

Browse source 
## Summary

This PR adds support for a stricter version of help end escape
commands[^1] in the parser. By stricter, I mean that the escape tokens
are only at the end of the command and there are no tokens at the start.
This makes it difficult to implement it in the lexer without having to
do a lot of look aheads or keeping track of previous tokens.

Now, as we're adding this in the parser, the lexer needs to recognize
and emit a new token for `?`. So, `Question` token is added which will
be recognized only in `Jupyter` mode.

The conditions applied are the same as the ones in the original
implementation in IPython codebase (which is a regex):
* There can only be either 1 or 2 question mark(s) at the end
* The node before the question mark can be a `Name`, `Attribute`,
`Subscript` (only with integer constants in slice position), or any
combination of the 3 nodes.

## Test Plan

Added test cases for various combination of the possible nodes in the
command value position and update the snapshots.

fixes: #6359
fixes: #5030 (This is the final piece)

[^1]: https://github.com/astral-sh/ruff/pull/6272#issue-1833094281
This commit is contained in:
Dhruv Manilawala 2023-08-09 10:28:52 +05:30 • committed by GitHub
parent 887a47cad9
commit e257c5af32
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
6 changed files with 40274 additions and 34786 deletions
Download patch file Download diff file Expand all files Collapse all files

6
crates/ruff_python_parser/src/token.rs
View file

@ -64,6 +64,8 @@ pub enum Tok {
/// Token value for a dedent.
Dedent,
EndOfFile,
/// Token value for a question mark `?`. This is only used in [`Mode::Jupyter`].
Question,
/// Token value for a left parenthesis `(`.
Lpar,
/// Token value for a right parenthesis `)`.
@ -240,6 +242,7 @@ impl fmt::Display for Tok {
StartModule => f.write_str("StartProgram"),
StartExpression => f.write_str("StartExpression"),
EndOfFile => f.write_str("EOF"),
Question => f.write_str("'?'"),
Lpar => f.write_str("'('"),
Rpar => f.write_str("')'"),
Lsqb => f.write_str("'['"),
@ -461,6 +464,8 @@ pub enum TokenKind {
/// Token value for a dedent.
Dedent,
EndOfFile,
/// Token value for a question mark `?`.
Question,
/// Token value for a left parenthesis `(`.
Lpar,
/// Token value for a right parenthesis `)`.
@ -783,6 +788,7 @@ impl TokenKind {
Tok::Indent => TokenKind::Indent,
Tok::Dedent => TokenKind::Dedent,
Tok::EndOfFile => TokenKind::EndOfFile,
Tok::Question => TokenKind::Question,
Tok::Lpar => TokenKind::Lpar,
Tok::Rpar => TokenKind::Rpar,
Tok::Lsqb => TokenKind::Lsqb,