Dhruv Manilawala e4a4660925 Support help end escape command with priority (#6272) ... ## Summary This PR adds support for help end escape command in the lexer. ### What are "help end escape commands"? First, the escape commands are special IPython syntax which enhances the functionality for the IPython REPL. There are 9 types of escape kinds which are recognized by the tokens which are present at the start of the command (`?`, `??`, `!`, `!!`, etc.). Here, the help command is using either the `?` or `??` token at the start (`?str.replace` for example). Those 2 tokens are also supported when they're at the end of the command (`str.replace?`), but the other tokens aren't supported in that position. There are mainly two types of help end escape commands: 1. Ending with either `?` or `??`, but it also starts with one of the escape tokens (`%matplotlib?`) 2. On the other hand, there's a stricter version for (1) which doesn't start with any escape tokens (`str.replace?`) This PR adds support for (1) while (2) will be supported in the parser. ### Priority Now, if the command starts and ends with an escape token, how do we decide the kind of this command? This is where priority comes into picture. This is simple as there's only one priority where `?`/`??` at the end takes priority over any other escape token and all of the other tokens are at the same priority. Remember that only `?`/`??` at the end is considered valid. This is mainly useful in the case where someone would want to invoke the help command on the magic command itself. For example, in `%matplotlib?` the help command takes priority which means that we want help for the `matplotlib` magic function instead of calling the magic function itself. ### Specification Here's where things get a bit tricky. What if there are question mark tokens at both ends. How do we decide if it's `Help` (`?`) kind or `Help2` (`??`) kind? | | Magic | Value | Kind | | --- | --- | --- | --- | | 1 | `?foo?` | `foo` | `Help` | | 2 | `??foo?` | `foo` | `Help` | | 3 | `?foo??` | `foo` | `Help2` | | 4 | `??foo??` | `foo` | `Help2` | | 5 | `???foo??` | `foo` | `Help2` | | 6 | `??foo???` | `foo???` | `Help2` | | 7 | `???foo???` | `?foo???` | `Help2` | Looking at the above table: - The question mark tokens on the right takes priority over the ones on the left but only if the number of question mark on the right is 1 or 2. - If there are more than 2 question mark tokens on the right side, then the left side is used to determine the same. - If the right side is used to determine the kind, then all of the question marks and whitespaces on the left side are ignored in the `value`, but if it’s the other way around, then all of the extra question marks are part of the `value`. ### References - IPython implementation using the regex: 292e3a2345/IPython/core/inputtransformer2.py (L454-L462) - Priorities: 292e3a2345/IPython/core/inputtransformer2.py (L466-L469) ## Test Plan Add a bunch of test cases for the lexer and verify that it matches the behavior of IPython transformer. resolves: #6357		2023-08-07 21:01:02 +05:30
..
src	Support help end escape command with priority (#6272)	2023-08-07 21:01:02 +05:30
build.rs	Pull in RustPython parser (#6099)	2023-07-27 09:29:11 +00:00
Cargo.toml	Support help end escape command with priority (#6272)	2023-08-07 21:01:02 +05:30