mirrors/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-10-09 16:34:44 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 17

gh-97973: Return all necessary information from the tokenizer (GH-97984)

Browse source 
Right now, the tokenizer only returns type and two pointers to the start and end of the token.
This PR modifies the tokenizer to return the type and set all of the necessary information,
so that the parser does not have to this.
This commit is contained in:
Lysandros Nikolaou 2022-10-06 16:07:17 -07:00 • committed by GitHub
parent b9d2e81716
commit cbf0afd8a1
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
6 changed files with 161 additions and 148 deletions
Download patch file Download diff file Expand all files Collapse all files

5
Parser/pegen_errors.c
View file

@ -164,11 +164,10 @@ _PyPegen_tokenize_full_source_to_check_for_errors(Parser *p) {
Py_ssize_t current_err_line = current_token->lineno;
int ret = 0;
struct token new_token;
for (;;) {
const char *start;
const char *end;
switch (_PyTokenizer_Get(p->tok, &start, &end)) {
switch (_PyTokenizer_Get(p->tok, &new_token)) {
case ERRORTOKEN:
if (p->tok->level != 0) {
int error_lineno = p->tok->parenlinenostack[p->tok->level-1];