[3.10] bpo-45494: Fix parser crash when reporting errors involving invalid continuation characters (GH-28993) (GH-29070)

There are two errors that this commit fixes:

* The parser was not correctly computing the offset and the string
  source for E_LINECONT errors due to the incorrect usage of strtok().
* The parser was not correctly unwinding the call stack when a tokenizer
  exception happened in rules involving optionals ('?', [...]) as we
  always make them return valid results by using the comma operator. We
  need to check first if we don't have an error before continuing..
(cherry picked from commit a106343f63)

Co-authored-by: Pablo Galindo Salgado <Pablogsal@gmail.com>
This commit is contained in:
Łukasz Langa 2021-10-19 22:31:18 +02:00 committed by GitHub
parent 325b2c2234
commit 5c9cab595e
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
5 changed files with 137 additions and 123 deletions

View file

@ -209,6 +209,10 @@ class ExceptionTests(unittest.TestCase):
src = src.decode(encoding, 'replace')
line = src.split('\n')[lineno-1]
self.assertIn(line, cm.exception.text)
def test_error_offset_continuation_characters(self):
check = self.check
check('"\\\n"(1 for c in I,\\\n\\', 2, 2)
def testSyntaxErrorOffset(self):
check = self.check