bpo-45494: Fix parser crash when reporting errors involving invalid continuation characters (GH-28993)

There are two errors that this commit fixes:

* The parser was not correctly computing the offset and the string
  source for E_LINECONT errors due to the incorrect usage of strtok().
* The parser was not correctly unwinding the call stack when a tokenizer
  exception happened in rules involving optionals ('?', [...]) as we
  always make them return valid results by using the comma operator. We
  need to check first if we don't have an error before continuing.
This commit is contained in:
Pablo Galindo Salgado 2021-10-19 20:24:12 +01:00 committed by GitHub
parent bda69abe84
commit a106343f63
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
5 changed files with 137 additions and 123 deletions

View file

@ -209,6 +209,10 @@ class ExceptionTests(unittest.TestCase):
src = src.decode(encoding, 'replace')
line = src.split('\n')[lineno-1]
self.assertIn(line, cm.exception.text)
def test_error_offset_continuation_characters(self):
check = self.check
check('"\\\n"(1 for c in I,\\\n\\', 2, 2)
def testSyntaxErrorOffset(self):
check = self.check