Thomas Wouters 7eaf2aaf48 Fix crashing bug in tokenizer, when tokenizing files with non-ASCII bytes ... but without a specified encoding: decoding_fgets() (and decoding_feof()) can return NULL and fiddle with the 'tok' struct, making tok->buf NULL. This is okay in the other cases of calls to decoding_*(), it seems, but not in this one. This should get a test added, somewhere, but the testsuite doesn't seem to test encoding anywhere (although plenty of tests use it.) It seems to me that decoding errors in other places in the code (like at the start of a token, instead of in the middle of one) make the code end up adding small integers to NULL pointers, but happen to check for error states before using the calculated new pointers. I haven't been able to trigger any other crashes, in any case. I would nominate this file for a comlete rewrite for Py3k. The whole decoding trick is too bolted-on for my tastes.		2006-03-02 20:41:27 +00:00
..
acceler.c	[Patch #974633] Check PyObject_MALLOC return for error	2004-06-29 14:03:04 +00:00
asdl.py	Patch #1440601: Add col_offset attribute to AST nodes.	2006-03-01 22:49:05 +00:00
asdl_c.py	Fix memory leak on attributes.	2006-03-02 00:31:27 +00:00
bitset.c
firstsets.c	Silence parser generator output.	2001-09-11 16:43:16 +00:00
grammar.c	Fix SF bug #1072182, problems with signed characters.	2005-12-19 06:05:18 +00:00
grammar.mak	Merge ast-branch to head	2005-10-20 19:59:25 +00:00
grammar1.c	PyGrammar_LabelRepr(): sprintf -> PyOS_snprintf.	2001-12-04 03:36:01 +00:00
intrcheck.c	Patch #975056 - fixes for restartable signals on *BSD. In addition,	2004-10-13 14:48:50 +00:00
listnode.c
metagrammar.c	Changes from Jonathan Riehl to allow his pgen extension (PEP 269) to	2003-04-17 14:55:42 +00:00
myreadline.c	In a threads-disabled build, typing Ctrl-C into a raw_input() crashed,	2005-04-07 10:11:19 +00:00
node.c	Patch #1440601: Add col_offset attribute to AST nodes.	2006-03-01 22:49:05 +00:00
parser.c	Patch #1440601: Add col_offset attribute to AST nodes.	2006-03-01 22:49:05 +00:00
parser.h	Patch #1440601: Add col_offset attribute to AST nodes.	2006-03-01 22:49:05 +00:00
parsetok.c	Patch #1440601: Add col_offset attribute to AST nodes.	2006-03-01 22:49:05 +00:00
pgen.c	Revert previous checkin, the check is for <, not ==. i is unsed in non-debug builds, but is used in debug builds	2006-01-08 02:06:01 +00:00
pgenmain.c	- Fix segfault with invalid coding.	2005-10-02 01:48:49 +00:00
printgrammar.c
Python.asdl	Patch #1440601: Add col_offset attribute to AST nodes.	2006-03-01 22:49:05 +00:00
spark.py	Whitespace normalization.	2005-12-25 23:18:31 +00:00
tokenizer.c	Fix crashing bug in tokenizer, when tokenizing files with non-ASCII bytes	2006-03-02 20:41:27 +00:00
tokenizer.h	Patch #1440601: Add col_offset attribute to AST nodes.	2006-03-01 22:49:05 +00:00
tokenizer_pgen.c	Patch #534304: Implement phase 1 of PEP 263.	2002-08-04 17:29:52 +00:00