1) #8271: when a byte sequence is invalid, only the start byte and all the
valid continuation bytes are now replaced by U+FFFD, instead of replacing
the number of bytes specified by the start byte.
See http://www.unicode.org/versions/Unicode5.2.0/ch03.pdf (pages 94-95);
2) 5- and 6-bytes-long UTF-8 sequences are now considered invalid (no changes
in behavior);
3) Add code and tests to reject surrogates (U+D800-U+DFFF) as defined in
RFC 3629, but leave it commented out since it's not backward compatible;
4) Change the error messages "unexpected code byte" to "invalid start byte"
and "invalid data" to "invalid continuation byte";
5) Add an extensive set of tests in test_unicode;
6) Fix test_codeccallbacks because it was failing after this change.
Previously, unexpected results occurred when email was passed, for example,
'utf8' as a charset name, since email would accept it but would *not* use
the 'utf-8' codec for it, even though Python itself recognises that as
an alias for utf-8. Now Charset checks with codecs for aliases as well
as its own internal table. Issue 8898 has been opened to change this
further in py3k so that all aliasing is routed through the codecs module.
FreeBSD doesn't have socket.EAI_NODATA. I rewrote the routine because
there's no easy way to conditionally include a context manager in a
with statement. As a side benefit, instead of a stack of context
managers there's now only one.
If a body part ended with \r\n, feedparser, using '$' to terminate its
search for the newline, would match on the \r\n, and think that it needed
to strip two characters in order to account for the line end before the
boundary. That made it chop one too many characters off the end of
the body part. Using \Z makes the match correct.
Patch and test by Tony Nelson.
the curses module must be linked against ncurses as well. Otherwise it
is not safe to load both the readline and curses modules in an application.
Thanks Thomas Dickey for answering questions about ncurses/ncursesw
and readline!
for extracting symbolic and hard link entries as regular files as a
work-around on platforms that do not support filesystem links.
This stopped working reliably after a change in r74571. I also added
a few tests for this functionality.
honor the MacOSX SDK when one is specified.
This is needed to be able to build using the 10.4u SDK while running
on OSX 10.6.
This is a fixed version of the patch in r80963, I've tested this patch
on OSX and Linux.
Fixes (mysterious, to the end user) UnicodeErrors when using utf-8 as
the charset and unicode as the _text argument. Also makes the way in
which unicode gets encoded to quoted printable for other charsets more
sane (it only worked by accident previously). The _payload now is encoded
to the charset.output_charset if it is unicode.
unquote is duplicated in the two files to avoid a circular reference.
(This is fixed in Python3.) Updates keep getting made to the public unquote
without fixing the urlparse one, however, so this fix syncs the two
and adds a comment to both to make sure changes are applied to both.
