mirrors/cpython
1
0
Fork 0
mirror of https://github.com/python/cpython.git synced 2025-09-26 18:29:57 +00:00
Code Issues Projects Releases Packages Wiki Activity Actions 10

fixes gh-109559: Update unicodedata for Unicode 15.1.0 (GH-109560)

Browse source 
---------

Co-authored-by: Benjamin Peterson <benjamin@python.org>
This commit is contained in:
James Gerity 2023-09-20 01:07:47 -04:00 • committed by GitHub
parent 1293fcc3c6
commit def828995a
No known key found for this signature in database
GPG key ID: 4AEE18F83AFDEB23
9 changed files with 19551 additions and 19133 deletions
Download patch file Download diff file Expand all files Collapse all files

8
Doc/library/stdtypes.rst
View file

@ -1641,7 +1641,7 @@ expression support in the :mod:`re` module).
The casefolding algorithm is The casefolding algorithm is
`described in section 3.13 'Default Case Folding' of the Unicode Standard `described in section 3.13 'Default Case Folding' of the Unicode Standard
<https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf>`__. <https://www.unicode.org/versions/Unicode15.1.0/ch03.pdf>`__.
.. versionadded:: 3.3 .. versionadded:: 3.3
@ -1805,7 +1805,7 @@ expression support in the :mod:`re` module).
property being one of "Lm", "Lt", "Lu", "Ll", or "Lo". Note that this is different property being one of "Lm", "Lt", "Lu", "Ll", or "Lo". Note that this is different
from the `Alphabetic property defined in the section 4.10 'Letters, Alphabetic, and from the `Alphabetic property defined in the section 4.10 'Letters, Alphabetic, and
Ideographic' of the Unicode Standard Ideographic' of the Unicode Standard
<https://www.unicode.org/versions/Unicode15.0.0/ch04.pdf>`_. <https://www.unicode.org/versions/Unicode15.1.0/ch04.pdf>`_.
.. method:: str.isascii() .. method:: str.isascii()
@ -1941,7 +1941,7 @@ expression support in the :mod:`re` module).
The lowercasing algorithm used is The lowercasing algorithm used is
`described in section 3.13 'Default Case Folding' of the Unicode Standard `described in section 3.13 'Default Case Folding' of the Unicode Standard
<https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf>`__. <https://www.unicode.org/versions/Unicode15.1.0/ch03.pdf>`__.
.. method:: str.lstrip([chars]) .. method:: str.lstrip([chars])
@ -2290,7 +2290,7 @@ expression support in the :mod:`re` module).
The uppercasing algorithm used is The uppercasing algorithm used is
`described in section 3.13 'Default Case Folding' of the Unicode Standard `described in section 3.13 'Default Case Folding' of the Unicode Standard
<https://www.unicode.org/versions/Unicode15.0.0/ch03.pdf>`__. <https://www.unicode.org/versions/Unicode15.1.0/ch03.pdf>`__.
.. method:: str.zfill(width) .. method:: str.zfill(width)

8
Doc/library/unicodedata.rst
View file

@ -17,8 +17,8 @@
This module provides access to the Unicode Character Database (UCD) which This module provides access to the Unicode Character Database (UCD) which
defines character properties for all Unicode characters. The data contained in defines character properties for all Unicode characters. The data contained in
this database is compiled from the `UCD version 15.0.0 this database is compiled from the `UCD version 15.1.0
<https://www.unicode.org/Public/15.0.0/ucd>`_. <https://www.unicode.org/Public/15.1.0/ucd>`_.
The module uses the same names and symbols as defined by Unicode The module uses the same names and symbols as defined by Unicode
Standard Annex #44, `"Unicode Character Database" Standard Annex #44, `"Unicode Character Database"
@ -175,6 +175,6 @@ Examples:
.. rubric:: Footnotes .. rubric:: Footnotes
.. [#] https://www.unicode.org/Public/15.0.0/ucd/NameAliases.txt .. [#] https://www.unicode.org/Public/15.1.0/ucd/NameAliases.txt
.. [#] https://www.unicode.org/Public/15.0.0/ucd/NamedSequences.txt .. [#] https://www.unicode.org/Public/15.1.0/ucd/NamedSequences.txt

8
Doc/reference/lexical_analysis.rst
View file

@ -315,7 +315,7 @@ The Unicode category codes mentioned above stand for:
* *Nd* - decimal numbers * *Nd* - decimal numbers
* *Pc* - connector punctuations * *Pc* - connector punctuations
* *Other_ID_Start* - explicit list of characters in `PropList.txt * *Other_ID_Start* - explicit list of characters in `PropList.txt
<https://www.unicode.org/Public/15.0.0/ucd/PropList.txt>`_ to support backwards <https://www.unicode.org/Public/15.1.0/ucd/PropList.txt>`_ to support backwards
compatibility compatibility
* *Other_ID_Continue* - likewise * *Other_ID_Continue* - likewise
@ -323,8 +323,8 @@ All identifiers are converted into the normal form NFKC while parsing; compariso
of identifiers is based on NFKC. of identifiers is based on NFKC.
A non-normative HTML file listing all valid identifier characters for Unicode A non-normative HTML file listing all valid identifier characters for Unicode
15.0.0 can be found at 15.1.0 can be found at
https://www.unicode.org/Public/15.0.0/ucd/DerivedCoreProperties.txt https://www.unicode.org/Public/15.1.0/ucd/DerivedCoreProperties.txt
.. _keywords: .. _keywords:
@ -1045,4 +1045,4 @@ occurrence outside string literals and comments is an unconditional error:
.. rubric:: Footnotes .. rubric:: Footnotes
.. [#] https://www.unicode.org/Public/15.0.0/ucd/NameAliases.txt .. [#] https://www.unicode.org/Public/15.1.0/ucd/NameAliases.txt

1
Misc/NEWS.d/next/Library/2023-09-19-01-22-43.gh-issue-109559.ijaycU.rst Normal file
View file

@ -0,0 +1 @@
Update :mod:`unicodedata` database to Unicode 15.1.0.

1
Modules/unicodedata.c
View file

@ -1035,6 +1035,7 @@ is_unified_ideograph(Py_UCS4 code)
(0x2B740 <= code && code <= 0x2B81D) || /* CJK Ideograph Extension D */ (0x2B740 <= code && code <= 0x2B81D) || /* CJK Ideograph Extension D */
(0x2B820 <= code && code <= 0x2CEA1) || /* CJK Ideograph Extension E */ (0x2B820 <= code && code <= 0x2CEA1) || /* CJK Ideograph Extension E */
(0x2CEB0 <= code && code <= 0x2EBE0) || /* CJK Ideograph Extension F */ (0x2CEB0 <= code && code <= 0x2EBE0) || /* CJK Ideograph Extension F */
(0x2EBF0 <= code && code <= 0x2EE5D) || /* CJK Ideograph Extension I */
(0x30000 <= code && code <= 0x3134A) || /* CJK Ideograph Extension G */ (0x30000 <= code && code <= 0x3134A) || /* CJK Ideograph Extension G */
(0x31350 <= code && code <= 0x323AF); /* CJK Ideograph Extension H */ (0x31350 <= code && code <= 0x323AF); /* CJK Ideograph Extension H */
} }

3146
Modules/unicodedata_db.h generated
View file

File diff suppressed because it is too large Load diff

33172
Modules/unicodename_db.h generated
View file

File diff suppressed because it is too large Load diff

2311
Objects/unicodetype_db.h generated
View file

File diff suppressed because it is too large Load diff

29
Tools/unicode/makeunicodedata.py
View file

@ -44,7 +44,7 @@ VERSION = "3.3"
# * Doc/library/stdtypes.rst, and # * Doc/library/stdtypes.rst, and
# * Doc/library/unicodedata.rst # * Doc/library/unicodedata.rst
# * Doc/reference/lexical_analysis.rst (two occurrences) # * Doc/reference/lexical_analysis.rst (two occurrences)
UNIDATA_VERSION = "15.0.0" UNIDATA_VERSION = "15.1.0"
UNICODE_DATA = "UnicodeData%s.txt" UNICODE_DATA = "UnicodeData%s.txt"
COMPOSITION_EXCLUSIONS = "CompositionExclusions%s.txt" COMPOSITION_EXCLUSIONS = "CompositionExclusions%s.txt"
EASTASIAN_WIDTH = "EastAsianWidth%s.txt" EASTASIAN_WIDTH = "EastAsianWidth%s.txt"
@ -101,15 +101,16 @@ EXTENDED_CASE_MASK = 0x4000
# these ranges need to match unicodedata.c:is_unified_ideograph # these ranges need to match unicodedata.c:is_unified_ideograph
cjk_ranges = [ cjk_ranges = [
('3400', '4DBF'), ('3400', '4DBF'), # CJK Ideograph Extension A CJK
('4E00', '9FFF'), ('4E00', '9FFF'), # CJK Ideograph
('20000', '2A6DF'), ('20000', '2A6DF'), # CJK Ideograph Extension B
('2A700', '2B739'), ('2A700', '2B739'), # CJK Ideograph Extension C
('2B740', '2B81D'), ('2B740', '2B81D'), # CJK Ideograph Extension D
('2B820', '2CEA1'), ('2B820', '2CEA1'), # CJK Ideograph Extension E
('2CEB0', '2EBE0'), ('2CEB0', '2EBE0'), # CJK Ideograph Extension F
('30000', '3134A'), ('2EBF0', '2EE5D'), # CJK Ideograph Extension I
('31350', '323AF'), ('30000', '3134A'), # CJK Ideograph Extension G
('31350', '323AF'), # CJK Ideograph Extension H
] ]
@ -1105,11 +1106,15 @@ class UnicodeData:
table[i].east_asian_width = widths[i] table[i].east_asian_width = widths[i]
self.widths = widths self.widths = widths
for char, (p,) in UcdFile(DERIVED_CORE_PROPERTIES, version).expanded(): for char, (propname, *propinfo) in UcdFile(DERIVED_CORE_PROPERTIES, version).expanded():
if propinfo:
# this is not a binary property, ignore it
continue
if table[char]: if table[char]:
# Some properties (e.g. Default_Ignorable_Code_Point) # Some properties (e.g. Default_Ignorable_Code_Point)
# apply to unassigned code points; ignore them # apply to unassigned code points; ignore them
table[char].binary_properties.add(p) table[char].binary_properties.add(propname)
for char_range, value in UcdFile(LINE_BREAK, version): for char_range, value in UcdFile(LINE_BREAK, version):
if value not in MANDATORY_LINE_BREAKS: if value not in MANDATORY_LINE_BREAKS: