Commit Graph

233 Commits

Author SHA1 Message Date
Victor Stinner
ca612a9728 bpo-36778: Remove outdated comment from CodePageTest (GH-13807)
CP65001Test has been removed.
2019-06-04 17:09:10 +02:00
Ammar Askar
a6ec1ce1ac bpo-33361: Fix bug with seeking in StreamRecoders (GH-8278) 2019-05-31 22:44:00 +03:00
Jelle Zijlstra
b3be407288 bpo-33482: fix codecs.StreamRecoder.writelines (GH-6779)
A very simple fix. I found this while writing typeshed stubs for StreamRecoder.


https://bugs.python.org/issue33482
2019-05-22 08:18:26 -07:00
Victor Stinner
d267ac20c3 bpo-36778: cp65001 encoding becomes an alias to utf_8 (GH-13230) 2019-05-10 03:19:54 +02:00
Paul Monson
62dfd7d6fe bpo-35920: Windows 10 ARM32 platform support (GH-11774) 2019-04-25 18:36:45 +00:00
Serhiy Storchaka
7a465cb5ee bpo-24214: Fixed the UTF-8 incremental decoder. (GH-12603)
The bug occurred when the encoded surrogate character is passed
to the incremental decoder in two chunks.
2019-03-30 08:23:38 +02:00
Serhiy Storchaka
c1e2c288f4 bpo-36312: Fix decoders for some code pages. (GH-12369) 2019-03-20 21:45:18 +02:00
Inada Naoki
6a16b18224 bpo-36297: remove "unicode_internal" codec (GH-12342) 2019-03-18 15:44:11 +09:00
Serhiy Storchaka
5b10b98247 bpo-22831: Use "with" to avoid possible fd leaks in tests (part 2). (GH-10929) 2019-03-05 10:06:26 +02:00
Serhiy Storchaka
4013c17911 bpo-35372: Fix the code page decoder for input > 2 GiB. (GH-10848) 2018-12-03 10:36:45 +02:00
Victor Stinner
bde9d6bbb4 bpo-34523, bpo-35322: Fix unicode_encode_locale() (GH-10759)
Fix memory leak in PyUnicode_EncodeLocale() and
PyUnicode_EncodeFSDefault() on error handling.

Changes:

* Fix unicode_encode_locale() error handling
* Fix test_codecs.LocaleCodecTest
2018-11-28 10:26:20 +01:00
Victor Stinner
3d4226a832 bpo-34523: Support surrogatepass in locale codecs (GH-8995)
Add support for the "surrogatepass" error handler in
PyUnicode_DecodeFSDefault() and PyUnicode_EncodeFSDefault()
for the UTF-8 encoding.

Changes:

* _Py_DecodeUTF8Ex() and _Py_EncodeUTF8Ex() now support the
  surrogatepass error handler (_Py_ERROR_SURROGATEPASS).
* _Py_DecodeLocaleEx() and _Py_EncodeLocaleEx() now use
  the _Py_error_handler enum instead of "int surrogateescape" to pass
  the error handler. These functions now return -3 if the error
  handler is unknown.
* Add unit tests on _Py_DecodeLocaleEx() and _Py_EncodeLocaleEx()
  in test_codecs.
* Rename get_error_handler() to _Py_GetErrorHandler() and expose it
  as a private function.
* _freeze_importlib doesn't need config.filesystem_errors="strict"
  workaround anymore.
2018-08-29 22:21:32 +02:00
Zackery Spytz
e349bf2358 bpo-22602: Raise an exception in the UTF-7 decoder for ill-formed sequences starting with "+". (GH-8741)
The UTF-7 decoder now raises UnicodeDecodeError for ill-formed
sequences starting with "+" (as specified in RFC 2152).
2018-08-19 07:43:38 +03:00
Victor Stinner
91106cd9ff bpo-29240: PEP 540: Add a new UTF-8 Mode (#855)
* Add -X utf8 command line option, PYTHONUTF8 environment variable
  and a new sys.flags.utf8_mode flag.
* If the LC_CTYPE locale is "C" at startup: enable automatically the
  UTF-8 mode.
* Add _winapi.GetACP(). encodings._alias_mbcs() now calls
  _winapi.GetACP() to get the ANSI code page
* locale.getpreferredencoding() now returns 'UTF-8' in the UTF-8
  mode. As a side effect, open() now uses the UTF-8 encoding by
  default in this mode.
* Py_DecodeLocale() and Py_EncodeLocale() now use the UTF-8 encoding
  in the UTF-8 Mode.
* Update subprocess._args_from_interpreter_flags() to handle -X utf8
* Skip some tests relying on the current locale if the UTF-8 mode is
  enabled.
* Add test_utf8mode.py.
* _Py_DecodeUTF8_surrogateescape() gets a new optional parameter to
  return also the length (number of wide characters).
* pymain_get_global_config() and pymain_set_global_config() now
  always copy flag values, rather than only copying if the new value
  is greater than the old value.
2017-12-13 12:29:09 +01:00
Serhiy Storchaka
219c2de5ad bpo-32110: codecs.StreamReader.read(n) now returns not more than n (#4499)
characters/bytes for non-negative n.  This makes it compatible with
read() methods of other file-like objects.
2017-11-29 01:30:00 +02:00
Serhiy Storchaka
56cb465cc9 bpo-31825: Fixed OverflowError in the 'unicode-escape' codec (#4058)
and in codecs.escape_decode() when decode an escaped non-ascii byte.
2017-10-20 17:08:15 +03:00
Berker Peksag
7b4bcd2004 Issue #25270: Merge from 3.5 2016-09-16 17:32:06 +03:00
Berker Peksag
4a72a7b6c4 Issue #25270: Prevent codecs.escape_encode() from raising SystemError when an empty bytestring is passed 2016-09-16 17:31:06 +03:00
R David Murray
110b6fecbb #27364: Deprecate invalid escape strings in str/byutes.
Patch by Emanuel Barry, reviewed by Serhiy Storchaka and Martin Panter.
2016-09-08 15:34:08 -04:00
R David Murray
44b548dda8 #27364: fix "incorrect" uses of escape character in the stdlib.
And most of the tools.

Patch by Emanual Barry, reviewed by me, Serhiy Storchaka, and
Martin Panter.
2016-09-08 13:59:53 -04:00
Steve Dower
f5aba58480 Issue #27959: Adds oem encoding, alias ansi to mbcs, move aliasmbcs to codec lookup 2016-09-06 19:42:27 -07:00
Serhiy Storchaka
e437a10d15 Issue #23277: Remove unused imports in tests. 2016-04-24 21:41:02 +03:00
Martin Panter
8b04a945ef Merge typo fixes from 3.5 2016-04-16 09:29:17 +00:00
Martin Panter
119e502277 Fix typos in code comments and documentation 2016-04-16 09:28:57 +00:00
Martin Panter
cda80940ed Issue #15984: Merge PyUnicode doc from 3.5 2016-04-15 02:27:11 +00:00