Commit Graph

105 Commits

Author SHA1 Message Date
Terry Jan Reedy
58719a7de7 Merge with 3.3 2014-02-23 23:40:16 -05:00
Terry Jan Reedy
f106f8f29c whitespace 2014-02-23 23:39:57 -05:00
Terry Jan Reedy
40f8c6774b Merge with 3.3 2014-02-23 23:33:44 -05:00
Terry Jan Reedy
9dc3a36c84 Issue #9974: When untokenizing, use row info to insert backslash+newline.
Original patches by A. Kuchling and G. Rees (#12691).
2014-02-23 23:33:08 -05:00
Terry Jan Reedy
79bf89986c Merge with 3.3 2014-02-17 23:12:37 -05:00
Terry Jan Reedy
5b8d2c3af7 Issue #8478: Untokenizer.compat now processes first token from iterator input.
Patch based on lines from Georg Brandl, Eric Snow, and Gareth Rees.
2014-02-17 23:12:16 -05:00
Terry Jan Reedy
8c8d77254f Untokenize, bad assert: Merge with 3.3 2014-02-17 16:46:43 -05:00
Terry Jan Reedy
5e6db31368 Untokenize: An logically incorrect assert tested user input validity.
Replace it with correct logic that raises ValueError for bad input.
Issues #8478 and #12691 reported the incorrect logic.
Add an Untokenize test case and an initial test method.
2014-02-17 16:45:48 -05:00
Serhiy Storchaka
7282ff6d5b Issue #18960: Fix bugs with Python source code encoding in the second line.
* The first line of Python script could be executed twice when the source
encoding (not equal to 'utf-8') was specified on the second line.

* Now the source encoding declaration on the second line isn't effective if
the first line contains anything except a comment.

* As a consequence, 'python -x' works now again with files with the source
encoding declarations specified on the second file, and can be used again
to make Python batch files on Windows.

* The tokenize module now ignore the source encoding declaration on the second
line if the first line contains anything except a comment.

* IDLE now ignores the source encoding declaration on the second line if the
first line contains anything except a comment.

* 2to3 and the findnocoding.py script now ignore the source encoding
declaration on the second line if the first line contains anything except
a comment.
2014-01-09 18:41:59 +02:00
Serhiy Storchaka
768c16ce02 Issue #18960: Fix bugs with Python source code encoding in the second line.
* The first line of Python script could be executed twice when the source
encoding (not equal to 'utf-8') was specified on the second line.

* Now the source encoding declaration on the second line isn't effective if
the first line contains anything except a comment.

* As a consequence, 'python -x' works now again with files with the source
encoding declarations specified on the second file, and can be used again
to make Python batch files on Windows.

* The tokenize module now ignore the source encoding declaration on the second
line if the first line contains anything except a comment.

* IDLE now ignores the source encoding declaration on the second line if the
first line contains anything except a comment.

* 2to3 and the findnocoding.py script now ignore the source encoding
declaration on the second line if the first line contains anything except
a comment.
2014-01-09 18:36:09 +02:00
Ezio Melotti
5833c00427 #19620: merge with 3.3. 2013-11-25 05:16:09 +02:00
Ezio Melotti
4bcc796acc #19620: Fix typo in docstring (noticed by Christopher Welborn). 2013-11-25 05:14:51 +02:00
Serhiy Storchaka
935349406a Issue #18873: The tokenize module, IDLE, 2to3, and the findnocoding.py script
now detect Python source code encoding only in comment lines.
2013-09-16 23:57:00 +03:00
Serhiy Storchaka
dafea85190 Issue #18873: The tokenize module, IDLE, 2to3, and the findnocoding.py script
now detect Python source code encoding only in comment lines.
2013-09-16 23:51:56 +03:00
Andrew Svetlov
f7a17b48d7 Replace IOError with OSError (#16715) 2012-12-25 16:47:37 +02:00
Ezio Melotti
fafa8b7797 #16152: merge with 3.2. 2012-11-03 17:46:51 +02:00
Ezio Melotti
2cc3b4ba9f #16152: fix tokenize to ignore whitespace at the end of the code when no newline is found. Patch by Ned Batchelder. 2012-11-03 17:38:43 +02:00
Florent Xicluna
fed2c51eea Merge branch 2012-07-07 12:26:56 +02:00
Florent Xicluna
11f0b41e9d Issue #14990: tokenize: correctly fail with SyntaxError on invalid encoding declaration. 2012-07-07 12:13:35 +02:00
Christian Heimes
0b3847de6d Issue #15096: Drop support for the ur string prefix 2012-06-20 11:17:58 +02:00
Meador Inge
8d5c0b8c19 Issue #15054: Fix incorrect tokenization of 'b' string literals.
Patch by Serhiy Storchaka.
2012-06-16 21:49:08 -05:00
Brett Cannon
c33f3f2339 Issue #14629: Mention the filename in SyntaxError exceptions from
tokenizer.detect_encoding() (when available).
2012-04-20 13:23:54 -04:00
Martin v. Löwis
63c39fe38e merge 3.2: issue 14629 2012-04-20 14:37:17 +02:00
Martin v. Löwis
63674f4b52 Issue #14629: Raise SyntaxError in tokenizer.detect_encoding
if the first two lines have non-UTF-8 characters without an encoding declaration.
2012-04-20 14:36:47 +02:00
Armin Ronacher
c0eaecafe9 Updated tokenize to support the inverse byte literals new in 3.3 2012-03-04 13:07:57 +00:00