Andrew Svetlov
f7a17b48d7
Replace IOError with OSError ( #16715 )
2012-12-25 16:47:37 +02:00
Ezio Melotti
fafa8b7797
#16152 : merge with 3.2.
2012-11-03 17:46:51 +02:00
Ezio Melotti
2cc3b4ba9f
#16152 : fix tokenize to ignore whitespace at the end of the code when no newline is found. Patch by Ned Batchelder.
2012-11-03 17:38:43 +02:00
Florent Xicluna
fed2c51eea
Merge branch
2012-07-07 12:26:56 +02:00
Florent Xicluna
11f0b41e9d
Issue #14990 : tokenize: correctly fail with SyntaxError on invalid encoding declaration.
2012-07-07 12:13:35 +02:00
Christian Heimes
0b3847de6d
Issue #15096 : Drop support for the ur string prefix
2012-06-20 11:17:58 +02:00
Meador Inge
8d5c0b8c19
Issue #15054 : Fix incorrect tokenization of 'b' string literals.
...
Patch by Serhiy Storchaka.
2012-06-16 21:49:08 -05:00
Brett Cannon
c33f3f2339
Issue #14629 : Mention the filename in SyntaxError exceptions from
...
tokenizer.detect_encoding() (when available).
2012-04-20 13:23:54 -04:00
Martin v. Löwis
63c39fe38e
merge 3.2: issue 14629
2012-04-20 14:37:17 +02:00
Martin v. Löwis
63674f4b52
Issue #14629 : Raise SyntaxError in tokenizer.detect_encoding
...
if the first two lines have non-UTF-8 characters without an encoding declaration.
2012-04-20 14:36:47 +02:00
Armin Ronacher
c0eaecafe9
Updated tokenize to support the inverse byte literals new in 3.3
2012-03-04 13:07:57 +00:00
Armin Ronacher
6ecf77b3f8
Basic support for PEP 414 without docs or tests.
2012-03-04 12:04:06 +00:00
Meador Inge
00c7f85298
Issue #2134 : Add support for tokenize.TokenInfo.exact_type.
2012-01-19 00:44:45 -06:00
Antoine Pitrou
10a99b024d
Issue #13150 : The tokenize module doesn't compile large regular expressions at startup anymore.
...
Instead, the re module's standard caching does its work.
2011-10-11 15:45:56 +02:00
Meador Inge
14c0f03b58
Issue #12943 : python -m tokenize support has been added to tokenize.
2011-10-07 08:53:38 -05:00
Brett Cannon
45b96d373e
Merged revisions 88498 via svnmerge from
...
svn+ssh://pythondev@svn.python.org/python/branches/py3k
........
r88498 | brett.cannon | 2011-02-21 19:25:12 -0800 (Mon, 21 Feb 2011) | 8 lines
Issue #11074 : Make 'tokenize' so it can be reloaded.
The module stored away the 'open' object as found in the global namespace
(which fell through to the built-in namespace) since it defined its own 'open'.
Problem is that if you reloaded the module it then grabbed the 'open' defined
in the previous load, leading to code that infinite recursed. Switched to
simply call builtins.open directly.
........
2011-02-22 03:35:18 +00:00
Brett Cannon
f3042782af
Issue #11074 : Make 'tokenize' so it can be reloaded.
...
The module stored away the 'open' object as found in the global namespace
(which fell through to the built-in namespace) since it defined its own 'open'.
Problem is that if you reloaded the module it then grabbed the 'open' defined
in the previous load, leading to code that infinite recursed. Switched to
simply call builtins.open directly.
2011-02-22 03:25:12 +00:00
Alexander Belopolsky
b9d10d08c4
Issue #10386 : Added __all__ to token module; this simplifies importing
...
in tokenize module and prevents leaking of private names through
import *.
2010-11-11 14:07:41 +00:00
Victor Stinner
58c0752a33
Issue #10335 : Add tokenize.open(), detect the file encoding using
...
tokenize.detect_encoding() and open it in read only mode.
2010-11-09 01:08:59 +00:00
Raymond Hettinger
a0e79408bc
A little bit more readable repr method.
2010-09-09 08:29:05 +00:00
Raymond Hettinger
3fb79c747b
Experiment: Let collections.namedtuple() do the work. This should work now that _collections is pre-built. The buildbots will tell us shortly.
2010-09-09 07:15:18 +00:00
Raymond Hettinger
6c60d099e5
Improve the repr for the TokenInfo named tuple.
2010-09-09 04:32:39 +00:00
Florent Xicluna
43e4ea1b17
Remove unused import, fix typo and rewrap docstrings.
2010-09-03 19:54:02 +00:00
Benjamin Peterson
33856de84d
handle names starting with non-ascii characters correctly #9712
2010-08-30 14:41:20 +00:00
Benjamin Peterson
1613ed8108
fix for files with coding cookies and BOMs
2010-03-18 22:34:15 +00:00
