The majority of this PR is tediously passing `end_lineno` and `end_col_offset` everywhere. Here are non-trivial points:
* It is not possible to reconstruct end positions in AST "on the fly", some information is lost after an AST node is constructed, so we need two more attributes for every AST node `end_lineno` and `end_col_offset`.
* I add end position information to both CST and AST. Although it may be technically possible to avoid adding end positions to CST, the code becomes more cumbersome and less efficient.
* Since the end position is not known for non-leaf CST nodes while the next token is added, this requires a bit of extra care (see `_PyNode_FinalizeEndPos`). Unless I made some mistake, the algorithm should be linear.
* For statements, I "trim" the end position of suites to not include the terminal newlines and dedent (this seems to be what people would expect), for example in
```python
class C:
pass
pass
```
the end line and end column for the class definition is (2, 8).
* For `end_col_offset` I use the common Python convention for indexing, for example for `pass` the `end_col_offset` is 4 (not 3), so that `[0:4]` gives one the source code that corresponds to the node.
* I added a helper function `ast.get_source_segment()`, to get source text segment corresponding to a given AST node. It is also useful for testing.
An (inevitable) downside of this PR is that AST now takes almost 25% more memory. I think however it is probably justified by the benefits.
Remove the docstring attribute of AST types and restore docstring
expression as a first stmt in their body.
Co-authored-by: INADA Naoki <methane@users.noreply.github.com>
* bpo-29463: Add docstring field to some AST nodes.
ClassDef, ModuleDef, FunctionDef, and AsyncFunctionDef has docstring
field for now. It was first statement of there body.
* fix document. thanks travis!
* doc fixes
Issue #26146: Add a new kind of AST node: ast.Constant. It can be used by
external AST optimizers, but the compiler does not emit directly such node.
An optimizer can replace the following AST nodes with ast.Constant:
* ast.NameConstant: None, False, True
* ast.Num: int, float, complex
* ast.Str: str
* ast.Bytes: bytes
* ast.Tuple if items are constants too: tuple
* frozenset
Update code to accept ast.Constant instead of ast.Num and/or ast.Str:
* compiler
* docstrings
* ast.literal_eval()
* Tools/parser/unparse.py
svn+ssh://pythondev@svn.python.org/python/trunk
........
r73376 | benjamin.peterson | 2009-06-11 17:29:23 -0500 (Thu, 11 Jun 2009) | 1 line
remove check for case handled in sub-function
........
r73393 | alexandre.vassalotti | 2009-06-12 13:56:57 -0500 (Fri, 12 Jun 2009) | 2 lines
Clear reference to the static PyExc_RecursionErrorInst in _PyExc_Fini.
........
r73398 | alexandre.vassalotti | 2009-06-12 15:57:12 -0500 (Fri, 12 Jun 2009) | 3 lines
Add const qualifier to PyErr_SetFromErrnoWithFilename and to
PyErr_SetFromErrnoWithUnicodeFilename.
........
r73400 | alexandre.vassalotti | 2009-06-12 16:43:47 -0500 (Fri, 12 Jun 2009) | 2 lines
Delete outdated make file for building the parser with MSVC 6.
........
r73404 | benjamin.peterson | 2009-06-12 20:40:00 -0500 (Fri, 12 Jun 2009) | 1 line
keep the slice.step field as NULL if no step expression is given
........
r73405 | benjamin.peterson | 2009-06-12 22:46:30 -0500 (Fri, 12 Jun 2009) | 1 line
prevent import statements from assigning to None
........
r73409 | benjamin.peterson | 2009-06-13 08:06:21 -0500 (Sat, 13 Jun 2009) | 1 line
allow importing from a module named None if it has an 'as' clause
........
r73419 | benjamin.peterson | 2009-06-13 11:19:19 -0500 (Sat, 13 Jun 2009) | 1 line
set Print.values to NULL if there are no values
........
r73420 | benjamin.peterson | 2009-06-13 12:08:53 -0500 (Sat, 13 Jun 2009) | 1 line
give a better error message when deleting ()
........
r73421 | benjamin.peterson | 2009-06-13 15:23:33 -0500 (Sat, 13 Jun 2009) | 1 line
when no module is given in a 'from' relative import, make ImportFrom.module NULL
........
r73432 | amaury.forgeotdarc | 2009-06-14 16:20:40 -0500 (Sun, 14 Jun 2009) | 3 lines
#6227: Because of a wrong indentation, the test was not testing what it should.
Ensure that the snippet in doctest_aliases actually contains aliases.
........
r73457 | benjamin.peterson | 2009-06-16 18:13:09 -0500 (Tue, 16 Jun 2009) | 1 line
add underscores
........
r73460 | benjamin.peterson | 2009-06-16 22:23:04 -0500 (Tue, 16 Jun 2009) | 1 line
remove unused 'encoding' member from the compiler struct
........
r73485 | benjamin.peterson | 2009-06-19 17:07:47 -0500 (Fri, 19 Jun 2009) | 1 line
remove duplicate test
........
r73486 | benjamin.peterson | 2009-06-19 17:09:17 -0500 (Fri, 19 Jun 2009) | 1 line
add missing assertion #6313
........
r73488 | benjamin.peterson | 2009-06-19 17:16:28 -0500 (Fri, 19 Jun 2009) | 1 line
show that this one isn't used
........
r73489 | benjamin.peterson | 2009-06-19 17:21:12 -0500 (Fri, 19 Jun 2009) | 1 line
use closures
........
r73501 | benjamin.peterson | 2009-06-21 18:01:07 -0500 (Sun, 21 Jun 2009) | 1 line
don't need to add the name 'lambda' as assigned
........
r73502 | benjamin.peterson | 2009-06-21 18:03:36 -0500 (Sun, 21 Jun 2009) | 1 line
remove tmpname support since it's no longer used
........
r73513 | benjamin.peterson | 2009-06-22 20:18:57 -0500 (Mon, 22 Jun 2009) | 1 line
fix grammar
........
r73514 | benjamin.peterson | 2009-06-22 22:01:56 -0500 (Mon, 22 Jun 2009) | 1 line
remove some unused symtable constants
........
