Commit Graph

193 Commits

Author SHA1 Message Date
Guido van Rossum
dcfcd146f8 bpo-35766: Merge typed_ast back into CPython (GH-11645) 2019-01-31 12:40:27 +01:00
Ivan Levkivskyi
9932a22897 bpo-33416: Add end positions to Python AST (GH-11605)
The majority of this PR is tediously passing `end_lineno` and `end_col_offset` everywhere. Here are non-trivial points:
* It is not possible to reconstruct end positions in AST "on the fly", some information is lost after an AST node is constructed, so we need two more attributes for every AST node `end_lineno` and `end_col_offset`.
* I add end position information to both CST and AST.  Although it may be technically possible to avoid adding end positions to CST, the code becomes more cumbersome and less efficient.
* Since the end position is not known for non-leaf CST nodes while the next token is added, this requires a bit of extra care (see `_PyNode_FinalizeEndPos`). Unless I made some mistake, the algorithm should be linear.
* For statements, I "trim" the end position of suites to not include the terminal newlines and dedent (this seems to be what people would expect), for example in
  ```python
  class C:
      pass

  pass
  ```
  the end line and end column for the class definition is (2, 8).
* For `end_col_offset` I use the common Python convention for indexing, for example for `pass` the `end_col_offset` is 4 (not 3), so that `[0:4]` gives one the source code that corresponds to the node.
* I added a helper function `ast.get_source_segment()`, to get source text segment corresponding to a given AST node. It is also useful for testing.

An (inevitable) downside of this PR is that AST now takes almost 25% more memory. I think however it is probably justified by the benefits.
2019-01-22 11:18:22 +00:00
Serhiy Storchaka
62be74290a bpo-33012: Fix invalid function cast warnings with gcc 8. (GH-6749)
Fix invalid function cast warnings with gcc 8
for method conventions different from METH_NOARGS, METH_O and
METH_VARARGS excluding Argument Clinic generated code.
2018-11-27 13:27:31 +02:00
Victor Stinner
3bb183d7fb bpo-35177, Python-ast.h: Fix "Yield" compiler warning (GH-10664)
Partially revert commit 5f2df88b63:
add "#undef Yield" to .c files after including Python-ast.h.

Fix the warning:

    winbase.h(102): warning C4005: 'Yield': macro redefinition
2018-11-22 18:38:38 +01:00
Victor Stinner
5f2df88b63 bpo-35177: Add dependencies between header files (GH-10361)
* ast.h now includes Python-ast.h and node.h
* parsetok.h now includes node.h and grammar.h
* symtable.h now includes Python-ast.h
* Modify asdl_c.py to enhance Python-ast.h:

  * Add #ifndef/#define Py_PYTHON_AST_H to be able to include the header
    twice
  * Add "extern { ... }" for C++
  * Undefine "Yield" macro conflicting with winbase.h

* Remove "#undef Yield" from C files, it's now done in Python-ast.h
* Remove now useless includes in C files
2018-11-12 00:56:19 +01:00
Serhiy Storchaka
e5362eaa75 bpo-33308: Fix a crash in the parser module when convert an ST object. (#6519)
Converting with line_info=False and col_info=True crashed before.
2018-04-19 01:55:37 +03:00
stratakis
e8b1965639 bpo-23699: Use a macro to reduce boilerplate code in rich comparison functions (GH-793) 2017-11-02 20:32:54 +10:00
Serhiy Storchaka
a79f4c2195 bpo-30070: Fixed leaks and crashes in errors handling in the parser module. (#1131) 2017-04-19 21:09:21 +03:00
Serhiy Storchaka
228b12edcc Issue #28999: Use Py_RETURN_NONE, Py_RETURN_TRUE and Py_RETURN_FALSE wherever
possible.  Patch is writen with Coccinelle.
2017-01-23 09:47:21 +02:00
Serhiy Storchaka
06515833fe Replaced outdated macros _PyUnicode_AsString and _PyUnicode_AsStringAndSize
with PyUnicode_AsUTF8 and PyUnicode_AsUTF8AndSize.
2016-11-20 09:13:07 +02:00
Benjamin Peterson
53595c4879 replace custom validation logic in the parse module with a simple DFA validator (closes #26526)
Patch from A. Skrobov.
2016-06-02 11:30:18 -07:00
Serhiy Storchaka
ef1585eb9a Issue #25923: Added more const qualifiers to signatures of static and private functions. 2015-12-25 20:01:53 +02:00
Serhiy Storchaka
2d06e84455 Issue #25923: Added the const qualifier to static constant arrays. 2015-12-25 19:53:18 +02:00
Serhiy Storchaka
5c4064e8bd Issue #25421: __sizeof__ methods of builtin types now use dynamic basic size.
This allows sys.getsize() to work correctly with their subclasses with
__slots__ defined.
2015-12-19 20:05:25 +02:00
Yury Selivanov
14acf5f41d Issue #24791: Fix grammar regression for call syntax: 'g(*a or b)'. 2015-08-05 17:54:10 -04:00
Benjamin Peterson
de12b79cd6 allow test node after ** in calls (closes #24176) 2015-05-16 09:44:45 -04:00
Yury Selivanov
7544508f02 PEP 0492 -- Coroutines with async and await syntax. Issue #24017. 2015-05-11 22:57:16 -04:00
Benjamin Peterson
025e9ebd0a PEP 448: additional unpacking generalizations (closes #2292)
Patch by Neil Girdhar.
2015-05-05 20:16:41 -04:00
Serhiy Storchaka
009b811d67 Removed unintentional trailing spaces in non-external and non-generated C files. 2015-03-18 21:53:15 +02:00
Victor Stinner
f4c68db162 (Merge 3.3) parser: fix usage of Py_BuildValue() to build a parser error
Fix typo: "os" format => "Os"
2014-01-02 11:50:10 +01:00
Victor Stinner
5f8d485982 parser: fix usage of Py_BuildValue() to build a parser error
Fix typo: "os" format => "Os"
2014-01-02 11:49:27 +01:00
Victor Stinner
14e461d5b9 Close #11619: The parser and the import machinery do not encode Unicode
filenames anymore on Windows.
2013-08-26 22:28:21 +02:00
Victor Stinner
6684bdf73d Issue #18408: Fix typo in build_node_tree() of the parser module
Type "o" format of Py_BuildValue() is invalid: it must be "O".
2013-07-17 00:13:52 +02:00
Victor Stinner
df4572cc71 Issue #18408: parser module: fix error handling in node2tuple()
Handle PyLong_FromLong() and PyUnicode_FromString() failures
2013-07-12 01:35:10 +02:00
Victor Stinner
3bd6abd129 Issue #18408: Fix parser.sequence2st() and parser.tuple2st(): raise MemoryError
on memory allocation failure

Instead of ignoring the memory allocation failure and create invalid objects.
2013-07-12 01:33:59 +02:00