Commit Graph

295 Commits

Author SHA1 Message Date
Ivan Levkivskyi
9932a22897 bpo-33416: Add end positions to Python AST (GH-11605)
The majority of this PR is tediously passing `end_lineno` and `end_col_offset` everywhere. Here are non-trivial points:
* It is not possible to reconstruct end positions in AST "on the fly", some information is lost after an AST node is constructed, so we need two more attributes for every AST node `end_lineno` and `end_col_offset`.
* I add end position information to both CST and AST.  Although it may be technically possible to avoid adding end positions to CST, the code becomes more cumbersome and less efficient.
* Since the end position is not known for non-leaf CST nodes while the next token is added, this requires a bit of extra care (see `_PyNode_FinalizeEndPos`). Unless I made some mistake, the algorithm should be linear.
* For statements, I "trim" the end position of suites to not include the terminal newlines and dedent (this seems to be what people would expect), for example in
  ```python
  class C:
      pass

  pass
  ```
  the end line and end column for the class definition is (2, 8).
* For `end_col_offset` I use the common Python convention for indexing, for example for `pass` the `end_col_offset` is 4 (not 3), so that `[0:4]` gives one the source code that corresponds to the node.
* I added a helper function `ast.get_source_segment()`, to get source text segment corresponding to a given AST node. It is also useful for testing.

An (inevitable) downside of this PR is that AST now takes almost 25% more memory. I think however it is probably justified by the benefits.
2019-01-22 11:18:22 +00:00
Anthony Sottile
995d9b9297 bpo-16806: Fix lineno and col_offset for multi-line string tokens (GH-10021) 2019-01-13 13:05:13 +09:00
Serhiy Storchaka
58159ef856 bpo-35494: Improve syntax error messages for unbalanced parentheses in f-string. (GH-11161) 2019-01-12 09:46:50 +02:00
Zackery Spytz
4c49da0cb7 bpo-35436: Add missing PyErr_NoMemory() calls and other minor bug fixes. (GH-11015)
Set MemoryError when appropriate, add missing failure checks,
and fix some potential leaks.
2018-12-07 12:11:30 +02:00
Serhiy Storchaka
b619b09792 bpo-31241: Fix AST node position for list and generator comprehensions. (GH-10633)
The lineno and col_offset attributes of AST nodes for list comprehensions,
generator expressions and tuples are now point to the opening parenthesis or
square brace. For tuples without parenthesis they point to the position
of the first item.
2018-11-27 09:40:29 +02:00
Victor Stinner
4d73ae7761 bpo-18407: ast.c uses Py_ssize_t for asdl_seq_LEN() iterator (GH-10655)
When iterating using asdl_seq_LEN(), use 'Py_ssize_t' type instead of
'int' for the iterator variable, to avoid downcast on 64-bit platforms.

_Py_asdl_int_seq_new() now also ensures that the index is greater than
or equal to 0.
2018-11-22 14:45:16 +01:00
Serhiy Storchaka
97f1efb606 bpo-35169: Improve error messages for forbidden assignments. (GH-10342) 2018-11-20 19:27:16 +02:00
Serhiy Storchaka
95b6acf951 bpo-34876: Change the lineno of the AST for decorated function and class. (GH-9731)
It was overridden by the lineno of the first decorator. Now it is
the lineno of 'def' or 'class'.
2018-10-30 13:16:02 +02:00
Serhiy Storchaka
6543912c90 bpo-32912: Replace a DeprecationWarning with a SyntaxWarning (GH-9652)
for invalid escape sequences in string and bytes literals.
2018-10-19 17:42:06 +03:00
Serhiy Storchaka
3f22811fef bpo-32892: Use ast.Constant instead of specific constant AST types. (GH-9445) 2018-09-27 17:42:37 +03:00
Ammar Askar
025eb98dc0 bpo-34683: Make SyntaxError column offsets consistently 1-indexed (gh-9338)
Also point to start of tokens in parsing errors.

Fixes bpo-34683
2018-09-24 14:12:49 -07:00
Benjamin Peterson
c9a71dd223 closes bpo-34641: Further restrict the LHS of keyword argument function call syntax. (GH-9212) 2018-09-12 17:14:39 -07:00
Benjamin Peterson
d13e59c1b5 Make sure the line comes from the same node as the col offset. (GH-9189)
Followup to 90fc8980bb.

<!--
Thanks for your contribution!
Please read this comment in its entirety. It's quite important.

# Pull Request title

It should be in the following format:

```
bpo-NNNN: Summary of the changes made
```

Where: bpo-NNNN refers to the issue number in the https://bugs.python.org.

Most PRs will require an issue number. Trivial changes, like fixing a typo, do not need an issue.

# Backport Pull Request title

If this is a backport PR (PR made against branches other than `master`),
please ensure that the PR title is in the following format:

```
[X.Y] <title from the original PR> (GH-NNNN)
```

Where: [X.Y] is the branch name, e.g. [3.6].

GH-NNNN refers to the PR number from `master`.

-->
2018-09-11 15:29:57 -07:00
guoci
90fc8980bb closes bpo-31902: Fix the col_offset attribute for ast.Async* nodes to point to the "async" keyword. (GH-4175)
Previously, col_offset points to the keyword after "async".
2018-09-11 14:45:45 -07:00
Alexey Izbyshev
28853a249b bpo-34457: Python/ast.c: Add missing NULL check to alias_for_import_name(). (GH-8852)
Reported by Svace static analyzer.
2018-08-22 07:55:16 +03:00
Serhiy Storchaka
143ce5c6db bpo-33691: Add _PyAST_GetDocString(). (GH-7236) 2018-05-30 10:56:16 +03:00
Serhiy Storchaka
73cbe7a01a bpo-32911: Revert bpo-29463. (GH-7121) (GH-7197)
Remove the docstring attribute of AST types and restore docstring
expression as a first stmt in their body.

Co-authored-by: INADA Naoki <methane@users.noreply.github.com>
2018-05-29 12:04:55 +03:00
Jelle Zijlstra
898ff92dac fix error message in ast.c (#6776)
small_stmt -> compound_stmt
2018-05-13 17:04:53 -04:00
Victor Stinner
fb7e7992be bpo-30465: Fix C downcast warning on Windows in ast.c (#6593)
ast.c: fstring_fix_node_location() downcasts a pointer difference to
a C int. Replace int with Py_ssize_t to fix the compiler warning.
2018-04-30 14:51:02 -07:00
Serhiy Storchaka
a561862048 Don't hide unexpected errors in PyErr_WarnExplicitObject(). (#4585) 2017-12-01 08:40:23 +02:00
Serhiy Storchaka
ddbce13786 bpo-32023: Disallow genexprs without parenthesis in class definitions. (#4400) 2017-11-15 17:39:37 +02:00
Serhiy Storchaka
9165f77d5f bpo-32012: Disallow trailing comma after genexpr without parenthesis. (#4382) 2017-11-15 08:49:40 +02:00
luzpaz
a5293b4ff2 Fix miscellaneous typos (#4275) 2017-11-05 15:37:50 +02:00
Serhiy Storchaka
56cb465cc9 bpo-31825: Fixed OverflowError in the 'unicode-escape' codec (#4058)
and in codecs.escape_decode() when decode an escaped non-ascii byte.
2017-10-20 17:08:15 +03:00
Jelle Zijlstra
ac317700ce bpo-30406: Make async and await proper keywords (#1669)
Per PEP 492, 'async' and 'await' should become proper keywords in 3.7.
2017-10-05 23:24:46 -04:00