re.match(r'^ *', ...) can't fail, but mypy doesn't know that and
complains:
scripts/qapi/parser.py:444: error: Item "None" of "Match[str] | None" has no attribute "end" [union-attr]
Work around by using must_match() instead.
Fixes: 8107ba47fd (qapi: Add documentation format validation)
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
Message-ID: <20251105152219.311154-1-armbru@redhat.com>
Add explicit validation for QAPI documentation formatting rules:
1. Lines must not exceed 70 columns in width (including '# ' prefix)
2. Sentences must be separated by two spaces
Example sections and literal :: blocks (seldom case) are excluded, we
don't require them to be <= 70, that would be too restrictive. Anyway,
they share common 80-columns recommendations (not requirements).
Add two simple tests, illustrating the change.
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@yandex-team.ru>
Message-ID: <20251031183129.246814-1-vsementsov@yandex-team.ru>
The detection of example and literal blocks isn't quite correct, but
it works well enough, and we can improve on top.
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Comments, error messages, and test file names tweaked]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Generated command documentation lacks information on return value in
several cases, e.g. query-tpm.
The obvious fix would be to require a Returns: section when a command
returns something.
However, note that many existing Returns: sections are pretty useless:
the description is basically the return type, which then gets rendered
like "Return: <Type> – <basically the return type>". This suggests
that a description is often not really necessary, and requiring one
isn't useful.
Instead, generate the obvious minimal thing when Returns: is absent:
"Return: <Type>".
This auto-generated Return documentation is placed is as follows:
1. If we have arguments, return goes right after them.
2. Else if we have errors, return goes right before them.
3. Else if we have features, return goes right before them.
4. Else return goes right after the intro
To facilitate this algorithm, a "TODO:" hack line is used to separate
the intro from the remainder of the documentation block in cases where
there are no other sections to separate the intro from e.g. examples and
additional detail meant to appear below the key sections of interest.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250711051045.51110-3-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[_insert_near_kind() code replaced by something simpler, commit
message amended to explain why we're doing this]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
We reject lines starting with '=' in definition documentation. This
made sense when such lines were headings in free-form documentation,
but not in definition documentation.
Before the previous commit, lines starting with '=' were headings in
free-form documentation, and rejected in definition documentation,
where such headings could not work.
The previous commit dropped the headings feature from free-form
documentation, because we can simply use plain rST headings.
Rejecting them in definition documentation no longer makes sense, so
drop that, too.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250618165353.1980365-6-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Amend commit message to explain why]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Remove the QAPI doc section heading syntax, use plain rST section
headings instead.
Tests and documentation are updated to match.
Interestingly, Plain rST headings work fine before this patch, except
for over- and underlining with '=', which the doc parser rejected as
invalid QAPI doc section heading in free-form comments.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250618165353.1980365-5-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Add more detail to commit message]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Parser and doc generator cooperate on generating stub documentation for
undocumented members. The parser makes up an ArgSection with an empty
description, and the doc generator makes up a description.
Right now, the made-up ArgSections go into doc.args. However, the new
doc generator uses .all_sections, not .args. So put them into
.all_sections, too.
Insert them right after existing 'member' sections. If there are none,
insert directly after the leading section.
Doesn't affect the old generator, because that one doesn't use
.all_sections.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250311034303.75779-60-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Commit message rewritten]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
We have several kinds of sections, and to tell them apart, we use
Section attribute @tag and also the section object's Python type:
type @tag
untagged Section None
@foo: ArgSection 'foo'
Returns: Section 'Returns'
Errors: Section 'Errors'
Since: Section 'Since'
TODO: Section 'TODO'
Note:
* @foo can be a member or a feature description, depending on context.
* tag == 'Since' can be a Since: section or a member or feature
description. If it's a Section, it's the former, and if it's an
ArgSection, it's the latter.
Clean this up as follows. Move the member or feature name to new
ArgSection attribute @name, and replace @tag by enum @kind like this:
type kind name
untagged Section PLAIN
@foo: ArgSection MEMBER 'foo' if member or argument
ArgSection FEATURE 'foo' if feature
Returns: Section RETURNS
Errors: Section ERRORS
Since: Section SINCE
TODO: Section TODO
The qapi-schema tests are updated to account for the new section names;
"TODO" becomes "Todo" and `None` becomes "Plain" there.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250311034303.75779-34-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Instead of using the info object for the doc block as a whole (which
always points to the very first line of the block), update the info
pointer for each call to ensure_untagged_section when the existing
section is otherwise empty. This way, Sphinx error information will
match precisely to where the text actually starts.
For example, this patch will move the info pointer for the "Hello!"
untagged section ...
> ## <-- from here ...
> # Hello! <-- ... to here.
> ##
This doesn't seem to improve error reporting now. It will with the
forthcoming QAPI doc transmogrifier.
If I stick bad rST into qapi/block-core.json like this:
> ##
> # @SnapshotInfo:
> #
> +# rST syntax error: *ahh!
> +#
> # @id: unique shapshot id
> #
> # @name: user chosen name
The existing code's error message will point to the beginning of the doc
comment, which is less than helpful. The transmogrifier's message will
point to the erroneous line, but to accomplish this, it needs this
patch.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250311034303.75779-33-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
We use OrderedDict to ensure dictionary order is insertion order.
Plain dict does that since Python 3.6, but it wasn't guaranteed until
3.7. Since we have 3.7 now, replace OrderedDict by dict.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20250227080757.3978333-3-armbru@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Fully eliminate the "Example" sections in QAPI doc blocks now that they
have all been converted to arbitrary rST syntax using the
".. qmp-example::" directive. Update tests to match.
Migrating to the new syntax
---------------------------
The old "Example:" or "Examples:" section syntax is now caught as an
error, but "Example::" is stil permitted as explicit rST syntax for an
un-lexed, generic preformatted text block.
('Example' is not special in this case, any sentence that ends with "::"
will start an indented code block in rST.)
Arbitrary rST for Examples is now possible, but it's strongly
recommended that documentation authors use the ".. qmp-example::"
directive for consistent visual formatting in rendered HTML docs. The
":title:" directive option may be used to add extra information into the
title bar for the example. The ":annotated:" option can be used to write
arbitrary rST instead, with nested "::" blocks applying QMP formatting
where desired.
Other choices available are ".. code-block:: QMP" which will not create
an "Example:" box, or the short-form "::" code-block syntax which will
not apply QMP highlighting when used outside of the qmp-example
directive.
Why?
----
This patch has several benefits:
1. Example sections can now be written more arbitrarily, mixing
explanatory paragraphs and code blocks however desired.
2. Example sections can now use fully arbitrary rST.
3. All code blocks are now lexed and validated as QMP; increasing
usability of the docs and ensuring validity of example snippets.
(To some extent - This patch only gaurantees it lexes correctly, not
that it's valid under the JSON or QMP grammars. It will catch most
small mistakes, however.)
4. Each qmp-example can be titled or annotated independently without
bypassing the QMP lexer/validator.
(i.e. code blocks are now for *code* only, so we don't have to
sacrifice exposition for having lexically valid examples.)
NOTE: As with the "Notes" conversion (d461c27973), this patch (and the
three preceding) may change the rendering order for Examples in
the current generator. The forthcoming qapidoc rewrite will fix
this by always generating documentation in source order.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20240717021312.606116-10-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
The double-colon synax is rST formatting that precedes a literal code
block. We do not want to capture these as QAPI-specific sections.
Coerce blocks that start with e.g. "Example::" to be parsed as untagged
paragraphs instead of special tagged sections.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20240626222128.406106-14-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Indentation tweaked for consistency]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
We do not need a dedicated section for notes. By eliminating a specially
parsed section, these notes can be treated as normal rST paragraphs in
the new QMP reference manual, and can be placed and styled much more
flexibly.
Convert all existing "Note" and "Notes" sections to pure rST. As part of
the conversion, capitalize the first letter of each sentence and add
trailing punctuation where appropriate to ensure notes look sensible and
consistent in rendered HTML documentation. Markup is also re-aligned to
the de-facto standard of 3 spaces for directives.
Update docs/devel/qapi-code-gen.rst to reflect the new paradigm, and
update the QAPI parser to prohibit "Note" sections while suggesting a
new syntax. The exact formatting to use is a matter of taste, but a good
candidate is simply:
.. note:: lorem ipsum ...
... dolor sit amet ...
... consectetur adipiscing elit ...
... but there are other choices, too. The Sphinx readthedocs theme
offers theming for the following forms (capitalization unimportant); all
are adorned with a (!) symbol () in the title bar for rendered HTML
docs.
See
https://sphinx-rtd-theme.readthedocs.io/en/stable/demo/demo.html#admonitions
for examples of each directive/admonition in use.
These are rendered in orange:
.. Attention:: ...
.. Caution:: ...
.. WARNING:: ...
These are rendered in red:
.. DANGER:: ...
.. Error:: ...
These are rendered in green:
.. Hint:: ...
.. Important:: ...
.. Tip:: ...
These are rendered in blue:
.. Note:: ...
.. admonition:: custom title
admonition body text
This patch uses ".. note::" almost everywhere, with just two "caution"
directives. Several instances of "Notes:" have been converted to
merely ".. note::", or multiple ".. note::" where appropriate.
".. admonition:: notes" is used in a few places where we had an
ordered list of multiple notes that would not make sense as
standalone/separate admonitions. Two "Note:" following "Example:"
have been turned into ordinary paragraphs within the example.
NOTE: Because qapidoc.py does not attempt to preserve source ordering of
sections, the conversion of Notes from a "tagged section" to an
"untagged section" means that rendering order for some notes *may
change* as a result of this patch. The forthcoming qapidoc.py rewrite
strictly preserves source ordering in the rendered documentation, so
this issue will be rectified in the new generator.
Signed-off-by: John Snow <jsnow@redhat.com>
Acked-by: Stefan Hajnoczi <stefanha@redhat.com> [for block*.json]
Message-ID: <20240626222128.406106-11-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Commit message clarified slightly, period added to one more note]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
If a comment immediately follows a doc block, the parser doesn't ignore
that token appropriately. Fix that.
e.g.
> ##
> # = Hello World!
> ##
>
> # I'm a comment!
will break the parser, because it does not properly ignore the comment
token if it immediately follows a doc block.
Fixes: 3d035cd2cc (qapi: Rewrite doc comment parser)
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240626222128.406106-7-jsnow@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Change get_doc_indented() to preserve indentation on all subsequent text
lines, and create a compatibility dedent() function for qapidoc.py that
removes indentation the same way get_doc_indented() did.
This is being done for the benefit of a new qapidoc generator which
requires that indentation in argument and features sections are
preserved.
Prior to this patch, a section like this:
```
@name: lorem ipsum
dolor sit amet
consectetur adipiscing elit
```
would have its body text be parsed into:
```
lorem ipsum
dolor sit amet
consectetur adipiscing elit
```
We want to preserve the indentation for even the first body line so that
the entire block can be parsed directly as rST. This patch would now
parse that segment into:
```
lorem ipsum
dolor sit amet
consectetur adipiscing elit
```
This is helpful for formatting arguments and features as field lists in
rST, where the new generator will format this information as:
```
:arg type name: lorem ipsum
dolor sit amet
consectetur apidiscing elit
```
...and can be formed by the simple concatenation of the field list
construct and the body text. The indents help preserve the continuation
of a block-level element, and further allow the use of additional rST
block-level constructs such as code blocks, lists, and other such
markup.
This understandably breaks the existing qapidoc.py; so a new function is
added there to dedent the text for compatibility. Once the new generator
is merged, this function will not be needed any longer and can be
dropped.
I verified this patch changes absolutely nothing by comparing the
md5sums of the QMP ref html pages both before and after the change, so
it's certified inert. QAPI test output has been updated to reflect the
new strategy of preserving indents for rST.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240626222128.406106-6-jsnow@redhat.com>
[Lost commit message paragraph restored]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Dict[str, object] is a stricter type, but with the way that code is
currently arranged, it is infeasible to enforce this strictness.
In particular, although expr.py's entire raison d'être is normalization
and type-checking of QAPI Expressions, that type information is not
"remembered" in any meaningful way by mypy because each individual
expression is not downcast to a specific expression type that holds all
the details of each expression's unique form.
As a result, all of the code in schema.py that deals with actually
creating type-safe specialized structures has no guarantee (myopically)
that the data it is being passed is correct.
There are two ways to solve this:
(1) Re-assert that the incoming data is in the shape we expect it to be, or
(2) Disable type checking for this data.
(1) is appealing to my sense of strictness, but I gotta concede that it
is asinine to re-check the shape of a QAPIExpression in schema.py when
expr.py has just completed that work at length. The duplication of code
and the nightmare thought of needing to update both locations if and
when we change the shape of these structures makes me extremely
reluctant to go down this route.
(2) allows us the chance to miss updating types in the case that types
are updated in expr.py, but it *is* an awful lot simpler and,
importantly, gets us closer to type checking schema.py *at
all*. Something is better than nothing, I'd argue.
So, do the simpler dumber thing and worry about future strictness
improvements later.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240315152301.3621858-20-armbru@redhat.com>
We use section "Returns" for documenting both success and error
response of commands.
I intend to generate better command success response documentation.
Easier when "Returns" documents just he success response.
Create new section tag "Errors". The next two commits will move error
response documentation from "Returns" sections to "Errors" sections.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240227113921.236097-4-armbru@redhat.com>
QAPIDoc stores a reference to QAPIParser just to pass it to
QAPIParseError. The resulting error position depends on the state of
the parser. It happens to be the current comment line. Servicable,
but action at a distance.
The commit before previous moved most uses of QAPIParseError from
QAPIDoc to QAPIParser. There are just three left. Convert them to
QAPISemError. This involves passing info to a few methods. Then drop
the reference to QAPIParser.
The three errors lose the column number. Not really interesting here:
it's the comment line's indentation.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20240216145841.2099240-17-armbru@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
