Generated command documentation lacks information on return value in
several cases, e.g. query-tpm.
The obvious fix would be to require a Returns: section when a command
returns something.
However, note that many existing Returns: sections are pretty useless:
the description is basically the return type, which then gets rendered
like "Return: <Type> – <basically the return type>". This suggests
that a description is often not really necessary, and requiring one
isn't useful.
Instead, generate the obvious minimal thing when Returns: is absent:
"Return: <Type>".
This auto-generated Return documentation is placed is as follows:
1. If we have arguments, return goes right after them.
2. Else if we have errors, return goes right before them.
3. Else if we have features, return goes right before them.
4. Else return goes right after the intro
To facilitate this algorithm, a "TODO:" hack line is used to separate
the intro from the remainder of the documentation block in cases where
there are no other sections to separate the intro from e.g. examples and
additional detail meant to appear below the key sections of interest.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250711051045.51110-3-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[_insert_near_kind() code replaced by something simpler, commit
message amended to explain why we're doing this]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
We reject lines starting with '=' in definition documentation. This
made sense when such lines were headings in free-form documentation,
but not in definition documentation.
Before the previous commit, lines starting with '=' were headings in
free-form documentation, and rejected in definition documentation,
where such headings could not work.
The previous commit dropped the headings feature from free-form
documentation, because we can simply use plain rST headings.
Rejecting them in definition documentation no longer makes sense, so
drop that, too.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250618165353.1980365-6-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Amend commit message to explain why]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Remove the QAPI doc section heading syntax, use plain rST section
headings instead.
Tests and documentation are updated to match.
Interestingly, Plain rST headings work fine before this patch, except
for over- and underlining with '=', which the doc parser rejected as
invalid QAPI doc section heading in free-form comments.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250618165353.1980365-5-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Add more detail to commit message]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Since the previous commit, python/setup.cfg applies to scripts/qapi/ as
well. Configuration files in scripts/qapi/ override python/setup.cfg.
scripts/qapi/.flake8 and scripts/qapi/.isort.cfg actually match
python/setup.cfg exactly, and can go.
The differences between scripts/qapi/mypy.ini and python/setup.cfg are
harmless: namespace_packages being set to True is a requirement for the
PEP420 nested package structure of QEMU but not for scripts/qapi, but
has no effect on type checking the QAPI code. warn_unused_ignores is
used in python/ to be able to target a wide variety of mypy versions;
some of which that have added new ignore categories that are not present
in older versions.
Ultimately, scripts/qapi/mypy.ini can be removed without any real change
in behavior to how mypy enforces type safety there.
The pylint config is being left in place because the settings differ
enough from the python/ directory settings that we need a chit-chat on
how to merge them O:-)
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-id: 20250604200354.459501-7-jsnow@redhat.com
Update the python tests to also check QAPI and the QAPI Sphinx
extensions. The docs/sphinx/qapidoc_legacy.py file is not included in
these checks, as it is destined for removal soon. mypy is also not
called on the QAPI Sphinx extensions, owing to difficulties supporting
Sphinx 3.x - 8.x while maintaining static type checking support. mypy
*is* called on all of the QAPI tools themselves, though.
flake8, isort and mypy use the tool configuration from the existing
python directory (in setup.cfg). pylint continues to use the special
configuration located in scripts/qapi/ - that configuration is more
permissive. If we wish to unify the two configurations, that's a
separate series and a discussion for a later date.
The list of pylint ignores is also updated, owing again to the wide
window of pylint version support: newer versions require pragmas to
occasionally silence the "too many positional arguments" warning, but
older versions do not have such a warning category and will instead yelp
about an unrecognized option. Silence that warning, too.
As a result of this patch, one would be able to run any of the following
tests locally from the qemu.git/python directory and have it cover the
QAPI tooling as well. All of the following options run the python tests,
static analysis tests, and linter checks; but with different
combinations of dependencies and interpreters.
- "make check-minreqs" Run tests specifically under our oldest supported
Python and our oldest supported dependencies. This is the test that
runs on GitLab as "check-python-minreqs". This helps ensure we do not
regress support on older platforms accidentally.
- "make check-tox" Runs the tests under the newest supported
dependencies, but under each supported version of Python in turn. At
time of writing, this is Python 3.8 to 3.13 inclusive. This test helps
catch bleeding-edge problems before they become problems for developer
workstations. This is the GitLab test "check-python-tox" and is an
optionally run, may-fail test due to the unpredictable nature of new
dependencies being released into the ecosystem that may cause
regressions.
- "make check-dev" Runs the tests under the newest supported
dependencies using whatever version of Python the user happens to have
installed. This is a quick convenience check that does not map to any
particular GitLab test.
(Note! check-dev may be busted on Fedora 41 and bleeding edge versions
of setuptools. That's unrelated to this patch and I'll address it
separately and soon. Thank you for your patience, --mgmt)
Finally, finally, finally: this means that QAPI tooling will be linted
and type-checked from the GitLab pipelines.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-id: 20250604200354.459501-5-jsnow@redhat.com
[Edited license choice per review --js]
Signed-off-by: John Snow <jsnow@redhat.com>
Python 3.8 went "end of life" in October 2024 and Fedora 42 dropped
this version already, so the "python" CI job is currently failing.
Thus it's time to drop support for this Python version in QEMU, too.
While we're at it, also look for "python3.13" in the configure script.
Message-ID: <20250425120710.879518-1-thuth@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
create_backend()'s caller catches QAPIError, and returns non-zero exit
code on catch. The caller's caller passes the exit code to
sys.exit().
create_backend() doesn't care: it reports errors to stderr and
sys.exit()s.
Change it to raise QAPIError instead.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20250311065352.992307-1-armbru@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Parser and doc generator cooperate on generating stub documentation for
undocumented members. The parser makes up an ArgSection with an empty
description, and the doc generator makes up a description.
Right now, the made-up ArgSections go into doc.args. However, the new
doc generator uses .all_sections, not .args. So put them into
.all_sections, too.
Insert them right after existing 'member' sections. If there are none,
insert directly after the leading section.
Doesn't affect the old generator, because that one doesn't use
.all_sections.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250311034303.75779-60-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Commit message rewritten]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
This is for the sake of the new rST generator (the "transmogrifier") so
we can advance multiple lines on occasion while keeping the
generated<-->source mappings accurate.
next_line now simply takes an optional n parameter which chooses the
number of lines to advance.
The next patch will use this when converting section syntax in free-form
documentation to more traditional rST section header syntax, which does
not always line up 1:1 for line counts.
For example:
```
##
# = Section <-- Info is pointing here, "L1"
#
# Lorem Ipsum
##
```
would be transformed to rST as:
```
======= <-- L1
Section <-- L1
======= <-- L1
<-- L2
Lorem Ipsum <-- L3
```
After consuming the single "Section" line from the source, we want to
advance the source pointer to the next non-empty line which requires
jumping by more than one line.
Signed-off-by: John Snow <jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20250311034303.75779-42-jsnow@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
We have several kinds of sections, and to tell them apart, we use
Section attribute @tag and also the section object's Python type:
type @tag
untagged Section None
@foo: ArgSection 'foo'
Returns: Section 'Returns'
Errors: Section 'Errors'
Since: Section 'Since'
TODO: Section 'TODO'
Note:
* @foo can be a member or a feature description, depending on context.
* tag == 'Since' can be a Since: section or a member or feature
description. If it's a Section, it's the former, and if it's an
ArgSection, it's the latter.
Clean this up as follows. Move the member or feature name to new
ArgSection attribute @name, and replace @tag by enum @kind like this:
type kind name
untagged Section PLAIN
@foo: ArgSection MEMBER 'foo' if member or argument
ArgSection FEATURE 'foo' if feature
Returns: Section RETURNS
Errors: Section ERRORS
Since: Section SINCE
TODO: Section TODO
The qapi-schema tests are updated to account for the new section names;
"TODO" becomes "Todo" and `None` becomes "Plain" there.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250311034303.75779-34-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Instead of using the info object for the doc block as a whole (which
always points to the very first line of the block), update the info
pointer for each call to ensure_untagged_section when the existing
section is otherwise empty. This way, Sphinx error information will
match precisely to where the text actually starts.
For example, this patch will move the info pointer for the "Hello!"
untagged section ...
> ## <-- from here ...
> # Hello! <-- ... to here.
> ##
This doesn't seem to improve error reporting now. It will with the
forthcoming QAPI doc transmogrifier.
If I stick bad rST into qapi/block-core.json like this:
> ##
> # @SnapshotInfo:
> #
> +# rST syntax error: *ahh!
> +#
> # @id: unique shapshot id
> #
> # @name: user chosen name
The existing code's error message will point to the beginning of the doc
comment, which is less than helpful. The transmogrifier's message will
point to the erroneous line, but to accomplish this, it needs this
patch.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250311034303.75779-33-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
A TODO comment in class Annotated reminds us to simplify it once we
can use @dataclass, new in Python 3.7. We have that now, so do it.
There's a similar comment in scripts/qapi/source.py, but I can't
figure out how to use @dataclass there. Left for another day.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20250227080757.3978333-4-armbru@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
We use OrderedDict to ensure dictionary order is insertion order.
Plain dict does that since Python 3.6, but it wasn't guaranteed until
3.7. Since we have 3.7 now, replace OrderedDict by dict.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20250227080757.3978333-3-armbru@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
The 'qapi.backend.QAPIBackend' class defines an API contract for code
generators. The current generator is put into a new class
'qapi.backend.QAPICBackend' and made to be the default impl.
A custom generator can be requested using the '-k' arg which takes a
fully qualified python class name
qapi-gen.py -B the.python.module.QAPIMyBackend
This allows out of tree code to use the QAPI generator infrastructure
to create new language bindings for QAPI schemas. This has the caveat
that the QAPI generator APIs are not guaranteed stable, so consumers
of this feature may have to update their code to be compatible with
future QEMU releases.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20250224182030.2089959-1-berrange@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Error checking and messages tweaked]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
If you've got a newer pylint, it'll whine about positional arguments
separately from the regular ones. Update the configuration to ignore
both categories of warning.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250224033741.222749-2-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
This replaces use of the constants from the QapiSpecialFeatures
enum, with constants from the auto-generate QapiFeatures enum
in qapi-features.h
The 'deprecated' and 'unstable' features still have a little bit of
special handling, being force defined to be the 1st + 2nd features
in the enum, regardless of whether they're used in the schema. This
retains compatibility with common code that references the features
via the QapiSpecialFeatures constants.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20250205123550.2754387-5-berrange@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Imports tidied up with isort]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
This updates the QAPI code generation to refer to 'features' instead
of 'special_features', in preparation for generalizing their exposure.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20250205123550.2754387-4-berrange@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Imports tidied up with isort]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
The "special_features" field / parameter holds the subset of schema
features that are for internal code use. Specifically 'DEPRECATED'
and 'UNSTABLE'.
This special casing of internal features is going to be removed, so
prepare for that by renaming to 'features'. Using a fixed size type
is also best practice for bit fields.
Reviewed-by: Philippe Mathieu-Daudé <philmd@linaro.org>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20250205123550.2754387-3-berrange@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
The general expectation is that header files should follow the same
file/path naming scheme as the corresponding source file. There are
various historical exceptions to this practice in QEMU, with one of
the most notable being the include/qapi/qmp/ directory.
include/qapi/qmp/dispatch.h corresponds mostly to qapi/qmp-registry.c.
Move and rename it to include/qapi/qmp-registry.h.
Now just qerror.h is left in include/qapi/qmp/. Since it's deprecated
& (slowly) getting eliminated anyway, it isn't worth moving.
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Zhao Liu <zhao1.liu@intel.com>
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20241118151235.2665921-3-armbru@redhat.com>
The general expectation is that header files should follow the same
file/path naming scheme as the corresponding source file. There are
various historical exceptions to this practice in QEMU, with one of
the most notable being the include/qapi/qmp/ directory. Most of the
headers there correspond to source files in qobject/.
This patch corrects most of that inconsistency by creating
include/qobject/ and moving the headers for qobject/ there.
This also fixes MAINTAINERS for include/qapi/qmp/dispatch.h:
scripts/get_maintainer.pl now reports "QAPI" instead of "No
maintainers found".
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Zhao Liu <zhao1.liu@intel.com>
Acked-by: Halil Pasic <pasic@linux.ibm.com> #s390x
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Message-ID: <20241118151235.2665921-2-armbru@redhat.com>
[Rebased]
camel_to_upper() converts its argument from camel case to upper case
with '_' between words. Used for generated enumeration constant
prefixes.
When some of the words are spelled all caps, where exactly to insert
'_' is guesswork. camel_to_upper()'s guesses are bad enough in places
to make people override them with a 'prefix' in the schema.
Rewrite it to guess better:
1. Insert '_' after a non-upper case character followed by an upper
case character:
OneTwo -> ONE_TWO
One2Three -> ONE2_THREE
2. Insert '_' before the last upper case character followed by a
non-upper case character:
ACRONYMWord -> ACRONYM_Word
Except at the beginning (as in OneTwo above), or when there is
already one:
AbCd -> AB_CD
This changes the default enumeration constant prefix for a number of
enums. Generated enumeration constants change only where the default
is not overridden with 'prefix'.
The following enumerations without a 'prefix' change:
enum old camel_to_upper()
new camel_to_upper()
------------------------------------------------------------------
DisplayGLMode DISPLAYGL_MODE
DISPLAY_GL_MODE
EbpfProgramID EBPF_PROGRAMID
EBPF_PROGRAM_ID
HmatLBDataType HMATLB_DATA_TYPE
HMAT_LB_DATA_TYPE
HmatLBMemoryHierarchy HMATLB_MEMORY_HIERARCHY
HMAT_LB_MEMORY_HIERARCHY
MultiFDCompression MULTIFD_COMPRESSION
MULTI_FD_COMPRESSION
OffAutoPCIBAR OFF_AUTOPCIBAR
OFF_AUTO_PCIBAR
QCryptoBlockFormat Q_CRYPTO_BLOCK_FORMAT
QCRYPTO_BLOCK_FORMAT
QCryptoBlockLUKSKeyslotState Q_CRYPTO_BLOCKLUKS_KEYSLOT_STATE
QCRYPTO_BLOCK_LUKS_KEYSLOT_STATE
QKeyCode Q_KEY_CODE
QKEY_CODE
XDbgBlockGraphNodeType X_DBG_BLOCK_GRAPH_NODE_TYPE
XDBG_BLOCK_GRAPH_NODE_TYPE
TestUnionEnumA TEST_UNION_ENUMA
TEST_UNION_ENUM_A
Add a 'prefix' so generated code doesn't change now. Subsequent
commits will remove most of them again. Two will remain:
MULTIFD_COMPRESSION, because migration code generally spells "multifd"
that way, and Q_KEY_CODE, because that one is baked into
subprojects/keycodemapdb/tools/keymap-gen.
The following enumerations with a 'prefix' change so that the prefix
is now superfluous:
enum old camel_to_upper()
new camel_to_upper() [equal to prefix]
------------------------------------------------------------------
BlkdebugIOType BLKDEBUGIO_TYPE
BLKDEBUG_IO_TYPE
QCryptoTLSCredsEndpoint Q_CRYPTOTLS_CREDS_ENDPOINT
QCRYPTO_TLS_CREDS_ENDPOINT
QCryptoSecretFormat Q_CRYPTO_SECRET_FORMAT
QCRYPTO_SECRET_FORMAT
QCryptoCipherMode Q_CRYPTO_CIPHER_MODE
QCRYPTO_CIPHER_MODE
QCryptodevBackendType Q_CRYPTODEV_BACKEND_TYPE
QCRYPTODEV_BACKEND_TYPE
QType [builtin] Q_TYPE
QTYPE
Drop these prefixes.
The following enumerations with a 'prefix' change without making the
'prefix' superfluous:
enum old camel_to_upper()
new camel_to_upper() [equal to prefix]
prefix
------------------------------------------------------------------
CpuS390Entitlement CPUS390_ENTITLEMENT
CPU_S390_ENTITLEMENT
S390_CPU_ENTITLEMENT
CpuS390Polarization CPUS390_POLARIZATION
CPU_S390_POLARIZATION
S390_CPU_POLARIZATION
CpuS390State CPUS390_STATE
CPU_S390_STATE
S390_CPU_STATE
QAuthZListFormat Q_AUTHZ_LIST_FORMAT
QAUTH_Z_LIST_FORMAT
QAUTHZ_LIST_FORMAT
QAuthZListPolicy Q_AUTHZ_LIST_POLICY
QAUTH_Z_LIST_POLICY
QAUTHZ_LIST_POLICY
QCryptoAkCipherAlgorithm Q_CRYPTO_AK_CIPHER_ALGORITHM
QCRYPTO_AK_CIPHER_ALGORITHM
QCRYPTO_AKCIPHER_ALG
QCryptoAkCipherKeyType Q_CRYPTO_AK_CIPHER_KEY_TYPE
QCRYPTO_AK_CIPHER_KEY_TYPE
QCRYPTO_AKCIPHER_KEY_TYPE
QCryptoCipherAlgorithm Q_CRYPTO_CIPHER_ALGORITHM
QCRYPTO_CIPHER_ALGORITHM
QCRYPTO_CIPHER_ALG
QCryptoHashAlgorithm Q_CRYPTO_HASH_ALGORITHM
QCRYPTO_HASH_ALGORITHM
QCRYPTO_HASH_ALG
QCryptoIVGenAlgorithm Q_CRYPTOIV_GEN_ALGORITHM
QCRYPTO_IV_GEN_ALGORITHM
QCRYPTO_IVGEN_ALG
QCryptoRSAPaddingAlgorithm Q_CRYPTORSA_PADDING_ALGORITHM
QCRYPTO_RSA_PADDING_ALGORITHM
QCRYPTO_RSA_PADDING_ALG
QCryptodevBackendAlgType Q_CRYPTODEV_BACKEND_ALG_TYPE
QCRYPTODEV_BACKEND_ALG_TYPE
QCRYPTODEV_BACKEND_ALG
QCryptodevBackendServiceType Q_CRYPTODEV_BACKEND_SERVICE_TYPE
QCRYPTODEV_BACKEND_SERVICE_TYPE
QCRYPTODEV_BACKEND_SERVICE
Subsequent commits will tweak things to remove most of these prefixes.
Only QAUTHZ_LIST_FORMAT and QAUTHZ_LIST_POLICY will remain.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20240904111836.3273842-2-armbru@redhat.com>
