Note that a reference to MIGRATION needs to be disambiguated with a
:qapi:event: prefix. Without this, Sphinx complains
more than one target found for 'any' cross-reference 'MIGRATION': could be :std:ref:`Migration framework` or :qapi:event:`QMP:migration.MIGRATION`
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250711054005.60969-9-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Commit message amended to explain need for :qapi:event:]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
The new qapidoc.py can generate "Returns" statements with type
information just fine, so we can remove it from the source where it
doesn't add anything particularly novel or helpful and just repeats the
type info.
This patch is fairly "gentle" and doesn't aggressively touch other
"Returns" lines that could be rephrased to omit repeating type
information; it only removes lines that appear appropriate to wholly
remove.
To help facilitate auto-generated placement, a few doc blocks have a
"TODO:" line inserted to help the placement algorithm differentiate the
introductory paragraph(s) from the rest of the documentation.
The auto-generated returns are in the exact same spot, except for
query-migrationthreads, query-machines, and
x-query-virtio-queue-element. These auto-generation moves to a better
spot.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250711051045.51110-4-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Two more Returns: removed, commit message amended to explain
auto-generated returns generated into a different spot]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Remove the QAPI doc section heading syntax, use plain rST section
headings instead.
Tests and documentation are updated to match.
Interestingly, Plain rST headings work fine before this patch, except
for over- and underlining with '=', which the doc parser rejected as
invalid QAPI doc section heading in free-form comments.
Signed-off-by: John Snow <jsnow@redhat.com>
Message-ID: <20250618165353.1980365-5-jsnow@redhat.com>
Reviewed-by: Markus Armbruster <armbru@redhat.com>
[Add more detail to commit message]
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Add the latency distribution too for blocktime, using order-of-two buckets.
It accounts for all the faults, from either vCPU or non-vCPU threads. With
prior rework, it's very easy to achieve by adding an array to account for
faults in each buckets.
Sample output for HMP (while for QMP it's simply an array):
Postcopy Latency Distribution:
[ 1 us - 2 us ]: 0
[ 2 us - 4 us ]: 0
[ 4 us - 8 us ]: 1
[ 8 us - 16 us ]: 2
[ 16 us - 32 us ]: 2
[ 32 us - 64 us ]: 3
[ 64 us - 128 us ]: 10169
[ 128 us - 256 us ]: 50151
[ 256 us - 512 us ]: 12876
[ 512 us - 1 ms ]: 97
[ 1 ms - 2 ms ]: 42
[ 2 ms - 4 ms ]: 44
[ 4 ms - 8 ms ]: 93
[ 8 ms - 16 ms ]: 138
[ 16 ms - 32 ms ]: 0
[ 32 ms - 65 ms ]: 0
[ 65 ms - 131 ms ]: 0
[ 131 ms - 262 ms ]: 0
[ 262 ms - 524 ms ]: 0
[ 524 ms - 1 sec ]: 0
[ 1 sec - 2 sec ]: 0
[ 2 sec - 4 sec ]: 0
[ 4 sec - 8 sec ]: 0
[ 8 sec - 16 sec ]: 0
Cc: Markus Armbruster <armbru@redhat.com>
Acked-by: Dr. David Alan Gilbert <dave@treblig.org>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Link: https://lore.kernel.org/r/20250613141217.474825-15-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
When used to report page fault latencies, the blocktime feature can be
almost useless when KVM async page fault is enabled, because in most cases
such remote fault will kickoff async page faults, then it's not trackable
from blocktime layer.
After all these recent rewrites to blocktime layer, it's finally so easy to
also support tracking non-vCPU faults. It'll be even faster if we could
always index fault records with TIDs, unfortunately we need to maintain the
blocktime API which report things in vCPU indexes.
Of course this can work not only for kworkers, but also any guest accesses
that may reach a missing page, for example, very likely when in the QEMU
main thread too (and all other threads whenever applicable).
In this case, we don't care about "how long the threads are blocked", but
we only care about "how long the fault will be resolved".
Cc: Markus Armbruster <armbru@redhat.com>
Cc: Dr. David Alan Gilbert <dave@treblig.org>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Tested-by: Mario Casquero <mcasquer@redhat.com>
Link: https://lore.kernel.org/r/20250613141217.474825-14-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Blocktime so far only cares about the time one vcpu (or the whole system)
got blocked. It would be also be helpful if it can also report the latency
of page requests, which could be very sensitive during postcopy.
Blocktime itself is sometimes not very important, especially when one
thinks about KVM async PF support, which means vCPUs are literally almost
not blocked at all because the guest OS is smart enough to switch to
another task when a remote fault is needed.
However, latency is still sensitive and important because even if the guest
vCPU is running on threads that do not need a remote fault, the workload
that accesses some missing page is still affected.
Add two entries to the report, showing how long it takes to resolve a
remote fault. Mention in the QAPI doc that this is not the real average
fault latency, but only the ones that was requested for a remote fault.
Unwrap get_vcpu_blocktime_list() so we don't need to walk the list twice,
meanwhile add the entry checks in qtests for all postcopy tests.
Cc: Markus Armbruster <armbru@redhat.com>
Cc: Dr. David Alan Gilbert <dave@treblig.org>
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Tested-by: Mario Casquero <mcasquer@redhat.com>
Link: https://lore.kernel.org/r/20250613141217.474825-9-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>
Signed-off-by: Fabiano Rosas <farosas@suse.de>
DEVICE state was introduced back in 2017:
https://lore.kernel.org/qemu-devel/20171020090556.18631-1-dgilbert@redhat.com/
Quote from Dave's cover letter, when the pre-switchover phase was enabled,
the state transition looks like this:
The precopy flow is:
active->pre-switchover->device->completed
The postcopy flow is:
active->pre-switchover->postcopy-active->completed
To supplement above, when the cap is not enabled:
The precopy flow is:
active->completed
The postcopy flow is:
active->postcopy-active->completed
It works for us, though we have some code just to special case these state
transitions, so the DEVICE state currently is special only to precopy, and
only conditionally.
I had a quick discussion with Libvirt developers, it turns out that this
may not be necessary. IOW, it seems okay we can have DEVICE state to be
generic, so that we don't have over-complicated state machines. It not
only helps align all the migration state machine, help cleanup the code
path especially on pre-switchover handling (see the patch itself), another
side benefit is we can unconditionally have a specific state to mark the
switchover phase, which might be helpful for debugging too.
This patch makes the DEVICE state to be present always, marking that source
QEMU is switching over. Then the state machine will be always as simple
as:
active-> [pre-switchover->] -> device -> [postcopy-active->] -> complete
After the change, no matter whether pre-switchover or postcopy is enabled
or not, we always have DEVICE state showing the switchover phase. When
pre-switchover enabled, we'll have an extra stage before that. When
postcopy is enabled, we'll have an extra stage after that.
A few qtests need touch up in QEMU tree for this change:
- A few iotest outputs (194, 203, 234, 262, 280)
- Teach libqos's migrate() on "device" state
Cc: Jiri Denemark <jdenemar@redhat.com>
Cc: Daniel P. Berrangé <berrange@redhat.com>
Cc: Dr. David Alan Gilbert <dave@treblig.org>
Signed-off-by: Peter Xu <peterx@redhat.com>
Tested-by: Jiri Denemark <jdenemar@redhat.com>
Reviewed-by: Juraj Marcin <jmarcin@redhat.com>
Link: https://lore.kernel.org/r/20250114230746.3268797-15-peterx@redhat.com
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Add the cpr-transfer migration mode, which allows the user to transfer
a guest to a new QEMU instance on the same host with minimal guest pause
time, by preserving guest RAM in place, albeit with new virtual addresses
in new QEMU, and by preserving device file descriptors. Pages that were
locked in memory for DMA in old QEMU remain locked in new QEMU, because the
descriptor of the device that locked them remains open.
cpr-transfer preserves memory and devices descriptors by sending them to
new QEMU over a unix domain socket using SCM_RIGHTS. Such CPR state cannot
be sent over the normal migration channel, because devices and backends
are created prior to reading the channel, so this mode sends CPR state
over a second "cpr" migration channel. New QEMU reads the cpr channel
prior to creating devices or backends. The user specifies the cpr channel
in the channel arguments on the outgoing side, and in a second -incoming
command-line parameter on the incoming side.
The user must start old QEMU with the the '-machine aux-ram-share=on' option,
which allows anonymous memory to be transferred in place to the new process
by transferring a memory descriptor for each ram block. Memory-backend
objects must have the share=on attribute, but memory-backend-epc is not
supported.
The user starts new QEMU on the same host as old QEMU, with command-line
arguments to create the same machine, plus the -incoming option for the
main migration channel, like normal live migration. In addition, the user
adds a second -incoming option with channel type "cpr". This CPR channel
must support file descriptor transfer with SCM_RIGHTS, i.e. it must be a
UNIX domain socket.
To initiate CPR, the user issues a migrate command to old QEMU, adding
a second migration channel of type "cpr" in the channels argument.
Old QEMU stops the VM, saves state to the migration channels, and enters
the postmigrate state. New QEMU mmap's memory descriptors, and execution
resumes.
The implementation splits qmp_migrate into start and finish functions.
Start sends CPR state to new QEMU, which responds by closing the CPR
channel. Old QEMU detects the HUP then calls finish, which connects the
main migration channel.
In summary, the usage is:
qemu-system-$arch -machine aux-ram-share=on ...
start new QEMU with "-incoming <main-uri> -incoming <cpr-channel>"
Issue commands to old QEMU:
migrate_set_parameter mode cpr-transfer
{"execute": "migrate", ...
{"channel-type": "main"...}, {"channel-type": "cpr"...} ... }
Signed-off-by: Steve Sistare <steven.sistare@oracle.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
Acked-by: Markus Armbruster <armbru@redhat.com>
Link: https://lore.kernel.org/r/1736967650-129648-17-git-send-email-steven.sistare@oracle.com
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Per previous discussion [1,2], this patch deprecates query-migrationthreads
command.
To summarize, the major reason of the deprecation is due to no sensible way
to consume the API properly:
(1) The reported list of threads are incomplete (ignoring destination
threads and non-multifd threads).
(2) For CPU pinning, there's no way to properly pin the threads with
the API if the threads will start running right away after migration
threads can be queried, so the threads will always run on the default
cores for a short window.
(3) For VM debugging, one can use "-name $VM,debug-threads=on" instead,
which will provide proper names for all migration threads.
[1] https://lore.kernel.org/r/20240930195837.825728-1-peterx@redhat.com
[2] https://lore.kernel.org/r/20241011153417.516715-1-peterx@redhat.com
Reviewed-by: Fabiano Rosas <farosas@suse.de>
Acked-by: Markus Armbruster <armbru@redhat.com>
Link: https://lore.kernel.org/r/20241022194501.1022443-1-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>
The zero-blocks capability was meant to be used along with the block
migration, which has been removed already in commit eef0bae3a7
("migration: Remove block migration").
Setting zero-blocks is currently a noop, but the outright removal of
the capability would cause and error in case some users are still
setting it. Put the capability through the deprecation process.
Signed-off-by: Fabiano Rosas <farosas@suse.de>
Reviewed-by: Peter Xu <peterx@redhat.com>
Link: https://lore.kernel.org/r/20240919134626.166183-4-dave@treblig.org
Signed-off-by: Peter Xu <peterx@redhat.com>
QAPI patches patches for 2024-09-10
# -----BEGIN PGP SIGNATURE-----
#
# iQJGBAABCAAwFiEENUvIs9frKmtoZ05fOHC0AOuRhlMFAmbgS0ASHGFybWJydUBy
# ZWRoYXQuY29tAAoJEDhwtADrkYZTm+sP/1kEuGdbjjVLQC7U+gCClhALpYUtIMHQ
# qNZWmnxFyW9Jx6Lk8+FZBRJegNlmA9AJ2ZzD/2I/WoNtcrNiE53rIRG71Lcy8I6U
# CRjekLPn0jhFuYhmRfwz6ilcLkyMJ15Yi7iG6ssFsO1joBtU5Q1qpzQmFUl9UbV9
# 9iBe7mE99VzNwi3hMELEM9YuSgAs8st4Itnn8SiHkJ3s6hlmF8N4HwNKfQrseGfb
# FuJNN4p8Gcu0aF/dkGE8ADJBvtgzkFnavXt0GQPC7SjGOF7rUXxnUQlszNZM0OcK
# UAfWU06sIyiOholQQt8H8jawT+rGCLomfgPUBVq5K8WByd21IeQeS0upSPeTjxS+
# d/C8wzpcbhdNOECwC/wDtgZHvCC95cHNrxuC4+4/Q1KIVVr+1cWqe8hQzFvm3TIg
# dduEFTQDIxhpE0GO1ZWNf90upzBYBWwIsh4bqsMZS7SpaYIZ6QV45yViZ1w2WfbH
# m3/F34Z3yqgFuZQrZv4OPo7xHK0+y7uZ60RrhSJtE0X080syRJKBbGmNBRidoVyu
# JOONWH44I/XN5enZV8StJnqJr9MCV0DBZUCi9ZhP/kAhBjLc5cQ6NByEa9/rebYX
# 1bLTiA1JDLzDVIan+A8dz0riWmHBvTyBwhLnEXvXb9lcB3ozgHTb7axE5RnZSYLI
# YQgBOBMFuQKM
# =YO/Q
# -----END PGP SIGNATURE-----
# gpg: Signature made Tue 10 Sep 2024 14:36:00 BST
# gpg: using RSA key 354BC8B3D7EB2A6B68674E5F3870B400EB918653
# gpg: issuer "armbru@redhat.com"
# gpg: Good signature from "Markus Armbruster <armbru@redhat.com>" [full]
# gpg: aka "Markus Armbruster <armbru@pond.sub.org>" [full]
# Primary key fingerprint: 354B C8B3 D7EB 2A6B 6867 4E5F 3870 B400 EB91 8653
* tag 'pull-qapi-2024-09-10' of https://repo.or.cz/qemu/armbru:
qapi/vfio: Rename VfioMigrationState to Qapi*, and drop prefix
qapi/cryptodev: Rename QCryptodevBackendAlgType to *Algo, and drop prefix
qapi/cryptodev: Drop unwanted 'prefix'
qapi/crypto: Rename QCryptoAFAlg to QCryptoAFAlgo
qapi/crypto: Rename QCryptoRSAPaddingAlgorithm to *Algo, and drop prefix
qapi/crypto: Rename QCryptoAkCipherAlgorithm to *Algo, and drop prefix
qapi/crypto: Rename QCryptoIVGenAlgorithm to *Algo, and drop prefix
qapi/crypto: Rename QCryptoCipherAlgorithm to *Algo, and drop prefix
qapi/crypto: Rename QCryptoHashAlgorithm to *Algo, and drop prefix
qapi/crypto: Drop unwanted 'prefix'
qapi/machine: Rename CpuS390* to S390Cpu*, and drop 'prefix'
qapi/ui: Drop temporary 'prefix'
qapi/machine: Drop temporary 'prefix'
qapi/ebpf: Drop temporary 'prefix'
qapi/crypto: Drop temporary 'prefix'
qapi/common: Drop temporary 'prefix'
qapi/block-core: Drop temporary 'prefix'
tests/qapi-schema: Drop temporary 'prefix'
qapi: Smarter camel_to_upper() to reduce need for 'prefix'
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
camel_to_upper() converts its argument from camel case to upper case
with '_' between words. Used for generated enumeration constant
prefixes.
When some of the words are spelled all caps, where exactly to insert
'_' is guesswork. camel_to_upper()'s guesses are bad enough in places
to make people override them with a 'prefix' in the schema.
Rewrite it to guess better:
1. Insert '_' after a non-upper case character followed by an upper
case character:
OneTwo -> ONE_TWO
One2Three -> ONE2_THREE
2. Insert '_' before the last upper case character followed by a
non-upper case character:
ACRONYMWord -> ACRONYM_Word
Except at the beginning (as in OneTwo above), or when there is
already one:
AbCd -> AB_CD
This changes the default enumeration constant prefix for a number of
enums. Generated enumeration constants change only where the default
is not overridden with 'prefix'.
The following enumerations without a 'prefix' change:
enum old camel_to_upper()
new camel_to_upper()
------------------------------------------------------------------
DisplayGLMode DISPLAYGL_MODE
DISPLAY_GL_MODE
EbpfProgramID EBPF_PROGRAMID
EBPF_PROGRAM_ID
HmatLBDataType HMATLB_DATA_TYPE
HMAT_LB_DATA_TYPE
HmatLBMemoryHierarchy HMATLB_MEMORY_HIERARCHY
HMAT_LB_MEMORY_HIERARCHY
MultiFDCompression MULTIFD_COMPRESSION
MULTI_FD_COMPRESSION
OffAutoPCIBAR OFF_AUTOPCIBAR
OFF_AUTO_PCIBAR
QCryptoBlockFormat Q_CRYPTO_BLOCK_FORMAT
QCRYPTO_BLOCK_FORMAT
QCryptoBlockLUKSKeyslotState Q_CRYPTO_BLOCKLUKS_KEYSLOT_STATE
QCRYPTO_BLOCK_LUKS_KEYSLOT_STATE
QKeyCode Q_KEY_CODE
QKEY_CODE
XDbgBlockGraphNodeType X_DBG_BLOCK_GRAPH_NODE_TYPE
XDBG_BLOCK_GRAPH_NODE_TYPE
TestUnionEnumA TEST_UNION_ENUMA
TEST_UNION_ENUM_A
Add a 'prefix' so generated code doesn't change now. Subsequent
commits will remove most of them again. Two will remain:
MULTIFD_COMPRESSION, because migration code generally spells "multifd"
that way, and Q_KEY_CODE, because that one is baked into
subprojects/keycodemapdb/tools/keymap-gen.
The following enumerations with a 'prefix' change so that the prefix
is now superfluous:
enum old camel_to_upper()
new camel_to_upper() [equal to prefix]
------------------------------------------------------------------
BlkdebugIOType BLKDEBUGIO_TYPE
BLKDEBUG_IO_TYPE
QCryptoTLSCredsEndpoint Q_CRYPTOTLS_CREDS_ENDPOINT
QCRYPTO_TLS_CREDS_ENDPOINT
QCryptoSecretFormat Q_CRYPTO_SECRET_FORMAT
QCRYPTO_SECRET_FORMAT
QCryptoCipherMode Q_CRYPTO_CIPHER_MODE
QCRYPTO_CIPHER_MODE
QCryptodevBackendType Q_CRYPTODEV_BACKEND_TYPE
QCRYPTODEV_BACKEND_TYPE
QType [builtin] Q_TYPE
QTYPE
Drop these prefixes.
The following enumerations with a 'prefix' change without making the
'prefix' superfluous:
enum old camel_to_upper()
new camel_to_upper() [equal to prefix]
prefix
------------------------------------------------------------------
CpuS390Entitlement CPUS390_ENTITLEMENT
CPU_S390_ENTITLEMENT
S390_CPU_ENTITLEMENT
CpuS390Polarization CPUS390_POLARIZATION
CPU_S390_POLARIZATION
S390_CPU_POLARIZATION
CpuS390State CPUS390_STATE
CPU_S390_STATE
S390_CPU_STATE
QAuthZListFormat Q_AUTHZ_LIST_FORMAT
QAUTH_Z_LIST_FORMAT
QAUTHZ_LIST_FORMAT
QAuthZListPolicy Q_AUTHZ_LIST_POLICY
QAUTH_Z_LIST_POLICY
QAUTHZ_LIST_POLICY
QCryptoAkCipherAlgorithm Q_CRYPTO_AK_CIPHER_ALGORITHM
QCRYPTO_AK_CIPHER_ALGORITHM
QCRYPTO_AKCIPHER_ALG
QCryptoAkCipherKeyType Q_CRYPTO_AK_CIPHER_KEY_TYPE
QCRYPTO_AK_CIPHER_KEY_TYPE
QCRYPTO_AKCIPHER_KEY_TYPE
QCryptoCipherAlgorithm Q_CRYPTO_CIPHER_ALGORITHM
QCRYPTO_CIPHER_ALGORITHM
QCRYPTO_CIPHER_ALG
QCryptoHashAlgorithm Q_CRYPTO_HASH_ALGORITHM
QCRYPTO_HASH_ALGORITHM
QCRYPTO_HASH_ALG
QCryptoIVGenAlgorithm Q_CRYPTOIV_GEN_ALGORITHM
QCRYPTO_IV_GEN_ALGORITHM
QCRYPTO_IVGEN_ALG
QCryptoRSAPaddingAlgorithm Q_CRYPTORSA_PADDING_ALGORITHM
QCRYPTO_RSA_PADDING_ALGORITHM
QCRYPTO_RSA_PADDING_ALG
QCryptodevBackendAlgType Q_CRYPTODEV_BACKEND_ALG_TYPE
QCRYPTODEV_BACKEND_ALG_TYPE
QCRYPTODEV_BACKEND_ALG
QCryptodevBackendServiceType Q_CRYPTODEV_BACKEND_SERVICE_TYPE
QCRYPTODEV_BACKEND_SERVICE_TYPE
QCRYPTODEV_BACKEND_SERVICE
Subsequent commits will tweak things to remove most of these prefixes.
Only QAUTHZ_LIST_FORMAT and QAUTHZ_LIST_POLICY will remain.
Signed-off-by: Markus Armbruster <armbru@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-ID: <20240904111836.3273842-2-armbru@redhat.com>
Markus Armbruster
John SnowandMarkus Armbruster
Peter XuandFabiano Rosas
Steve SistareandCédric Le Goater
Stefan Hajnoczi
Pierrick BouvierandMarkus Armbruster
Peter Maydell
Bryan ZhangandPeter Xu