izzy/xemu
0
0
Fork 0
mirror of https://github.com/izzy2lost/xemu.git synced 2026-07-06 00:20:22 -07:00
Code Issues Packages Projects Releases Wiki Activity

Merge tag 'migration-20240116-pull-request' of https://gitlab.com/peterx/qemu into staging

Browse Source 
Migration pull request 2nd batch for 9.0

- Het's cleanup on migration qmp command paths
- Fabiano's migration cleanups and test improvements
- Fabiano's patch to re-enable multifd-cancel test
- Peter's migration doc reorganizations
- Nick Briggs's fix for Solaries build on rdma

# -----BEGIN PGP SIGNATURE-----
#
# iIgEABYKADAWIQS5GE3CDMRX2s990ak7X8zN86vXBgUCZaX1PhIccGV0ZXJ4QHJl
# ZGhhdC5jb20ACgkQO1/MzfOr1wZSzwEAq6sp/ylNHLzNoMdWL28JLqCsb4DPYH2i
# u7XgYgT1qDAA/0vwoe4a5uFn1aaGCS+2d2syjJ8kOE7h+eZrbK520jsA
# =1zUG
# -----END PGP SIGNATURE-----
# gpg: Signature made Tue 16 Jan 2024 03:17:18 GMT
# gpg:                using EDDSA key B9184DC20CC457DACF7DD1A93B5FCCCDF3ABD706
# gpg:                issuer "peterx@redhat.com"
# gpg: Good signature from "Peter Xu <xzpeter@gmail.com>" [marginal]
# gpg:                 aka "Peter Xu <peterx@redhat.com>" [marginal]
# gpg: WARNING: This key is not certified with sufficiently trusted signatures!
# gpg:          It is not certain that the signature belongs to the owner.
# Primary key fingerprint: B918 4DC2 0CC4 57DA CF7D  D1A9 3B5F CCCD F3AB D706

* tag 'migration-20240116-pull-request' of https://gitlab.com/peterx/qemu:
  migration/rdma: define htonll/ntohll only if not predefined
  docs/migration: Further move virtio to be feature of migration
  docs/migration: Further move vfio to be feature of migration
  docs/migration: Organize "Postcopy" page
  docs/migration: Split "dirty limit"
  docs/migration: Split "Postcopy"
  docs/migration: Split "Debugging" and "Firmware"
  docs/migration: Split "Backwards compatibility" separately
  docs/migration: Convert virtio.txt into rST
  docs/migration: Create index page
  docs/migration: Create migration/ directory
  tests/qtest: Re-enable multifd cancel test
  tests/qtest/migration: Use the new migration_test_add
  tests/qtest/migration: Add a wrapper to print test names
  tests/qtest/migration: Print migration incoming errors
  migration: Report error in incoming migration
  migration/multifd: Change multifd_pages_init argument
  migration/multifd: Remove QEMUFile from where it is not needed
  migration/multifd: Remove MultiFDPages_t::packet_num
  migration: Simplify initial conditionals in migration for better readability

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
This commit is contained in:
Peter Maydell
2024-01-16 14:24:26 +00:00
parent 977542ded7 44ce1b5d2f
commit 9da8dfe4f5
20 changed files with 1861 additions and 1775 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/devel/index-internals.rst
+1 -2
View File
@@ -11,13 +11,12 @@ Details about QEMU's various subsystems including how to add features to them.
block-coroutine-wrapper
clocks
ebpf_rss
migration
migration/index
multi-process
reset
s390-cpu-topology
s390-dasd-ipl
tracing
vfio-migration
vfio-iommufd
writing-monitor-commands
virtio-backends
docs/devel/migration.rst
-1514
View File
File diff suppressed because it is too large Load Diff
docs/devel/migration/best-practices.rst
+48
View File
@@ -0,0 +1,48 @@
==============
Best practices
==============
Debugging
=========
The migration stream can be analyzed thanks to ``scripts/analyze-migration.py``.
Example usage:
.. code-block:: shell
$ qemu-system-x86_64 -display none -monitor stdio
(qemu) migrate "exec:cat > mig"
(qemu) q
$ ./scripts/analyze-migration.py -f mig
{
"ram (3)": {
"section sizes": {
"pc.ram": "0x0000000008000000",
...
See also ``analyze-migration.py -h`` help for more options.
Firmware
========
Migration migrates the copies of RAM and ROM, and thus when running
on the destination it includes the firmware from the source. Even after
resetting a VM, the old firmware is used. Only once QEMU has been restarted
is the new firmware in use.
- Changes in firmware size can cause changes in the required RAMBlock size
to hold the firmware and thus migration can fail. In practice it's best
to pad firmware images to convenient powers of 2 with plenty of space
for growth.
- Care should be taken with device emulation code so that newer
emulation code can work with older firmware to allow forward migration.
- Care should be taken with newer firmware so that backward migration
to older systems with older device emulation code will work.
In some cases it may be best to tie specific firmware versions to specific
versioned machine types to cut down on the combinations that will need
support. This is also useful when newer versions of firmware outgrow
the padding.
docs/devel/migration/compatibility.rst
+517
View File
File diff suppressed because it is too large Load Diff
docs/devel/migration/dirty-limit.rst
+71
View File
@@ -0,0 +1,71 @@
Dirty limit
===========
The dirty limit, short for dirty page rate upper limit, is a new capability
introduced in the 8.1 QEMU release that uses a new algorithm based on the KVM
dirty ring to throttle down the guest during live migration.
The algorithm framework is as follows:
::
------------------------------------------------------------------------------
main --------------> throttle thread ------------> PREPARE(1) <--------
thread \ | |
\ | |
\ V |
-\ CALCULATE(2) |
\ | |
\ | |
\ V |
\ SET PENALTY(3) -----
-\ |
\ |
\ V
-> virtual CPU thread -------> ACCEPT PENALTY(4)
------------------------------------------------------------------------------
When the qmp command qmp_set_vcpu_dirty_limit is called for the first time,
the QEMU main thread starts the throttle thread. The throttle thread, once
launched, executes the loop, which consists of three steps:
- PREPARE (1)
The entire work of PREPARE (1) is preparation for the second stage,
CALCULATE(2), as the name implies. It involves preparing the dirty
page rate value and the corresponding upper limit of the VM:
The dirty page rate is calculated via the KVM dirty ring mechanism,
which tells QEMU how many dirty pages a virtual CPU has had since the
last KVM_EXIT_DIRTY_RING_FULL exception; The dirty page rate upper
limit is specified by caller, therefore fetch it directly.
- CALCULATE (2)
Calculate a suitable sleep period for each virtual CPU, which will be
used to determine the penalty for the target virtual CPU. The
computation must be done carefully in order to reduce the dirty page
rate progressively down to the upper limit without oscillation. To
achieve this, two strategies are provided: the first is to add or
subtract sleep time based on the ratio of the current dirty page rate
to the limit, which is used when the current dirty page rate is far
from the limit; the second is to add or subtract a fixed time when
the current dirty page rate is close to the limit.
- SET PENALTY (3)
Set the sleep time for each virtual CPU that should be penalized based
on the results of the calculation supplied by step CALCULATE (2).
After completing the three above stages, the throttle thread loops back
to step PREPARE (1) until the dirty limit is reached.
On the other hand, each virtual CPU thread reads the sleep duration and
sleeps in the path of the KVM_EXIT_DIRTY_RING_FULL exception handler, that
is ACCEPT PENALTY (4). Virtual CPUs tied with writing processes will
obviously exit to the path and get penalized, whereas virtual CPUs involved
with read processes will not.
In summary, thanks to the KVM dirty ring technology, the dirty limit
algorithm will restrict virtual CPUs as needed to keep their dirty page
rate inside the limit. This leads to more steady reading performance during
live migration and can aid in improving large guest responsiveness.
docs/devel/migration/features.rst
+12
View File
@@ -0,0 +1,12 @@
Migration features
==================
Migration has plenty of features to support different use cases.
.. toctree::
:maxdepth: 2
postcopy
dirty-limit
vfio
virtio
docs/devel/migration/index.rst
+13
View File
@@ -0,0 +1,13 @@
Migration
=========
This is the main entry for QEMU migration documentations. It explains how
QEMU live migration works.
.. toctree::
:maxdepth: 2
main
features
compatibility
best-practices
docs/devel/migration/main.rst
+575
View File
File diff suppressed because it is too large Load Diff
docs/devel/migration/postcopy.rst
+313
View File
@@ -0,0 +1,313 @@
========
Postcopy
========
.. contents::
'Postcopy' migration is a way to deal with migrations that refuse to converge
(or take too long to converge) its plus side is that there is an upper bound on
the amount of migration traffic and time it takes, the down side is that during
the postcopy phase, a failure of *either* side causes the guest to be lost.
In postcopy the destination CPUs are started before all the memory has been
transferred, and accesses to pages that are yet to be transferred cause
a fault that's translated by QEMU into a request to the source QEMU.
Postcopy can be combined with precopy (i.e. normal migration) so that if precopy
doesn't finish in a given time the switch is made to postcopy.
Enabling postcopy
=================
To enable postcopy, issue this command on the monitor (both source and
destination) prior to the start of migration:
``migrate_set_capability postcopy-ram on``
The normal commands are then used to start a migration, which is still
started in precopy mode. Issuing:
``migrate_start_postcopy``
will now cause the transition from precopy to postcopy.
It can be issued immediately after migration is started or any
time later on. Issuing it after the end of a migration is harmless.
Blocktime is a postcopy live migration metric, intended to show how
long the vCPU was in state of interruptible sleep due to pagefault.
That metric is calculated both for all vCPUs as overlapped value, and
separately for each vCPU. These values are calculated on destination
side. To enable postcopy blocktime calculation, enter following
command on destination monitor:
``migrate_set_capability postcopy-blocktime on``
Postcopy blocktime can be retrieved by query-migrate qmp command.
postcopy-blocktime value of qmp command will show overlapped blocking
time for all vCPU, postcopy-vcpu-blocktime will show list of blocking
time per vCPU.
.. note::
During the postcopy phase, the bandwidth limits set using
``migrate_set_parameter`` is ignored (to avoid delaying requested pages that
the destination is waiting for).
Postcopy internals
==================
State machine
-------------
Postcopy moves through a series of states (see postcopy_state) from
ADVISE->DISCARD->LISTEN->RUNNING->END
- Advise
Set at the start of migration if postcopy is enabled, even
if it hasn't had the start command; here the destination
checks that its OS has the support needed for postcopy, and performs
setup to ensure the RAM mappings are suitable for later postcopy.
The destination will fail early in migration at this point if the
required OS support is not present.
(Triggered by reception of POSTCOPY_ADVISE command)
- Discard
Entered on receipt of the first 'discard' command; prior to
the first Discard being performed, hugepages are switched off
(using madvise) to ensure that no new huge pages are created
during the postcopy phase, and to cause any huge pages that
have discards on them to be broken.
- Listen
The first command in the package, POSTCOPY_LISTEN, switches
the destination state to Listen, and starts a new thread
(the 'listen thread') which takes over the job of receiving
pages off the migration stream, while the main thread carries
on processing the blob. With this thread able to process page
reception, the destination now 'sensitises' the RAM to detect
any access to missing pages (on Linux using the 'userfault'
system).
- Running
POSTCOPY_RUN causes the destination to synchronise all
state and start the CPUs and IO devices running. The main
thread now finishes processing the migration package and
now carries on as it would for normal precopy migration
(although it can't do the cleanup it would do as it
finishes a normal migration).
- Paused
Postcopy can run into a paused state (normally on both sides when
happens), where all threads will be temporarily halted mostly due to
network errors. When reaching paused state, migration will make sure
the qemu binary on both sides maintain the data without corrupting
the VM. To continue the migration, the admin needs to fix the
migration channel using the QMP command 'migrate-recover' on the
destination node, then resume the migration using QMP command 'migrate'
again on source node, with resume=true flag set.
- End
The listen thread can now quit, and perform the cleanup of migration
state, the migration is now complete.
Device transfer
---------------
Loading of device data may cause the device emulation to access guest RAM
that may trigger faults that have to be resolved by the source, as such
the migration stream has to be able to respond with page data *during* the
device load, and hence the device data has to be read from the stream completely
before the device load begins to free the stream up. This is achieved by
'packaging' the device data into a blob that's read in one go.
Source behaviour
----------------
Until postcopy is entered the migration stream is identical to normal
precopy, except for the addition of a 'postcopy advise' command at
the beginning, to tell the destination that postcopy might happen.
When postcopy starts the source sends the page discard data and then
forms the 'package' containing:
- Command: 'postcopy listen'
- The device state
A series of sections, identical to the precopy streams device state stream
containing everything except postcopiable devices (i.e. RAM)
- Command: 'postcopy run'
The 'package' is sent as the data part of a Command: ``CMD_PACKAGED``, and the
contents are formatted in the same way as the main migration stream.
During postcopy the source scans the list of dirty pages and sends them
to the destination without being requested (in much the same way as precopy),
however when a page request is received from the destination, the dirty page
scanning restarts from the requested location. This causes requested pages
to be sent quickly, and also causes pages directly after the requested page
to be sent quickly in the hope that those pages are likely to be used
by the destination soon.
Destination behaviour
---------------------
Initially the destination looks the same as precopy, with a single thread
reading the migration stream; the 'postcopy advise' and 'discard' commands
are processed to change the way RAM is managed, but don't affect the stream
processing.
::
------------------------------------------------------------------------------
1 2 3 4 5 6 7
main -----DISCARD-CMD_PACKAGED ( LISTEN DEVICE DEVICE DEVICE RUN )
thread | |
| (page request)
| \___
v \
listen thread: --- page -- page -- page -- page -- page --
a b c
------------------------------------------------------------------------------
- On receipt of ``CMD_PACKAGED`` (1)
All the data associated with the package - the ( ... ) section in the diagram -
is read into memory, and the main thread recurses into qemu_loadvm_state_main
to process the contents of the package (2) which contains commands (3,6) and
devices (4...)
- On receipt of 'postcopy listen' - 3 -(i.e. the 1st command in the package)
a new thread (a) is started that takes over servicing the migration stream,
while the main thread carries on loading the package. It loads normal
background page data (b) but if during a device load a fault happens (5)
the returned page (c) is loaded by the listen thread allowing the main
threads device load to carry on.
- The last thing in the ``CMD_PACKAGED`` is a 'RUN' command (6)
letting the destination CPUs start running. At the end of the
``CMD_PACKAGED`` (7) the main thread returns to normal running behaviour and
is no longer used by migration, while the listen thread carries on servicing
page data until the end of migration.
Source side page bitmap
-----------------------
The 'migration bitmap' in postcopy is basically the same as in the precopy,
where each of the bit to indicate that page is 'dirty' - i.e. needs
sending. During the precopy phase this is updated as the CPU dirties
pages, however during postcopy the CPUs are stopped and nothing should
dirty anything any more. Instead, dirty bits are cleared when the relevant
pages are sent during postcopy.
Postcopy features
=================
Postcopy recovery
-----------------
Comparing to precopy, postcopy is special on error handlings. When any
error happens (in this case, mostly network errors), QEMU cannot easily
fail a migration because VM data resides in both source and destination
QEMU instances. On the other hand, when issue happens QEMU on both sides
will go into a paused state. It'll need a recovery phase to continue a
paused postcopy migration.
The recovery phase normally contains a few steps:
- When network issue occurs, both QEMU will go into PAUSED state
- When the network is recovered (or a new network is provided), the admin
can setup the new channel for migration using QMP command
'migrate-recover' on destination node, preparing for a resume.
- On source host, the admin can continue the interrupted postcopy
migration using QMP command 'migrate' with resume=true flag set.
- After the connection is re-established, QEMU will continue the postcopy
migration on both sides.
During a paused postcopy migration, the VM can logically still continue
running, and it will not be impacted from any page access to pages that
were already migrated to destination VM before the interruption happens.
However, if any of the missing pages got accessed on destination VM, the VM
thread will be halted waiting for the page to be migrated, it means it can
be halted until the recovery is complete.
The impact of accessing missing pages can be relevant to different
configurations of the guest. For example, when with async page fault
enabled, logically the guest can proactively schedule out the threads
accessing missing pages.
Postcopy with hugepages
-----------------------
Postcopy now works with hugetlbfs backed memory:
a) The linux kernel on the destination must support userfault on hugepages.
b) The huge-page configuration on the source and destination VMs must be
identical; i.e. RAMBlocks on both sides must use the same page size.
c) Note that ``-mem-path /dev/hugepages`` will fall back to allocating normal
RAM if it doesn't have enough hugepages, triggering (b) to fail.
Using ``-mem-prealloc`` enforces the allocation using hugepages.
d) Care should be taken with the size of hugepage used; postcopy with 2MB
hugepages works well, however 1GB hugepages are likely to be problematic
since it takes ~1 second to transfer a 1GB hugepage across a 10Gbps link,
and until the full page is transferred the destination thread is blocked.
Postcopy with shared memory
---------------------------
Postcopy migration with shared memory needs explicit support from the other
processes that share memory and from QEMU. There are restrictions on the type of
memory that userfault can support shared.
The Linux kernel userfault support works on ``/dev/shm`` memory and on ``hugetlbfs``
(although the kernel doesn't provide an equivalent to ``madvise(MADV_DONTNEED)``
for hugetlbfs which may be a problem in some configurations).
The vhost-user code in QEMU supports clients that have Postcopy support,
and the ``vhost-user-bridge`` (in ``tests/``) and the DPDK package have changes
to support postcopy.
The client needs to open a userfaultfd and register the areas
of memory that it maps with userfault. The client must then pass the
userfaultfd back to QEMU together with a mapping table that allows
fault addresses in the clients address space to be converted back to
RAMBlock/offsets. The client's userfaultfd is added to the postcopy
fault-thread and page requests are made on behalf of the client by QEMU.
QEMU performs 'wake' operations on the client's userfaultfd to allow it
to continue after a page has arrived.
.. note::
There are two future improvements that would be nice:
a) Some way to make QEMU ignorant of the addresses in the clients
address space
b) Avoiding the need for QEMU to perform ufd-wake calls after the
pages have arrived
Retro-fitting postcopy to existing clients is possible:
a) A mechanism is needed for the registration with userfault as above,
and the registration needs to be coordinated with the phases of
postcopy. In vhost-user extra messages are added to the existing
control channel.
b) Any thread that can block due to guest memory accesses must be
identified and the implication understood; for example if the
guest memory access is made while holding a lock then all other
threads waiting for that lock will also be blocked.
Postcopy preemption mode
------------------------
Postcopy preempt is a new capability introduced in 8.0 QEMU release, it
allows urgent pages (those got page fault requested from destination QEMU
explicitly) to be sent in a separate preempt channel, rather than queued in
the background migration channel. Anyone who cares about latencies of page
faults during a postcopy migration should enable this feature. By default,
it's not enabled.
docs/devel/vfio-migration.rst → docs/devel/migration/vfio.rst
+1 -1
View File
@@ -1,5 +1,5 @@
=====================
VFIO device Migration
VFIO device migration
=====================
Migration of virtual machine involves saving the state for each device that
docs/devel/migration/virtio.rst
+115
View File
@@ -0,0 +1,115 @@
=======================
Virtio device migration
=======================
Copyright 2015 IBM Corp.
This work is licensed under the terms of the GNU GPL, version 2 or later. See
the COPYING file in the top-level directory.
Saving and restoring the state of virtio devices is a bit of a twisty maze,
for several reasons:
- state is distributed between several parts:
- virtio core, for common fields like features, number of queues, ...
- virtio transport (pci, ccw, ...), for the different proxy devices and
transport specific state (msix vectors, indicators, ...)
- virtio device (net, blk, ...), for the different device types and their
state (mac address, request queue, ...)
- most fields are saved via the stream interface; subsequently, subsections
have been added to make cross-version migration possible
This file attempts to document the current procedure and point out some
caveats.
Save state procedure
====================
::
virtio core virtio transport virtio device
----------- ---------------- -------------
save() function registered
via VMState wrapper on
device class
virtio_save() <----------
------> save_config()
- save proxy device
- save transport-specific
device fields
- save common device
fields
- save common virtqueue
fields
------> save_queue()
- save transport-specific
virtqueue fields
------> save_device()
- save device-specific
fields
- save subsections
- device endianness,
if changed from
default endianness
- 64 bit features, if
any high feature bit
is set
- virtio-1 virtqueue
fields, if VERSION_1
is set
Load state procedure
====================
::
virtio core virtio transport virtio device
----------- ---------------- -------------
load() function registered
via VMState wrapper on
device class
virtio_load() <----------
------> load_config()
- load proxy device
- load transport-specific
device fields
- load common device
fields
- load common virtqueue
fields
------> load_queue()
- load transport-specific
virtqueue fields
- notify guest
------> load_device()
- load device-specific
fields
- load subsections
- device endianness
- 64 bit features
- virtio-1 virtqueue
fields
- sanitize endianness
- sanitize features
- virtqueue index sanity
check
- feature-dependent setup
Implications of this setup
==========================
Devices need to be careful in their state processing during load: The
load_device() procedure is invoked by the core before subsections have
been loaded. Any code that depends on information transmitted in subsections
therefore has to be invoked in the device's load() function _after_
virtio_load() returned (like e.g. code depending on features).
Any extension of the state being migrated should be done in subsections
added to the core for compatibility reasons. If transport or device specific
state is added, core needs to invoke a callback from the new subsection.
docs/devel/virtio-migration.txt
-108
View File
@@ -1,108 +0,0 @@
Virtio devices and migration
============================
Copyright 2015 IBM Corp.
This work is licensed under the terms of the GNU GPL, version 2 or later. See
the COPYING file in the top-level directory.
Saving and restoring the state of virtio devices is a bit of a twisty maze,
for several reasons:
- state is distributed between several parts:
- virtio core, for common fields like features, number of queues, ...
- virtio transport (pci, ccw, ...), for the different proxy devices and
transport specific state (msix vectors, indicators, ...)
- virtio device (net, blk, ...), for the different device types and their
state (mac address, request queue, ...)
- most fields are saved via the stream interface; subsequently, subsections
have been added to make cross-version migration possible
This file attempts to document the current procedure and point out some
caveats.
Save state procedure
====================
virtio core virtio transport virtio device
----------- ---------------- -------------
save() function registered
via VMState wrapper on
device class
virtio_save() <----------
------> save_config()
- save proxy device
- save transport-specific
device fields
- save common device
fields
- save common virtqueue
fields
------> save_queue()
- save transport-specific
virtqueue fields
------> save_device()
- save device-specific
fields
- save subsections
- device endianness,
if changed from
default endianness
- 64 bit features, if
any high feature bit
is set
- virtio-1 virtqueue
fields, if VERSION_1
is set
Load state procedure
====================
virtio core virtio transport virtio device
----------- ---------------- -------------
load() function registered
via VMState wrapper on
device class
virtio_load() <----------
------> load_config()
- load proxy device
- load transport-specific
device fields
- load common device
fields
- load common virtqueue
fields
------> load_queue()
- load transport-specific
virtqueue fields
- notify guest
------> load_device()
- load device-specific
fields
- load subsections
- device endianness
- 64 bit features
- virtio-1 virtqueue
fields
- sanitize endianness
- sanitize features
- virtqueue index sanity
check
- feature-dependent setup
Implications of this setup
==========================
Devices need to be careful in their state processing during load: The
load_device() procedure is invoked by the core before subsections have
been loaded. Any code that depends on information transmitted in subsections
therefore has to be invoked in the device's load() function _after_
virtio_load() returned (like e.g. code depending on features).
Any extension of the state being migrated should be done in subsections
added to the core for compatibility reasons. If transport or device specific
state is added, core needs to invoke a callback from the new subsection.
migration/migration.c
+23 -20
View File
@@ -523,28 +523,26 @@ static void qemu_start_incoming_migration(const char *uri, bool has_channels,
/*
* Having preliminary checks for uri and channel
*/
if (uri && has_channels) {
error_setg(errp, "'uri' and 'channels' arguments are mutually "
"exclusive; exactly one of the two should be present in "
"'migrate-incoming' qmp command ");
if (!uri == !channels) {
error_setg(errp, "need either 'uri' or 'channels' argument");
return;
} else if (channels) {
}
if (channels) {
/* To verify that Migrate channel list has only item */
if (channels->next) {
error_setg(errp, "Channel list has more than one entries");
return;
}
addr = channels->value->addr;
} else if (uri) {
}
if (uri) {
/* caller uses the old URI syntax */
if (!migrate_uri_parse(uri, &channel, errp)) {
return;
}
addr = channel->addr;
} else {
error_setg(errp, "neither 'uri' or 'channels' argument are "
"specified in 'migrate-incoming' qmp command ");
return;
}
/* transport mechanism not suitable for migration? */
@@ -699,6 +697,13 @@ process_incoming_migration_co(void *opaque)
}
if (ret < 0) {
MigrationState *s = migrate_get_current();
if (migrate_has_error(s)) {
WITH_QEMU_LOCK_GUARD(&s->error_mutex) {
error_report_err(s->error);
}
}
error_report("load of migration failed: %s", strerror(-ret));
goto fail;
}
@@ -1924,28 +1929,26 @@ void qmp_migrate(const char *uri, bool has_channels,
/*
* Having preliminary checks for uri and channel
*/
if (uri && has_channels) {
error_setg(errp, "'uri' and 'channels' arguments are mutually "
"exclusive; exactly one of the two should be present in "
"'migrate' qmp command ");
if (!uri == !channels) {
error_setg(errp, "need either 'uri' or 'channels' argument");
return;
} else if (channels) {
}
if (channels) {
/* To verify that Migrate channel list has only item */
if (channels->next) {
error_setg(errp, "Channel list has more than one entries");
return;
}
addr = channels->value->addr;
} else if (uri) {
}
if (uri) {
/* caller uses the old URI syntax */
if (!migrate_uri_parse(uri, &channel, errp)) {
return;
}
addr = channel->addr;
} else {
error_setg(errp, "neither 'uri' or 'channels' argument are "
"specified in 'migrate' qmp command ");
return;
}
/* transport mechanism not suitable for migration? */
migration/multifd.c
+9 -10
View File
@@ -236,12 +236,12 @@ static int multifd_recv_initial_packet(QIOChannel *c, Error **errp)
return msg.id;
}
static MultiFDPages_t *multifd_pages_init(size_t size)
static MultiFDPages_t *multifd_pages_init(uint32_t n)
{
MultiFDPages_t *pages = g_new0(MultiFDPages_t, 1);
pages->allocated = size;
pages->offset = g_new0(ram_addr_t, size);
pages->allocated = n;
pages->offset = g_new0(ram_addr_t, n);
return pages;
}
@@ -250,7 +250,6 @@ static void multifd_pages_clear(MultiFDPages_t *pages)
{
pages->num = 0;
pages->allocated = 0;
pages->packet_num = 0;
pages->block = NULL;
g_free(pages->offset);
pages->offset = NULL;
@@ -391,7 +390,7 @@ struct {
* false.
*/
static int multifd_send_pages(QEMUFile *f)
static int multifd_send_pages(void)
{
int i;
static int next_channel;
@@ -437,7 +436,7 @@ static int multifd_send_pages(QEMUFile *f)
return 1;
}
int multifd_queue_page(QEMUFile *f, RAMBlock *block, ram_addr_t offset)
int multifd_queue_page(RAMBlock *block, ram_addr_t offset)
{
MultiFDPages_t *pages = multifd_send_state->pages;
bool changed = false;
@@ -457,12 +456,12 @@ int multifd_queue_page(QEMUFile *f, RAMBlock *block, ram_addr_t offset)
changed = true;
}
if (multifd_send_pages(f) < 0) {
if (multifd_send_pages() < 0) {
return -1;
}
if (changed) {
return multifd_queue_page(f, block, offset);
return multifd_queue_page(block, offset);
}
return 1;
@@ -584,7 +583,7 @@ static int multifd_zero_copy_flush(QIOChannel *c)
return ret;
}
int multifd_send_sync_main(QEMUFile *f)
int multifd_send_sync_main(void)
{
int i;
bool flush_zero_copy;
@@ -593,7 +592,7 @@ int multifd_send_sync_main(QEMUFile *f)
return 0;
}
if (multifd_send_state->pages->num) {
if (multifd_send_pages(f) < 0) {
if (multifd_send_pages() < 0) {
error_report("%s: multifd_send_pages fail", __func__);
return -1;
}
migration/multifd.h
+2 -4
View File
@@ -21,8 +21,8 @@ void multifd_load_shutdown(void);
bool multifd_recv_all_channels_created(void);
void multifd_recv_new_channel(QIOChannel *ioc, Error **errp);
void multifd_recv_sync_main(void);
int multifd_send_sync_main(QEMUFile *f);
int multifd_queue_page(QEMUFile *f, RAMBlock *block, ram_addr_t offset);
int multifd_send_sync_main(void);
int multifd_queue_page(RAMBlock *block, ram_addr_t offset);
/* Multifd Compression flags */
#define MULTIFD_FLAG_SYNC (1 << 0)
@@ -58,8 +58,6 @@ typedef struct {
uint32_t num;
/* number of allocated pages */
uint32_t allocated;
/* global number of generated multifd packets */
uint64_t packet_num;
/* offset of each page */
ram_addr_t *offset;
RAMBlock *block;
migration/ram.c
+7 -8
View File
@@ -1250,10 +1250,9 @@ static int ram_save_page(RAMState *rs, PageSearchStatus *pss)
return pages;
}
static int ram_save_multifd_page(QEMUFile *file, RAMBlock *block,
ram_addr_t offset)
static int ram_save_multifd_page(RAMBlock *block, ram_addr_t offset)
{
if (multifd_queue_page(file, block, offset) < 0) {
if (multifd_queue_page(block, offset) < 0) {
return -1;
}
stat64_add(&mig_stats.normal_pages, 1);
@@ -1336,7 +1335,7 @@ static int find_dirty_block(RAMState *rs, PageSearchStatus *pss)
if (migrate_multifd() &&
!migrate_multifd_flush_after_each_section()) {
QEMUFile *f = rs->pss[RAM_CHANNEL_PRECOPY].pss_channel;
int ret = multifd_send_sync_main(f);
int ret = multifd_send_sync_main();
if (ret < 0) {
return ret;
}
@@ -2067,7 +2066,7 @@ static int ram_save_target_page_legacy(RAMState *rs, PageSearchStatus *pss)
* still see partially copied pages which is data corruption.
*/
if (migrate_multifd() && !migration_in_postcopy()) {
return ram_save_multifd_page(pss->pss_channel, block, offset);
return ram_save_multifd_page(block, offset);
}
return ram_save_page(rs, pss);
@@ -2985,7 +2984,7 @@ static int ram_save_setup(QEMUFile *f, void *opaque)
migration_ops->ram_save_target_page = ram_save_target_page_legacy;
bql_unlock();
ret = multifd_send_sync_main(f);
ret = multifd_send_sync_main();
bql_lock();
if (ret < 0) {
return ret;
@@ -3109,7 +3108,7 @@ out:
if (ret >= 0
&& migration_is_setup_or_active(migrate_get_current()->state)) {
if (migrate_multifd() && migrate_multifd_flush_after_each_section()) {
ret = multifd_send_sync_main(rs->pss[RAM_CHANNEL_PRECOPY].pss_channel);
ret = multifd_send_sync_main();
if (ret < 0) {
return ret;
}
@@ -3183,7 +3182,7 @@ static int ram_save_complete(QEMUFile *f, void *opaque)
}
}
ret = multifd_send_sync_main(rs->pss[RAM_CHANNEL_PRECOPY].pss_channel);
ret = multifd_send_sync_main();
if (ret < 0) {
return ret;
}
migration/rdma.c
+4
View File
@@ -238,6 +238,7 @@ static const char *control_desc(unsigned int rdma_control)
return strs[rdma_control];
}
#if !defined(htonll)
static uint64_t htonll(uint64_t v)
{
union { uint32_t lv[2]; uint64_t llv; } u;
@@ -245,13 +246,16 @@ static uint64_t htonll(uint64_t v)
u.lv[1] = htonl(v & 0xFFFFFFFFULL);
return u.llv;
}
#endif
#if !defined(ntohll)
static uint64_t ntohll(uint64_t v)
{
union { uint32_t lv[2]; uint64_t llv; } u;
u.llv = v;
return ((uint64_t)ntohl(u.lv[0]) << 32) | (uint64_t) ntohl(u.lv[1]);
}
#endif
static void dest_block_to_network(RDMADestBlock *db)
{
tests/qtest/migration-helpers.c
+38
View File
@@ -111,6 +111,12 @@ void migrate_incoming_qmp(QTestState *to, const char *uri, const char *fmt, ...)
rsp = qtest_qmp(to, "{ 'execute': 'migrate-incoming', 'arguments': %p}",
args);
if (!qdict_haskey(rsp, "return")) {
g_autoptr(GString) s = qobject_to_json_pretty(QOBJECT(rsp), true);
g_test_message("%s", s->str);
}
g_assert(qdict_haskey(rsp, "return"));
qobject_unref(rsp);
@@ -285,3 +291,35 @@ char *resolve_machine_version(const char *alias, const char *var1,
return find_common_machine_version(machine_name, var1, var2);
}
typedef struct {
char *name;
void (*func)(void);
} MigrationTest;
static void migration_test_destroy(gpointer data)
{
MigrationTest *test = (MigrationTest *)data;
g_free(test->name);
g_free(test);
}
static void migration_test_wrapper(const void *data)
{
MigrationTest *test = (MigrationTest *)data;
g_test_message("Running /%s%s", qtest_get_arch(), test->name);
test->func();
}
void migration_test_add(const char *path, void (*fn)(void))
{
MigrationTest *test = g_new0(MigrationTest, 1);
test->func = fn;
test->name = g_strdup(path);
qtest_add_data_func_full(path, test, migration_test_wrapper,
migration_test_destroy);
}
tests/qtest/migration-helpers.h
+1
View File
@@ -52,4 +52,5 @@ char *find_common_machine_version(const char *mtype, const char *var1,
const char *var2);
char *resolve_machine_version(const char *alias, const char *var1,
const char *var2);
void migration_test_add(const char *path, void (*fn)(void));
#endif /* MIGRATION_HELPERS_H */
tests/qtest/migration-test.c
+111 -108
View File
@@ -3404,70 +3404,75 @@ int main(int argc, char **argv)
module_call_init(MODULE_INIT_QOM);
if (is_x86) {
qtest_add_func("/migration/precopy/unix/suspend/live",
test_precopy_unix_suspend_live);
qtest_add_func("/migration/precopy/unix/suspend/notlive",
test_precopy_unix_suspend_notlive);
migration_test_add("/migration/precopy/unix/suspend/live",
test_precopy_unix_suspend_live);
migration_test_add("/migration/precopy/unix/suspend/notlive",
test_precopy_unix_suspend_notlive);
}
if (has_uffd) {
qtest_add_func("/migration/postcopy/plain", test_postcopy);
qtest_add_func("/migration/postcopy/recovery/plain",
test_postcopy_recovery);
qtest_add_func("/migration/postcopy/preempt/plain", test_postcopy_preempt);
qtest_add_func("/migration/postcopy/preempt/recovery/plain",
test_postcopy_preempt_recovery);
migration_test_add("/migration/postcopy/plain", test_postcopy);
migration_test_add("/migration/postcopy/recovery/plain",
test_postcopy_recovery);
migration_test_add("/migration/postcopy/preempt/plain",
test_postcopy_preempt);
migration_test_add("/migration/postcopy/preempt/recovery/plain",
test_postcopy_preempt_recovery);
if (getenv("QEMU_TEST_FLAKY_TESTS")) {
qtest_add_func("/migration/postcopy/compress/plain",
test_postcopy_compress);
qtest_add_func("/migration/postcopy/recovery/compress/plain",
test_postcopy_recovery_compress);
migration_test_add("/migration/postcopy/compress/plain",
test_postcopy_compress);
migration_test_add("/migration/postcopy/recovery/compress/plain",
test_postcopy_recovery_compress);
}
#ifndef _WIN32
qtest_add_func("/migration/postcopy/recovery/double-failures",
test_postcopy_recovery_double_fail);
migration_test_add("/migration/postcopy/recovery/double-failures",
test_postcopy_recovery_double_fail);
#endif /* _WIN32 */
if (is_x86) {
qtest_add_func("/migration/postcopy/suspend",
test_postcopy_suspend);
migration_test_add("/migration/postcopy/suspend",
test_postcopy_suspend);
}
}
qtest_add_func("/migration/bad_dest", test_baddest);
migration_test_add("/migration/bad_dest", test_baddest);
#ifndef _WIN32
qtest_add_func("/migration/analyze-script", test_analyze_script);
if (!g_str_equal(arch, "s390x")) {
migration_test_add("/migration/analyze-script", test_analyze_script);
}
#endif
qtest_add_func("/migration/precopy/unix/plain", test_precopy_unix_plain);
qtest_add_func("/migration/precopy/unix/xbzrle", test_precopy_unix_xbzrle);
migration_test_add("/migration/precopy/unix/plain",
test_precopy_unix_plain);
migration_test_add("/migration/precopy/unix/xbzrle",
test_precopy_unix_xbzrle);
/*
* Compression fails from time to time.
* Put test here but don't enable it until everything is fixed.
*/
if (getenv("QEMU_TEST_FLAKY_TESTS")) {
qtest_add_func("/migration/precopy/unix/compress/wait",
test_precopy_unix_compress);
qtest_add_func("/migration/precopy/unix/compress/nowait",
test_precopy_unix_compress_nowait);
migration_test_add("/migration/precopy/unix/compress/wait",
test_precopy_unix_compress);
migration_test_add("/migration/precopy/unix/compress/nowait",
test_precopy_unix_compress_nowait);
}
qtest_add_func("/migration/precopy/file",
test_precopy_file);
qtest_add_func("/migration/precopy/file/offset",
test_precopy_file_offset);
qtest_add_func("/migration/precopy/file/offset/bad",
test_precopy_file_offset_bad);
migration_test_add("/migration/precopy/file",
test_precopy_file);
migration_test_add("/migration/precopy/file/offset",
test_precopy_file_offset);
migration_test_add("/migration/precopy/file/offset/bad",
test_precopy_file_offset_bad);
/*
* Our CI system has problems with shared memory.
* Don't run this test until we find a workaround.
*/
if (getenv("QEMU_TEST_FLAKY_TESTS")) {
qtest_add_func("/migration/mode/reboot", test_mode_reboot);
migration_test_add("/migration/mode/reboot", test_mode_reboot);
}
#ifdef CONFIG_GNUTLS
qtest_add_func("/migration/precopy/unix/tls/psk",
test_precopy_unix_tls_psk);
migration_test_add("/migration/precopy/unix/tls/psk",
test_precopy_unix_tls_psk);
if (has_uffd) {
/*
@@ -3475,110 +3480,108 @@ int main(int argc, char **argv)
* channels are tested under precopy. Here what we want to test is the
* general postcopy path that has TLS channel enabled.
*/
qtest_add_func("/migration/postcopy/tls/psk", test_postcopy_tls_psk);
qtest_add_func("/migration/postcopy/recovery/tls/psk",
test_postcopy_recovery_tls_psk);
qtest_add_func("/migration/postcopy/preempt/tls/psk",
test_postcopy_preempt_tls_psk);
qtest_add_func("/migration/postcopy/preempt/recovery/tls/psk",
test_postcopy_preempt_all);
migration_test_add("/migration/postcopy/tls/psk",
test_postcopy_tls_psk);
migration_test_add("/migration/postcopy/recovery/tls/psk",
test_postcopy_recovery_tls_psk);
migration_test_add("/migration/postcopy/preempt/tls/psk",
test_postcopy_preempt_tls_psk);
migration_test_add("/migration/postcopy/preempt/recovery/tls/psk",
test_postcopy_preempt_all);
}
#ifdef CONFIG_TASN1
qtest_add_func("/migration/precopy/unix/tls/x509/default-host",
test_precopy_unix_tls_x509_default_host);
qtest_add_func("/migration/precopy/unix/tls/x509/override-host",
test_precopy_unix_tls_x509_override_host);
migration_test_add("/migration/precopy/unix/tls/x509/default-host",
test_precopy_unix_tls_x509_default_host);
migration_test_add("/migration/precopy/unix/tls/x509/override-host",
test_precopy_unix_tls_x509_override_host);
#endif /* CONFIG_TASN1 */
#endif /* CONFIG_GNUTLS */
qtest_add_func("/migration/precopy/tcp/plain", test_precopy_tcp_plain);
migration_test_add("/migration/precopy/tcp/plain", test_precopy_tcp_plain);
qtest_add_func("/migration/precopy/tcp/plain/switchover-ack",
test_precopy_tcp_switchover_ack);
migration_test_add("/migration/precopy/tcp/plain/switchover-ack",
test_precopy_tcp_switchover_ack);
#ifdef CONFIG_GNUTLS
qtest_add_func("/migration/precopy/tcp/tls/psk/match",
test_precopy_tcp_tls_psk_match);
qtest_add_func("/migration/precopy/tcp/tls/psk/mismatch",
test_precopy_tcp_tls_psk_mismatch);
migration_test_add("/migration/precopy/tcp/tls/psk/match",
test_precopy_tcp_tls_psk_match);
migration_test_add("/migration/precopy/tcp/tls/psk/mismatch",
test_precopy_tcp_tls_psk_mismatch);
#ifdef CONFIG_TASN1
qtest_add_func("/migration/precopy/tcp/tls/x509/default-host",
test_precopy_tcp_tls_x509_default_host);
qtest_add_func("/migration/precopy/tcp/tls/x509/override-host",
test_precopy_tcp_tls_x509_override_host);
qtest_add_func("/migration/precopy/tcp/tls/x509/mismatch-host",
test_precopy_tcp_tls_x509_mismatch_host);
qtest_add_func("/migration/precopy/tcp/tls/x509/friendly-client",
test_precopy_tcp_tls_x509_friendly_client);
qtest_add_func("/migration/precopy/tcp/tls/x509/hostile-client",
test_precopy_tcp_tls_x509_hostile_client);
qtest_add_func("/migration/precopy/tcp/tls/x509/allow-anon-client",
test_precopy_tcp_tls_x509_allow_anon_client);
qtest_add_func("/migration/precopy/tcp/tls/x509/reject-anon-client",
test_precopy_tcp_tls_x509_reject_anon_client);
migration_test_add("/migration/precopy/tcp/tls/x509/default-host",
test_precopy_tcp_tls_x509_default_host);
migration_test_add("/migration/precopy/tcp/tls/x509/override-host",
test_precopy_tcp_tls_x509_override_host);
migration_test_add("/migration/precopy/tcp/tls/x509/mismatch-host",
test_precopy_tcp_tls_x509_mismatch_host);
migration_test_add("/migration/precopy/tcp/tls/x509/friendly-client",
test_precopy_tcp_tls_x509_friendly_client);
migration_test_add("/migration/precopy/tcp/tls/x509/hostile-client",
test_precopy_tcp_tls_x509_hostile_client);
migration_test_add("/migration/precopy/tcp/tls/x509/allow-anon-client",
test_precopy_tcp_tls_x509_allow_anon_client);
migration_test_add("/migration/precopy/tcp/tls/x509/reject-anon-client",
test_precopy_tcp_tls_x509_reject_anon_client);
#endif /* CONFIG_TASN1 */
#endif /* CONFIG_GNUTLS */
/* qtest_add_func("/migration/ignore_shared", test_ignore_shared); */
/* migration_test_add("/migration/ignore_shared", test_ignore_shared); */
#ifndef _WIN32
qtest_add_func("/migration/fd_proto", test_migrate_fd_proto);
migration_test_add("/migration/fd_proto", test_migrate_fd_proto);
#endif
qtest_add_func("/migration/validate_uuid", test_validate_uuid);
qtest_add_func("/migration/validate_uuid_error", test_validate_uuid_error);
qtest_add_func("/migration/validate_uuid_src_not_set",
test_validate_uuid_src_not_set);
qtest_add_func("/migration/validate_uuid_dst_not_set",
test_validate_uuid_dst_not_set);
migration_test_add("/migration/validate_uuid", test_validate_uuid);
migration_test_add("/migration/validate_uuid_error",
test_validate_uuid_error);
migration_test_add("/migration/validate_uuid_src_not_set",
test_validate_uuid_src_not_set);
migration_test_add("/migration/validate_uuid_dst_not_set",
test_validate_uuid_dst_not_set);
/*
* See explanation why this test is slow on function definition
*/
if (g_test_slow()) {
qtest_add_func("/migration/auto_converge", test_migrate_auto_converge);
migration_test_add("/migration/auto_converge",
test_migrate_auto_converge);
if (g_str_equal(arch, "x86_64") &&
has_kvm && kvm_dirty_ring_supported()) {
qtest_add_func("/migration/dirty_limit", test_migrate_dirty_limit);
migration_test_add("/migration/dirty_limit",
test_migrate_dirty_limit);
}
}
qtest_add_func("/migration/multifd/tcp/plain/none",
test_multifd_tcp_none);
/*
* This test is flaky and sometimes fails in CI and otherwise:
* don't run unless user opts in via environment variable.
*/
if (getenv("QEMU_TEST_FLAKY_TESTS")) {
qtest_add_func("/migration/multifd/tcp/plain/cancel",
migration_test_add("/migration/multifd/tcp/plain/none",
test_multifd_tcp_none);
migration_test_add("/migration/multifd/tcp/plain/cancel",
test_multifd_tcp_cancel);
}
qtest_add_func("/migration/multifd/tcp/plain/zlib",
test_multifd_tcp_zlib);
migration_test_add("/migration/multifd/tcp/plain/zlib",
test_multifd_tcp_zlib);
#ifdef CONFIG_ZSTD
qtest_add_func("/migration/multifd/tcp/plain/zstd",
test_multifd_tcp_zstd);
migration_test_add("/migration/multifd/tcp/plain/zstd",
test_multifd_tcp_zstd);
#endif
#ifdef CONFIG_GNUTLS
qtest_add_func("/migration/multifd/tcp/tls/psk/match",
test_multifd_tcp_tls_psk_match);
qtest_add_func("/migration/multifd/tcp/tls/psk/mismatch",
test_multifd_tcp_tls_psk_mismatch);
migration_test_add("/migration/multifd/tcp/tls/psk/match",
test_multifd_tcp_tls_psk_match);
migration_test_add("/migration/multifd/tcp/tls/psk/mismatch",
test_multifd_tcp_tls_psk_mismatch);
#ifdef CONFIG_TASN1
qtest_add_func("/migration/multifd/tcp/tls/x509/default-host",
test_multifd_tcp_tls_x509_default_host);
qtest_add_func("/migration/multifd/tcp/tls/x509/override-host",
test_multifd_tcp_tls_x509_override_host);
qtest_add_func("/migration/multifd/tcp/tls/x509/mismatch-host",
test_multifd_tcp_tls_x509_mismatch_host);
qtest_add_func("/migration/multifd/tcp/tls/x509/allow-anon-client",
test_multifd_tcp_tls_x509_allow_anon_client);
qtest_add_func("/migration/multifd/tcp/tls/x509/reject-anon-client",
test_multifd_tcp_tls_x509_reject_anon_client);
migration_test_add("/migration/multifd/tcp/tls/x509/default-host",
test_multifd_tcp_tls_x509_default_host);
migration_test_add("/migration/multifd/tcp/tls/x509/override-host",
test_multifd_tcp_tls_x509_override_host);
migration_test_add("/migration/multifd/tcp/tls/x509/mismatch-host",
test_multifd_tcp_tls_x509_mismatch_host);
migration_test_add("/migration/multifd/tcp/tls/x509/allow-anon-client",
test_multifd_tcp_tls_x509_allow_anon_client);
migration_test_add("/migration/multifd/tcp/tls/x509/reject-anon-client",
test_multifd_tcp_tls_x509_reject_anon_client);
#endif /* CONFIG_TASN1 */
#endif /* CONFIG_GNUTLS */
if (g_str_equal(arch, "x86_64") && has_kvm && kvm_dirty_ring_supported()) {
qtest_add_func("/migration/dirty_ring",
test_precopy_unix_dirty_ring);
qtest_add_func("/migration/vcpu_dirty_limit",
test_vcpu_dirty_limit);
migration_test_add("/migration/dirty_ring",
test_precopy_unix_dirty_ring);
migration_test_add("/migration/vcpu_dirty_limit",
test_vcpu_dirty_limit);
}
ret = g_test_run();