Commit Graph
1820 Commits
Author SHA1 Message Date
Stefan Hajnoczi 418ed14268 trace: make the 'log' backend timestamp configurable
Timestamps in tracing output can be distracting. Make it possible to
control tid/timestamp printing with -msg timestamp=on|off. The default
is no tid/timestamps. Previously they were always printed.

Suggested-by: BALATON Zoltan <balaton@eik.bme.hu>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: BALATON Zoltan <balaton@eik.bme.hu>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-id: 20210125113507.224287-3-stefanha@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
2021-02-01 10:50:55 +00:00
Daniel P. BerrangéandStefan Hajnoczi 09612de7e9 tracetool: also strip %l and %ll from systemtap format strings
All variables are 64-bit and so %l / %ll are not required, and the
latter is actually invalid:

  $ sudo stap -e 'probe begin{printf ("BEGIN")}'  -I .
  parse error: invalid or missing conversion specifier
          saw: operator ',' at ./qemu-system-x86_64-log.stp:15118:101
       source:     printf("%d@%d vhost_vdpa_set_log_base dev: %p base: 0x%x size: %llu
refcnt: %d fd: %d log: %p\n", pid(), gettimeofday_ns(), dev, base, size, refcnt, fd, log)

                       ^

Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
Reviewed-by: Laurent Vivier <lvivier@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Tested-by: Laurent Vivier <lvivier@redhat.com>
Message-id: 20210106130239.1004729-1-berrange@redhat.com

[Fixed "simiarly" typo found by Laurent Vivier <lvivier@redhat.com>
--Stefan]

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
2021-02-01 10:50:55 +00:00
Laurent VivierandStefan Hajnoczi 9f4e519fd7 tracetool: fix "PRI" macro decoding
macro is not reset after use, so the format decoded is always the
one of the first "PRI" in the format string.

For instance:

  vhost_vdpa_set_config(void *dev, uint32_t offset, uint32_t size, \
                        uint32_t flags) "dev: %p offset: %"PRIu32" \
                        size: %"PRIu32" flags: 0x%"PRIx32

generates:

  printf("%d@%d vhost_vdpa_set_config dev: %p offset: %u size: %u \
          flags: 0x%u\n", pid(), gettimeofday_ns(), dev, offset, \
          size, flags)

for the "flags" parameter, we can see a "0x%u" rather than a "0x%x"
because the first macro was "PRIu32" (for offset).

In the loop, macro becomes "PRIu32PRIu32PRIx32", and c_macro_to_format()
returns always macro[3] ('u' in this case). This patch resets macro after
the format has been decoded.

Signed-off-by: Laurent Vivier <lvivier@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Daniel P. Berrangé <berrange@redhat.com>
Message-id: 20210105191721.120463-3-lvivier@redhat.com
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
2021-02-01 10:50:54 +00:00
Peter Maydell 74208cd252 Merge remote-tracking branch 'remotes/berrange-gitlab/tags/misc-fixes-pull-request' into staging
* Replace --enable/disable-git-update with --with-git-submodules
  to allow improved control over use of git submodules
* Deprecate the -enable-fips option
* Ensure docs use prefer format for bool options
* Clarify platform support rules
* Misc fixes to keymap conversions
* Fix misc problems on macOS

# gpg: Signature made Fri 29 Jan 2021 17:10:13 GMT
# gpg:                using RSA key DAF3A6FDB26B62912D0E8E3FBE86EBB415104FDF
# gpg: Good signature from "Daniel P. Berrange <dan@berrange.com>" [full]
# gpg:                 aka "Daniel P. Berrange <berrange@redhat.com>" [full]
# Primary key fingerprint: DAF3 A6FD B26B 6291 2D0E  8E3F BE86 EBB4 1510 4FDF

* remotes/berrange-gitlab/tags/misc-fixes-pull-request:
  tests: Replace deprecated ASN1 code
  tests: Fix runtime error in test-authz-pam
  ui: update keycodemapdb submodule commit
  crypto: Add spaces around operator
  configure: replace --enable/disable-git-update with --with-git-submodules
  docs: fix missing backslash in certtool shell example
  docs: simplify and clarify the platform support rules
  Prefer 'on' | 'off' over 'yes' | 'no' for bool options
  os: deprecate the -enable-fips option and QEMU's FIPS enforcement
  crypto: Fix memory leaks in set_loaded for tls-*
  crypto: Forbid broken unloading of secrets
  crypto: Move USER_CREATABLE to secret_common base class
  crypto: Fix some code style problems, add spaces around operator

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2021-01-29 19:51:25 +00:00
Dan StreetmanandDaniel P. Berrangé 7d7dbf9dc1 configure: replace --enable/disable-git-update with --with-git-submodules
Replace the --enable-git-update and --disable-git-update configure params
with the param --with-git-submodules=(update|validate|ignore) to
allow 3 options for building from a git repo.

This is needed because downstream packagers, e.g. Debian, Ubuntu, etc,
also keep the source code in git, but do not want to enable the
'git_update' mode; with the current code, that's not possible even
if the downstream package specifies --disable-git-update.

The previous parameters are deprecated but still available; the
--enable-git-update parameter maps to --with-git-submodules=update and
--disable-git-update parameter maps to --with-git-submodules=validate.

The configure script behavior is slightly modified, where previously
the dtc, capstone, and slirp submodules were not validated when
--disable-git-update was specified (but were updated with git-update
enabled), now they are validated when using --with-git-submodules=validate
and are only ignored when using --with-git-submodules=ignore.

Signed-off-by: Dan Streetman <ddstreet@canonical.com>
Signed-off-by: Daniel P. Berrangé <berrange@redhat.com>
2021-01-29 17:07:53 +00:00
Alexander GrafandPeter Maydell 8a74ce618b hvf: Add hypervisor entitlement to output binaries
In macOS 11, QEMU only gets access to Hypervisor.framework if it has the
respective entitlement. Add an entitlement template and automatically self
sign and apply the entitlement in the build.

Signed-off-by: Alexander Graf <agraf@csgraf.de>
Reviewed-by: Roman Bolshakov <r.bolshakov@yadro.com>
Tested-by: Roman Bolshakov <r.bolshakov@yadro.com>
Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
2021-01-29 10:47:28 +00:00
Vladimir Sementsov-OgievskiyandMax Reitz c701f59253 simplebench: add bench-backup.py
Add script to benchmark new backup architecture.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Message-Id: <20210116214705.822267-24-vsementsov@virtuozzo.com>
[mreitz: s/not unsupported/not supported/]
Signed-off-by: Max Reitz <mreitz@redhat.com>
2021-01-26 14:36:37 +01:00
Vladimir Sementsov-OgievskiyandMax Reitz b2fcb0c575 simplebench: bench_block_job: add cmd_options argument
Add argument to allow additional block-job options.

Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210116214705.822267-23-vsementsov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2021-01-26 14:36:37 +01:00
Vladimir Sementsov-OgievskiyandMax Reitz 2096de521e simplebench/bench_block_job: use correct shebang line with python3
Signed-off-by: Vladimir Sementsov-Ogievskiy <vsementsov@virtuozzo.com>
Reviewed-by: Max Reitz <mreitz@redhat.com>
Message-Id: <20210116214705.822267-22-vsementsov@virtuozzo.com>
Signed-off-by: Max Reitz <mreitz@redhat.com>
2021-01-26 14:36:37 +01:00
Eric FarmanandCornelia Huck ab5ec23f9c update-linux-headers: Include const.h
Kernel commit a85cbe6159ff ("uapi: move constants from
<linux/kernel.h> to <linux/const.h>") breaks our script
because of the unrecognized include. Let's add that to
our processing.

Signed-off-by: Eric Farman <farman@linux.ibm.com>
Reviewed-by: Peter Xu <peterx@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210104202057.48048-2-farman@linux.ibm.com>
Signed-off-by: Cornelia Huck <cohuck@redhat.com>
2021-01-21 11:19:45 +01:00
Alex Bennée 9e5acb373d scripts/checkpatch.pl: fix git-show invocation to include diffstat
Without this checkpatch keeps complaining about new/changed files even
when MAINTAINERS has been updated. Normal invocations of checkpatch on
patch files rather than commit IDs are unaffected.

Signed-off-by: Alex Bennée <alex.bennee@linaro.org>
Tested-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Willian Rampazzo <willianr@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Message-Id: <20210114165730.31607-13-alex.bennee@linaro.org>
2021-01-18 10:04:31 +00:00
Philippe Mathieu-DaudéandRichard Henderson 4cacecaaa2 decodetree: Open files with encoding='utf-8'
When decodetree.py was added in commit 568ae7efae, QEMU was
using Python 2 which happily reads UTF-8 files in text mode.
Python 3 requires either UTF-8 locale or an explicit encoding
passed to open(). Now that Python 3 is required, explicit
UTF-8 encoding for decodetree source files.

To avoid further problems with the user locale, also explicit
UTF-8 encoding for the generated C files.

Explicit both input/output are plain text by using the 't' mode.

This fixes:

  $ /usr/bin/python3 scripts/decodetree.py test.decode
  Traceback (most recent call last):
    File "scripts/decodetree.py", line 1397, in <module>
      main()
    File "scripts/decodetree.py", line 1308, in main
      parse_file(f, toppat)
    File "scripts/decodetree.py", line 994, in parse_file
      for line in f:
    File "/usr/lib/python3.6/encodings/ascii.py", line 26, in decode
      return codecs.ascii_decode(input, self.errors)[0]
  UnicodeDecodeError: 'ascii' codec can't decode byte 0xc3 in position 80:
  ordinal not in range(128)

Reported-by: Peter Maydell <peter.maydell@linaro.org>
Suggested-by: Yonggang Luo <luoyonggang@gmail.com>
Reviewed-by: Eduardo Habkost <ehabkost@redhat.com>
Signed-off-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Message-Id: <20210110000240.761122-1-f4bug@amsat.org>
Signed-off-by: Richard Henderson <richard.henderson@linaro.org>
2021-01-13 08:39:08 -10:00
Maxim LevitskyandPaolo Bonzini b9a0de3773 scripts/gdb: implement 'qemu bt'
This script first runs the regular gdb's 'bt' command, and then if we are in a
coroutine it prints the coroutines backtraces in the order in which they
were called.

Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20201217155436.927320-3-mlevitsk@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-01-12 12:38:03 +01:00
Maxim LevitskyandPaolo Bonzini 4cbf8efc5b scripts/gdb: fix 'qemu coroutine' when users selects a non topmost stack frame
The code that dumps the stack frame works like that:
* save current registers
* overwrite current registers (including rip/rsp) with coroutine snapshot
  in the jmpbuf
* print backtrace
* restore the saved registers.

If the user has currently selected a non topmost stack frame in gdb,
the above code will still restore the selected frame registers,
but the gdb will then lose the selected frame index, which makes it impossible
to switch back to frame 0, to continue debugging the executable.

Therefore switch temporarily to the topmost frame of the stack
for the above code.

Signed-off-by: Maxim Levitsky <mlevitsk@redhat.com>
Message-Id: <20201217155436.927320-2-mlevitsk@redhat.com>
Signed-off-by: Paolo Bonzini <pbonzini@redhat.com>
2021-01-12 12:38:03 +01:00
Qiuhao LiandThomas Huth 4cc5752303 fuzz: heuristic split write based on past IOs
If previous write commands write the same length of data with the same step,
we view it as a hint.

Signed-off-by: Qiuhao Li <Qiuhao.Li@outlook.com>
Reviewed-by: Alexander Bulekov <alxndr@bu.edu>
Tested-by: Alexander Bulekov <alxndr@bu.edu>
Message-Id: <SYCPR01MB3502480AD07811A6A49B8FEAFCAB0@SYCPR01MB3502.ausprd01.prod.outlook.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-01-11 14:59:21 +01:00
Qiuhao LiandThomas Huth dd21ed0edf fuzz: add minimization options
-M1: remove IO commands iteratively
-M2: try setting bits in operand of write/out to zero

Signed-off-by: Qiuhao Li <Qiuhao.Li@outlook.com>
Reviewed-by: Alexander Bulekov <alxndr@bu.edu>
Tested-by: Alexander Bulekov <alxndr@bu.edu>
Message-Id: <SYCPR01MB350204C52E7A39E6B0EEC870FCAB0@SYCPR01MB3502.ausprd01.prod.outlook.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-01-11 14:59:21 +01:00
Qiuhao LiandThomas Huth 9d20f2af53 fuzz: set bits in operand of write/out to zero
Simplifying the crash cases by opportunistically setting bits in operands of
out/write to zero may help to debug, since usually bit one means turn on or
trigger a function while zero is the default turn-off setting.

Tested bug https://bugs.launchpad.net/qemu/+bug/1908062

Signed-off-by: Qiuhao Li <Qiuhao.Li@outlook.com>
Reviewed-by: Alexander Bulekov <alxndr@bu.edu>
Tested-by: Alexander Bulekov <alxndr@bu.edu>
Message-Id: <SYCPR01MB3502C84B6346A3E3DE708C7BFCAB0@SYCPR01MB3502.ausprd01.prod.outlook.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-01-11 14:59:21 +01:00
Qiuhao LiandThomas Huth 247ab240c2 fuzz: remove IO commands iteratively
Now we use a one-time scan and remove strategy in the minimizer,
which is not suitable for timing dependent instructions.

For example, instruction A will indicate an address where the config
chunk locates, and instruction B will make the configuration active.
If we have the following instruction sequence:

...
A1
B1
A2
B2
...

A2 and B2 are the actual instructions that trigger the bug.

If we scan from top to bottom, after we remove A1, the behavior of B1
might be unknowable, including not to crash the program. But we will
successfully remove B1 later cause A2 and B2 will crash the process
anyway:

...
A1
A2
B2
...

Now one more trimming will remove A1.

In the perfect case, we would need to be able to remove A and B (or C!) at
the same time. But for now, let's just add a loop around the minimizer.

Since we only remove instructions, this iterative algorithm is converging.

Tested with Bug 1908062.

Signed-off-by: Qiuhao Li <Qiuhao.Li@outlook.com>
Reviewed-by: Alexander Bulekov <alxndr@bu.edu>
Tested-by: Alexander Bulekov <alxndr@bu.edu>
Message-Id: <SYCPR01MB350263004448040ACCB9A9F1FCAB0@SYCPR01MB3502.ausprd01.prod.outlook.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-01-11 14:59:21 +01:00
Qiuhao LiandThomas Huth e72203abec fuzz: split write operand using binary approach
Currently, we split the write commands' data from the middle. If it does not
work, try to move the pivot left by one byte and retry until there is no
space.

But, this method has two flaws:

1. It may fail to trim all unnecessary bytes on the right side.

For example, there is an IO write command:

  write addr uuxxxxuu

u is the unnecessary byte for the crash. Unlike ram write commands, in most
case, a split IO write won't trigger the same crash, So if we split from the
middle, we will get:

  write addr uu (will be removed in next round)
  write addr xxxxuu

For xxxxuu, since split it from the middle and retry to the leftmost byte
won't get the same crash, we will be stopped from removing the last two
bytes.

2. The algorithm complexity is O(n) since we move the pivot byte by byte.

To solve the first issue, we can try a symmetrical position on the right if
we fail on the left. As for the second issue, instead moving by one byte, we
can approach the boundary exponentially, achieving O(log(n)).

Give an example:

                   xxxxuu len=6
                        +
                        |
                        +
                 xxx,xuu 6/2=3 fail
                        +
         +--------------+-------------+
         |                            |
         +                            +
  xx,xxuu 6/2^2=1 fail         xxxxu,u 6-1=5 success
                                 +   +
         +------------------+----+   |
         |                  |        +-------------+ u removed
         +                  +
   xx,xxu 5/2=2 fail  xxxx,u 6-2=4 success
                           +
                           |
                           +-----------+ u removed

In some rare cases, this algorithm will fail to trim all unnecessary bytes:

  xxxxxxxxxuxxxxxx
  xxxxxxxx-xuxxxxxx Fail
  xxxx-xxxxxuxxxxxx Fail
  xxxxxxxxxuxx-xxxx Fail
  ...

I think the trade-off is worth it.

Signed-off-by: Qiuhao Li <Qiuhao.Li@outlook.com>
Reviewed-by: Alexander Bulekov <alxndr@bu.edu>
Tested-by: Alexander Bulekov <alxndr@bu.edu>
Message-Id: <SYCPR01MB3502D26F1BEB680CBBC169E5FCAB0@SYCPR01MB3502.ausprd01.prod.outlook.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-01-11 14:59:21 +01:00
Qiuhao LiandThomas Huth 7b339f287f fuzz: double the IOs to remove for every loop
Instead of removing IO instructions one by one, we can try deleting multiple
instructions at once. According to the locality of reference, we double the
number of instructions to remove for the next round and recover it to one
once we fail.

This patch is usually significant for large input.

Test with quadrupled trace input at:
  https://bugs.launchpad.net/qemu/+bug/1890333/comments/1

Patched 1/6 version:
  real  0m45.904s
  user  0m16.874s
  sys   0m10.042s

Refined version:
  real  0m11.412s
  user  0m6.888s
  sys   0m3.325s

Signed-off-by: Qiuhao Li <Qiuhao.Li@outlook.com>
Reviewed-by: Alexander Bulekov <alxndr@bu.edu>
Tested-by: Alexander Bulekov <alxndr@bu.edu>
Message-Id: <SYCPR01MB350280A67BB55C3FADF173E3FCAB0@SYCPR01MB3502.ausprd01.prod.outlook.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-01-11 14:59:21 +01:00
Qiuhao LiandThomas Huth 22ec0c696f fuzz: accelerate non-crash detection
We spend much time waiting for the timeout program during the minimization
process until it passes a time limit. This patch hacks the CLOSED (indicates
the redirection file closed) notification in QTest's output if it doesn't
crash.

Test with quadrupled trace input at:
  https://bugs.launchpad.net/qemu/+bug/1890333/comments/1

Original version:
  real	1m37.246s
  user	0m13.069s
  sys	0m8.399s

Refined version:
  real	0m45.904s
  user	0m16.874s
  sys	0m10.042s

Note:

Sometimes the mutated or the same trace may trigger a different crash
summary (second-to-last line) but indicates the same bug. For example, Bug
1910826 [1], which will trigger a stack overflow, may output summaries
like:

SUMMARY: AddressSanitizer: stack-overflow
/home/qiuhao/hack/qemu/build/../softmmu/physmem.c:488 in
flatview_do_translate

or

SUMMARY: AddressSanitizer: stack-overflow
(/home/qiuhao/hack/qemu/build/qemu-system-i386+0x27ca049) in __asan_memcpy

Etc.

If we use the whole summary line as the token, we may be prevented from
further minimization. So in this patch, we only use the first three words
which indicate the type of crash:

SUMMARY: AddressSanitizer: stack-overflow

[1] https://bugs.launchpad.net/qemu/+bug/1910826

Signed-off-by: Qiuhao Li <Qiuhao.Li@outlook.com>
Reviewed-by: Alexander Bulekov <alxndr@bu.edu>
Tested-by: Alexander Bulekov <alxndr@bu.edu>
Message-Id: <SYCPR01MB350251DC04003450348FAF68FCAB0@SYCPR01MB3502.ausprd01.prod.outlook.com>
Signed-off-by: Thomas Huth <thuth@redhat.com>
2021-01-11 14:59:21 +01:00
Peter Maydell 01b3e68bb1 scripts/coccinelle: New script to remove unnecessary timer_del() calls
Now that timer_free() implicitly calls timer_del(), sequences
  timer_del(mytimer);
  timer_free(mytimer);

can be simplified to just
  timer_free(mytimer);

Add a Coccinelle script to do this transformation.

Signed-off-by: Peter Maydell <peter.maydell@linaro.org>
Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <philmd@redhat.com>
Reviewed-by: Richard Henderson <richard.henderson@linaro.org>
Message-id: 20201215154107.3255-3-peter.maydell@linaro.org
2021-01-08 15:13:38 +00:00
Stefan Hajnoczi 7fb48c0ee1 tracetool: show trace-events filename/lineno in fmt string errors
The compiler encounters trace event format strings in generated code.
Format strings are error-prone and therefore clear compiler errors are
important.

Use the #line directive to show the trace-events filename and line
number in format string errors:
https://gcc.gnu.org/onlinedocs/gcc-10.2.0/cpp/Line-Control.html

For example, if the cpu_in trace event's %u is changed to %p the
following error is reported:

  trace-events:29:18: error: format ‘%p’ expects argument of type ‘void *’, but argument 7 has type ‘unsigned int’ [-Werror=format=]

Line 29 in trace-events is where cpu_in is defined. This works for any
trace-events file in the QEMU source tree and the correct path is
displayed.

Unfortunately there does not seem to be a way to set the column, so "18"
is not the right character on that line.

Suggested-by: Peter Maydell <peter.maydell@linaro.org>
Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Message-Id: <20200827142915.108730-5-stefanha@redhat.com>
2021-01-04 14:24:58 +00:00
Stefan Hajnoczi 4e66c9ef64 tracetool: add input filename and line number to Event
Store the input filename and line number in Event.

A later patch will use this to improve error messages.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Philippe Mathieu-Daudé <f4bug@amsat.org>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Message-Id: <20200827142915.108730-4-stefanha@redhat.com>
2021-01-04 14:24:58 +00:00
Stefan Hajnoczi 294170c1dd tracetool: add out_lineno and out_next_lineno to out()
Make the output file line number and next line number available to
out().

A later patch will use this to improve error messages.

Signed-off-by: Stefan Hajnoczi <stefanha@redhat.com>
Reviewed-by: Peter Maydell <peter.maydell@linaro.org>
Message-Id: <20200827142915.108730-3-stefanha@redhat.com>
2021-01-04 14:24:58 +00:00