The thread-stack represents a thread's current stack. When a thread
exits there can still be many functions on the stack e.g. exit() can be
called many levels deep, so all the callers will never return. To get
that information output, the thread-stack must be flushed.
Previously it was assumed the thread-stack would be flushed when the
struct thread was deleted. With thread ref-counting it is no longer
clear when that will be, if ever. So instead explicitly flush all the
thread-stacks at the end of a session.
Signed-off-by: Adrian Hunter <adrian.hunter@intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1432906425-9911-3-git-send-email-adrian.hunter@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Now it is possible to press CTRL+z at anytime and that will disable the
events being monitored, essentially turning 'top' into 'report', with
pressing CTRL+z again making it enable the events again, returning to
the 'top' behaviour, i.e. dynamic + decaying of older samples.
One may want, for instance, play with:
-d, --delay <n> number of seconds to delay between refreshes
and:
-z, --zero zero history across updates
Plus CTRL+z to see only the events since last zeroing, etc.
Suggested-by: Ingo Molnar <mingo@kernel.org>
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: Borislav Petkov <bp@suse.de>
Cc: David Ahern <dsahern@gmail.com>
Cc: Don Zickus <dzickus@redhat.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/n/tip-zq7tnh5462blt2yda0bcxh5b@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
I get following crash on multiple systems and across several releases
(at least since v3.18).
Core was generated by `/tmp/perf trace sleep 0.2 '.
Program terminated with signal SIGSEGV, Segmentation fault.
#0 perf_mmap__read_head (mm=0x3fff9bf30070) at util/evlist.h:195
195 u64 head = ACCESS_ONCE(pc->data_head);
(gdb) bt
#0 perf_mmap__read_head (mm=0x3fff9bf30070) at util/evlist.h:195
#1 perf_evlist__mmap_read (evlist=0x10027f11910, idx=<optimized out>)
at util/evlist.c:637
#2 0x000000001003ce4c in trace__run (argv=<optimized out>,
argc=<optimized out>, trace=0x3fffd7b28288) at builtin-trace.c:2259
#3 cmd_trace (argc=<optimized out>, argv=<optimized out>,
prefix=<optimized out>) at builtin-trace.c:2799
#4 0x00000000100657b8 in run_builtin (p=0x10176798 <commands+480>, argc=3,
argv=0x3fffd7b2b550) at perf.c:370
#5 0x00000000100063e8 in handle_internal_command (argv=0x3fffd7b2b550, argc=3)
at perf.c:429
#6 run_argv (argv=0x3fffd7b2af70, argcp=0x3fffd7b2af7c) at perf.c:473
#7 main (argc=3, argv=0x3fffd7b2b550) at perf.c:588
The problem seems to be a race condition, when the application has just
exited. Some/all fds associated with the perf-events (tracepoints) go
into a POLLHUP/ POLLERR state and the mmap region associated with those
events are unmapped (in perf_evlist__filter_pollfd()).
But we go back and do a perf_evlist__mmap_read() which assumes that the
mmaps are still valid and we hit the crash.
If the mapping for an event is released, its refcnt is 0 (and ->base
is NULL), so ensure we have non-zero refcount before accessing the map.
Note that perf-record has a similar logic but unlike perf-trace, the
record__mmap_read_all() checks the evlist->mmap[i].base before accessing
the map.
Signed-off-by: Sukadev Bhattiprolu <sukadev@linux.vnet.ibm.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Li Zhang <zhlcindy@linux.vnet.ibm.com>
Link: http://lkml.kernel.org/r/20150612060003.GA19913@us.ibm.com
[ Fixed it up to use atomic_read() ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Speed up the "perf probe --list" by caching the last used debuginfo.
perf probe --list always open and load debuginfo for each entry of probe
list. This takes very a long time.
E.g. with vfs_* events (total 96 probes)
[root@localhost perf]# time ./perf probe -l &> /dev/null
real 0m25.376s
user 0m24.381s
sys 0m1.012s
To solve this issue, this adds debuginfo_cache to cache the
last used debuginfo on memory.
With this fix, the perf-probe --list significantly improves
its speed.
[root@localhost perf]# time ./perf probe -l &> /dev/null
real 0m0.161s
user 0m0.136s
sys 0m0.025s
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Naohiro Aota <naota@elisp.net>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20150617145854.19715.15314.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
When the last part of converted events are blacklisted or out-of-text,
those are skipped and perf probe doesn't show usage examples. This
fixes it to show the example even if the last part of event list is
skipped.
E.g. without this patch, events are added, but suddenly end:
# perf probe vfs_*
vfs_caches_init_early is out of .text, skip it.
vfs_caches_init is out of .text, skip it.
Added new events:
probe:vfs_fallocate (on vfs_*)
probe:vfs_open (on vfs_*)
...
probe:vfs_dentry_acceptable (on vfs_*)
probe:vfs_load_quota_inode (on vfs_*)
#
With this fix:
# perf probe vfs_*
vfs_caches_init_early is out of .text, skip it.
vfs_caches_init is out of .text, skip it.
Added new events:
probe:vfs_fallocate (on vfs_*)
...
probe:vfs_load_quota_inode (on vfs_*)
You can now use it in all perf tools, such as:
perf record -e probe:vfs_load_quota_inode -aR sleep 1
Note that this can be reproduced ONLY IF the vfs_caches_init* is the
last part of matched symbol list. I've checked this happens on
"3.19.0-generic #18-Ubuntu" kernel binary.
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Naohiro Aota <naota@elisp.net>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20150616115057.19906.5502.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Following error occurs when trying to use 'perf report' on x86_64 to
cross analysis a perf.data generated by an old perf on a big-endian
machine:
# perf report
*** Error in `/home/w00229757/perf': free(): invalid next size (fast): 0x00000000032c99f0 ***
======= Backtrace: =========
/lib64/libc.so.6(+0x6eeef)[0x7ff6ff7e2eef]
/lib64/libc.so.6(+0x78cae)[0x7ff6ff7eccae]
/lib64/libc.so.6(+0x79987)[0x7ff6ff7ed987]
/path/to/perf[0x4ac734]
/path/to/perf[0x4ac829]
/path/to/perf(perf_header__process_sections+0x129)[0x4ad2c9]
/path/to/perf(perf_session__read_header+0x2e1)[0x4ad9e1]
/path/to/perf(perf_session__new+0x168)[0x4bd458]
/path/to/perf(cmd_report+0xfa0)[0x43eb70]
/path/to/perf[0x47adc3]
/path/to/perf(main+0x5f6)[0x42fd06]
/lib64/libc.so.6(__libc_start_main+0xf5)[0x7ff6ff795bd5]
/path/to/perf[0x42fe35]
======= Memory map: ========
[SNIP]
The bug is in perf_event__attr_swap(). It swaps all fields in 'struct
perf_event_attr' without checking whether the swapped field exist or
not. In addition, in read_event_desc() allocs memory for attr according
to size read from perf.data.
Therefore, if the perf.data is collected by an old perf (without
aux_watermark, for example), when perf_event__attr_swap() swaping
attr->aux_watermark it destroy malloc's metadata.
This patch introduces boundary checking in perf_event__attr_swap(). It
adds macros bswap_field_64 and bswap_field_32 into
perf_event__attr_swap() to make it only swap exist fields.
Signed-off-by: Wang Nan <wangnan0@huawei.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Zefan Li <lizefan@huawei.com>
Cc: pi3orama@163.com
Link: http://lkml.kernel.org/r/1434534999-85347-1-git-send-email-wangnan0@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Fix perf probe to return an error if no probe is added due to the given
probe point being on the blacklist.
To fix this problem, this moves the blacklist checking to right after
finding symbols/probe-points and marks them as skipped.
If all the symbols are skipped, "perf probe" returns an error as it
fails to find the corresponding probe address.
E.g. currently if a blacklisted probe is given:
# perf probe do_trap && echo 'succeed'
Added new event:
Warning: Skipped probing on blacklisted function: sync_regs
succeed
No! It must fail! With this patch, it correctly fails:
# perf probe do_trap && echo 'succeed'
do_trap is blacklisted function, skip it.
Probe point 'do_trap' not found.
Error: Failed to add events.
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Naohiro Aota <naota@elisp.net>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20150616115055.19906.31359.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Since commit 5e17b28f1e ("perf probe: Add --quiet option to
suppress output result message") have replaced printf with pr_info,
perf probe -l outputs its result in stderr. However, that is not
what the commit expected.
E.g.:
# perf probe -l > /dev/null
probe:vfs_read (on vfs_read@ksrc/linux-3/fs/read_write.c)
With this fix:
# perf probe -l > list
# cat list
probe:vfs_read (on vfs_read@ksrc/linux-3/fs/read_write.c)
Of course, --quiet(-q) still works on --add/--del.
# perf probe -q vfs_write
# perf probe -l
probe:vfs_read (on vfs_read@ksrc/linux-3/fs/read_write.c)
probe:vfs_write (on vfs_write@ksrc/linux-3/fs/read_write.c)
-----
Reported-by: Naohiro Aota <naota@elisp.net>
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Naohiro Aota <naota@elisp.net>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20150613013116.24402.2923.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
'make build-test' finds an error that make_python_perf_so fails due to
missing of libtraceevent-dynamic-list:
'.../python2' util/setup.py \
--quiet build_ext; \
mkdir -p python && \
cp python_ext_build/lib/perf.so python/
/path/to/ld: cannot open linker script file /path/to/kernel/tools/lib/traceevent/libtraceevent-dynamic-list: No such file or directory
collect2: error: ld returned 1 exit status
error: command 'x86_64-linux-gcc' failed with exit status 1
cp: cannot stat 'python_ext_build/lib/perf.so': No such file or directory
make[3]: *** [python/perf.so] Error 1
make[2]: *** [python/perf.so] Error 2
test: test -f ./python/perf.so
make[1]: *** [make_python_perf_so] Error 1
make: *** [build-test] Error 2
make: Leaving directory `/path/to/kernel/tools/perf'
This is caused by commit e3d09ec812
("tools lib traceevent: Export dynamic symbols used by traceevent
plugins") that, it adds the list file to LDFLAGS but forgot to add it to
dependency list of python/perf.so.
This patch fixes this problem.
Signed-off-by: Wang Nan <wangnan0@huawei.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Cc: He Kuang <hekuang@huawei.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Zefan Li <lizefan@huawei.com>
Link: http://lkml.kernel.org/r/1434079031-123162-1-git-send-email-wangnan0@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cut off the postfixes which gcc added for optimized routines from the
event name automatically generated from symbol name, since *probe-events
doesn't accept it. Those symbols will be used if we don't use debuginfo
to find target functions.
E.g. without this fix;
-----
# perf probe -va alloc_buf.isra.23
probe-definition(0): alloc_buf.isra.23
symbol:alloc_buf.isra.23 file:(null) line:0 offset:0 return:0 lazy:(null)
[...]
Opening /sys/kernel/debug/tracing/kprobe_events write=1
Added new event:
Writing event: p:probe/alloc_buf.isra.23 _text+4869328
Failed to write event: Invalid argument
Error: Failed to add events. Reason: Invalid argument (Code: -22)
-----
With this fix;
-----
perf probe -va alloc_buf.isra.23
probe-definition(0): alloc_buf.isra.23
symbol:alloc_buf.isra.23 file:(null) line:0 offset:0 return:0 lazy:(null)
[...]
Opening /sys/kernel/debug/tracing/kprobe_events write=1
Added new event:
Writing event: p:probe/alloc_buf _text+4869328
probe:alloc_buf (on alloc_buf.isra.23)
You can now use it in all perf tools, such as:
perf record -e probe:alloc_buf -aR sleep 1
-----
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Tested-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Naohiro Aota <naota@elisp.net>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20150612050820.20548.41625.stgit@localhost.localdomain
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Failed in 32bit arch build like this:
CC /opt/h00206996/output/perf/arm32/builtin-record.o
util/session.c: In function ‘perf_session__warn_about_errors’:
util/session.c:1304:9: error: format ‘%lu’ expects argument of type ‘long unsigned int’,
but argument 2 has type ‘long long unsigned int’ [-Werror=format=]
builtin-report.c: In function ‘perf_evlist__tty_browse_hists’:
builtin-report.c:323:2: error: format ‘%lu’ expects argument of type ‘long unsigned int’,
but argument 3 has type ‘u64’ [-Werror=format=]
Replace %lu format strings in warning message with PRIu64 for u64
'total_lost_samples' to fix this problem.
Signed-off-by: He Kuang <hekuang@huawei.com>
Cc: Jiri Olsa <jolsa@kernel.org>
Cc: Kan Liang <kan.liang@intel.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Wang Nan <wangnan0@huawei.com>
Link: http://lkml.kernel.org/r/1434026664-71642-1-git-send-email-hekuang@huawei.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Arnaldo Carvalho de Melo
Martin Liška
Adrian Hunter
Sukadev Bhattiprolu
Masami Hiramatsu
Wang Nan
Hou Pengyang
Jiri Olsa
David Ahern
He Kuang