When processing events the session code has an ordered samples queue
which is used to time-sort events coming in across multiple mmaps. At a
later point in time samples on the queue are flushed up to some
timestamp at which point the event is actually processed.
When analyzing events live (ie., record/analysis path in the same
command) there is a race that leads to corrupted events and parse errors
which cause perf to terminate. The problem is that when the event is
placed in the ordered samples queue it is only a reference to the event
which is really sitting in the mmap buffer. Even though the event is
queued for later processing the mmap tail pointer is updated which
indicates to the kernel that the event has been processed. The race is
flushing the event from the queue before it gets overwritten by some
other event. For commands trying to process events live (versus just
writing to a file) and processing a high rate of events this leads to
parse failures and perf terminates.
Examples hitting this problem are 'perf kvm stat live', especially with
nested VMs which generate 100,000+ traces per second, and a command
processing scheduling events with a high rate of context switching --
e.g., running 'perf bench sched pipe'.
This patch offers live commands an option to copy the event when it is
placed in the ordered samples queue.
Based on a patch from David Ahern <dsahern@gmail.com>
Signed-off-by: Alexander Yarygin <yarygin@linux.vnet.ibm.com>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Cc: Christian Borntraeger <borntraeger@de.ibm.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Namhyung Kim <namhyung.kim@lge.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/1412347212-28237-2-git-send-email-yarygin@linux.vnet.ibm.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
The unw_addr_space_t in libunwind represents an address space to be used
for stack unwinding. It doesn't need to be create/destory everytime to
unwind callchain (as in get_entries) and can have a same lifetime as
thread (unless exec called).
So move the address space construction/destruction logic to the thread
lifetime handling functions. This is a preparation to enable caching in
the unwind library.
Note that it saves unw_addr_space_t object using thread__set_priv(). It
seems currently only used by perf trace and perf kvm stat commands which
don't use callchain.
Signed-off-by: Namhyung Kim <namhyung@kernel.org>
Acked-by: Jean Pihet <jean.pihet@linaro.org>
Acked-by: Jiri Olsa <jolsa@kernel.org>
Cc: Arun Sharma <asharma@fb.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Jean Pihet <jean.pihet@linaro.org>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung.kim@lge.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/1412556363-26229-3-git-send-email-namhyung@kernel.org
[ Fixup unwind-libunwind.c missing CALLCHAIN_DWARF definition, added
missing __maybe_unused on unused parameters in stubs at util/unwind.h ]
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Add test case in automated tests suite. It checks not only the two types
of pmu event stytle formats "pmu_event_name" and "cpu/pmu_event_name/",
but also the different formats mixtures which are more likely to trigger
parse issue.
The patch set including this one has been tested by the perf automated
test:
./perf test parse -v"
On haswell, ivybridge and Romley platform.
The patch set also has been tested on haswell by the following script.
Note: please make sure that your test system support TSX and
L1-dcache-loads events. Otherwise, you may want to change the events to
other pmu events.
[lk@localhost ~]$ cat perf_style_test.sh
# hardware events + kernel pmu event with different style
perf stat -x, -e cycles,mem-stores,tx-start sleep 2
perf stat -x, -e cpu-cycles,cycles-ct,cycles-t sleep 2
perf stat -x, -e cycles,cpu/cycles-ct/,cpu/cycles-t/ sleep 2
perf stat -x, -e instructions,cpu/tx-start/ sleep 2
perf stat -x, -e '{cycles,tx-start}' sleep 2
perf stat -x, -e '{cycles,cpu/tx-start/}' sleep 2
# HW Cache event + kernel pmu event with different style
perf stat -x, -e L1-dcache-loads,cpu/mem-stores/,tx-start sleep 2
perf stat -x, -e L1-dcache-loads,mem-stores,cpu/tx-start/ sleep 2
perf stat -x, -e '{L1-dcache-loads,mem-stores}' sleep 2
perf stat -x, -e '{L1-dcache-loads,cpu/tx-start/}' sleep 2
# Raw event + kernel pmu event with different style:
perf stat -x, -e cpu/event=0xc0,umask=0x00/,mem-loads,cpu/mem-stores/ sleep 2
perf stat -x, -e cpu/event=0xc0,umask=0x00/,tx-start,cpu/el-start/ sleep 2
perf stat -x, -e '{cpu/event=0xc0,umask=0x00/,tx-start}' sleep 2
Signed-off-by: Kan Liang <kan.liang@intel.com>
Acked-by: Jiri Olsa <jolsa@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1412694532-23391-5-git-send-email-kan.liang@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Add new rules for kernel PMU event.
Currently, the patch only want to handle the PMU event name as "a-b" and
"a".
event_pmu:
PE_KERNEL_PMU_EVENT sep_dc
|
PE_PMU_EVENT_PRE '-' PE_PMU_EVENT_SUF sep_dc
PE_KERNEL_PMU_EVENT token is for
cycles-ct/cycles-t/mem-loads/mem-stores.
The prefix cycles is mixed up with cpu-cycles. loads and stores are
mixed up with cache event So they have to be hardcode in lex.
PE_PMU_EVENT_PRE and PE_PMU_EVENT_SUF tokens are for other PMU events.
The lex looks generic identifier up in the table and return the matched
token. If there is no match, generic PE_NAME token will be return.
Using the rules, kernel PMU event could use new style format without //
so you can use:
perf record -e mem-loads ...
instead of:
perf record -e cpu/mem-loads/
Signed-off-by: Kan Liang <kan.liang@intel.com>
Acked-by: Jiri Olsa <jolsa@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1412694532-23391-4-git-send-email-kan.liang@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
There are two types of event formats for PMU events. E.g. el-abort OR
cpu/el-abort/. However, the lexer mistakenly recognizes the simple style
format as two events.
The parse_events_pmu_check function uses bsearch to search the name in
known pmu event list. It can tell the lexer that the name is a PE_NAME
or a PMU event name prefix or a PMU event name suffix. All these
information will be used for accurately parsing kernel PMU events.
The pmu events list will be read from sysfs at runtime.
Note: Currently, the patch only want to handle the PMU event name as
"a-b" and "a". The only exception, "stalled-cycles-frontend" and
"stalled-cycles-fronted", are already hardcoded in lexer.
Signed-off-by: Kan Liang <kan.liang@intel.com>
Acked-by: Jiri Olsa <jolsa@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1412694532-23391-3-git-send-email-kan.liang@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
This reverts commit 50e200f079 ("perf tools: Default to cpu// for
events v5")
The fixup cannot handle the case that
new style format(which without //) mixed with
other different formats.
For example,
group events with new style format: {mem-stores,mem-loads}
some hardware event + new style event: cycles,mem-loads
Cache event + new style event: LLC-loads,mem-loads
Raw event + new style event:
cpu/event=0xc8,umask=0x08/,mem-loads
old style event and new stytle mixture: mem-stores,cpu/mem-loads/
Signed-off-by: Kan Liang <kan.liang@intel.com>
Acked-by: Jiri Olsa <jolsa@redhat.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Link: http://lkml.kernel.org/r/1412694532-23391-2-git-send-email-kan.liang@intel.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
When 'perf top' is run, one can't easily find a difference
between -z option and normal output.
So I added a visual cue to know whether it is the zeroing or not.
Output is as below.
Before:
$ perf top
Samples: 61K of event 'cycles', Event count (approx.): 3908136933
Overhead Shared Object Symbol
1.42% firefox [.] 0x0000000000011e76
1.32% libpthread-2.17.so [.] pthread_mutex_lock
If you press key 'z' or run with zero option like '$ perf top --zero', it is as below.
After:
Samples: 61K of event 'cycles', Event count (approx.): 3908136933 [z]
Overhead Shared Object Symbol
1.42% firefox [.] 0x0000000000011e76
1.32% libpthread-2.17.so [.] pthread_mutex_lock
Signed-off-by: Taeung Song <treeze.taeung@gmail.com>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/r/1412665995-26359-1-git-send-email-treeze.taeung@gmail.com
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Pull perf/core improvements and fixes from Arnaldo Carvalho de Melo:
Infrastructure fixes and changes:
* Fix off-by-one bugs in map->end handling (Stephane Eranian)
* Fix off-by-one bug in maps__find(), also related to map->end handling (Namhyung Kim)
* Make struct symbol->end be the first addr after the symbol range, to make it
match the convention used for struct map->end. (Arnaldo Carvalho de Melo)
* Fix perf_evlist__add_pollfd() error handling in 'perf kvm stat live' (Jiri Olsa)
* Fix python test build by moving callchain_param to an object linked into the
python binding (Jiri Olsa)
* Do not include a struct hists per perf_evsel, untangling the histogram code
from perf_evsel, to pave the way for exporting a minimalistic
tools/lib/api/perf/ library usable by tools/perf and initially by the rasd
daemon being developed by Borislav Petkov, Robert Richter and Jean Pihet.
(Arnaldo Carvalho de Melo)
* Make perf_evlist__open(evlist, NULL, NULL), i.e. without cpu and thread
maps mean syswide monitoring, reducing the boilerplate for tools that
only want system wide mode. (Arnaldo Carvalho de Melo)
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
This patch fixes off-by-one errors in the management of maps.
A map is defined by start address and length as implemented by
map__new():
map__init(map, type, start, start + len, pgoff, dso);
map->start = addr;
map->end = end;
Consequently, the actual address range is [start; end[ map->end is the
first byte outside the range.
This patch fixes two bugs where upper bound checking was off-by-one.
In V2, we fix map_groups__fixup_overlappings() some more where
map->start was off-by-one as reported by Jiri.
Signed-off-by: Stephane Eranian <eranian@google.com>
Acked-by: Namhyung Kim <namhyung@kernel.org>
Cc: David Ahern <dsahern@gmail.com>
Cc: Ingo Molnar <mingo@kernel.org>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20141006083532.GA4850@quad
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
