We are using the Python scripting interface in perf to extract kernel
events relevant for performance analysis of HPC codes. We noticed that
the "perf script" call allocates a significant amount of memory (in the
order of several 100 MiB) during it's run, e.g. 125 MiB for a 25 MiB
input file:
$> perf record -o perf.data -a -R -g fp \
-e power:cpu_frequency -e sched:sched_switch \
-e sched:sched_migrate_task -e sched:sched_process_exit \
-e sched:sched_process_fork -e sched:sched_process_exec \
-e cycles -m 4096 --freq 4000
$> /usr/bin/time perf script -i perf.data -s dummy_script.py
0.84user 0.13system 0:01.92elapsed 51%CPU (0avgtext+0avgdata
125532maxresident)k
73072inputs+0outputs (57major+33086minor)pagefaults 0swaps
Upon further investigation using the valgrind massif tool, we noticed
that Python objects that are created in trace-event-python.c via
PyString_FromString*() (and their Integer and Long counterparts) are
never free'd.
The reason for this seem to be missing Py_DECREF calls on the objects
that are returned by these functions and stored in the Python
dictionaries. The Python dictionaries do not steal references (as
opposed to Python tuples and lists) but instead add their own reference.
Hence, the reference that is returned by these object creation functions
is never released and the memory is leaked. (see [1,2])
The attached patch fixes this by wrapping all relevant calls to
PyDict_SetItemString() and decrementing the reference counter
immediately after the Python function call.
This reduces the allocated memory to a reasonable amount:
$> /usr/bin/time perf script -i perf.data -s dummy_script.py
0.73user 0.05system 0:00.79elapsed 99%CPU (0avgtext+0avgdata
49132maxresident)k
0inputs+0outputs (0major+14045minor)pagefaults 0swaps
For comparison, with a 120 MiB input file the memory consumption
reported by time drops from almost 600 MiB to 146 MiB.
The patch has been tested using Linux 3.8.2 with Python 2.7.4 and Linux
3.11.6 with Python 2.7.5.
Please let me know if you need any further information.
[1] http://docs.python.org/2/c-api/tuple.html#PyTuple_SetItem
[2] http://docs.python.org/2/c-api/dict.html#PyDict_SetItemString
Signed-off-by: Joseph Schuchart <joseph.schuchart@tu-dresden.de>
Reviewed-by: Tom Zanussi <tom.zanussi@linux.intel.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Tom Zanussi <tom.zanussi@linux.intel.com>
Link: http://lkml.kernel.org/r/1381468543-25334-4-git-send-email-namhyung@kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Pull perf/urgent fixes from Arnaldo Carvalho de Melo:
" * Fix build error on Fedora 12.
* Fix to initialize fname always before use it, bug introduced
during this merge window, from Masami Hiramatsu.
* Disable PERF_RECORD_MMAP2 support, from Stephane Eranian. "
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
For now, we disable the extended MMAP record support (MMAP2).
We have identified cases where it would not report the correct mapping
information, clone(VM_CLONE) but with separate pids. We will revisit
the support once we find a solution for this case.
The patch changes the kernel to return EINVAL if attr->mmap2 is set. The
patch also modifies the perf tool to use regular PERF_RECORD_MMAP for
synthetic events and it also prevents the tool from requesting
attr->mmap2 mode because the kernel would reject it.
The support will be revisited once the kenrel interface is updated.
In V2, we reduce the patch to the strict minimum.
In V3, we avoid calling perf_event_open() with mmap2 set because we know
it will fail and require fallback retry.
Signed-off-by: Stephane Eranian <eranian@google.com>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Link: http://lkml.kernel.org/r/20131017173215.GA8820@quad
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Fix to find the correct (as much as possible) line information for
listing probes. Without this fix, perf probe --list action will show
incorrect line information as below;
probe:getname_flags (on getname_flags@ksrc/linux-3/fs/namei.c)
probe:getname_flags_1 (on getname:-89@x86/include/asm/current.h)
probe:getname_flags_2 (on user_path_at_empty:-2054@x86/include/asm/current.h)
The minus line number is obviously wrong, and current.h is not related
to the probe point. Deeper investigation discovered that there were 2
issues related to this bug, and minor typos too.
The 1st issue is the rack of considering about nested inlined functions,
which causes the wrong (relative) line number.
The 2nd issue is that the dwarf line info is not correct at those
points. It points 14th line of current.h.
Since it seems that the line info includes somewhat unreliable
information, this fixes perf to try to find correct line information
from both of debuginfo and line info as below.
1) Probe address is the entry of a function instance
In this case, the line is set as the function declared line.
2) Probe address is the entry of an expanded inline function block
In this case, the line is set as the function call-site line.
This means that the line number is relative from the entry line
of caller function (which can be an inlined function if nested)
3) Probe address is inside a function instance or an expanded
inline function block
In this case, perf probe queries the line number from lineinfo
and verify the function declared file is same as the file name
queried from lineinfo.
If the file name is different, it is a failure case. The probe
address is shown as symbol+offset.
4) Probe address is not in the any function instance
This is a failure case, the probe address is shown as
symbol+offset.
With this fix, perf probe -l shows correct probe lines as below;
probe:getname_flags (on getname_flags@ksrc/linux-3/fs/namei.c)
probe:getname_flags_1 (on getname:2@ksrc/linux-3/fs/namei.c)
probe:getname_flags_2 (on user_path_at_empty:4@ksrc/linux-3/fs/namei.c)
Changes at v2:
- Fix typos in the function comments. (Thanks to Namhyung Kim)
- Use die_find_top_inlinefunc instead of die_find_inlinefunc_next.
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Namhyung Kim <namhyung@kernel.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/20130930092144.1693.11058.stgit@udc4-manage.rcp.hitachi.co.jp
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
In ubuntu systems the libaudit test was always failing due to the
newline in the printf call not being escaped, which somehow didn't
prevented the test from working as expected on other systems, such
as fedora18.
Fix it by removing the newline, as this is just a test, that program is
just a compile test.
The error messages, obtained using 'make V=1':
CHK libaudit
<stdin>: In function ‘main’:
<stdin>:5:9: error: missing terminating " character [-Werror]
<stdin>:5:2: error: missing terminating " character
<stdin>:6:1: error: missing terminating " character [-Werror]
<stdin>:6:1: error: missing terminating " character
<stdin>:7:2: error: expected expression before ‘return’
<stdin>:8:1: error: expected ‘;’ before ‘}’ token
cc1: all warnings being treated as errors
config/Makefile:241: No libaudit.h found, disables 'trace' tool, please install audit-libs-devel or libaudit-dev
After this change the test works as expected in all systems tested and the
'trace' tool is built when the needed devel packages are installed.
Cc: Adrian Hunter <adrian.hunter@intel.com>
Cc: David Ahern <dsahern@gmail.com>
Cc: Frederic Weisbecker <fweisbec@gmail.com>
Cc: Jiri Olsa <jolsa@redhat.com>
Cc: Mike Galbraith <efault@gmx.de>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Stephane Eranian <eranian@google.com>
Link: http://lkml.kernel.org/n/tip-0trw8qs9hafeopc0vj1sicay@git.kernel.org
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
This reverts commit de95ab5364.
Markus Trippelsdorf reported that this commit broke 'perf top':
> I just see a gray screen with no text at all. Sometimes the
> following error messages are printed:
>
> *** Error in `perf': invalid fastbin entry (free): 0x00000000029b18c0
> ***
> *** Error in `perf': malloc(): memory corruption (fast): 0x0000000000ee0b10 ***
While this code is fixable, the commit itself fails on several levels:
- it should have been a separate helper function
- why the heck does it do strchr() twice
- it casts a const char * over into char *
- sloppy style
- it's not even a regression fix!
So lets revert it and re-try the patch in v3.13.
Reported-by: Markus Trippelsdorf <markus@trippelsdorf.de>
Cc: Andi Kleen <ak@linux.intel.com>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Cc: Arnaldo Carvalho de Melo <acme@redhat.com>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Fix perf probe to probe on some symbols which have some optimzation
suffixes, e.g. ".part", ".isra", and ".constprop".
To fix this issue, instead of using the DIE name, perf probe uses the
symbol name found by dwfl_module_addrsym().
This also involves a perf probe --vars operation update which now shows
the symbol name instead of the DIE name.
Without this patch, putting a probe on an inlined function which was
compiled with a suffixed symbol will fail like this:
$ perf probe -v getname_flags
probe-definition(0): getname_flags
symbol:getname_flags file:(null) line:0 offset:0 return:0 lazy:(null)
0 arguments
Looking at the vmlinux_path (6 entries long)
Using /lib/modules/3.11.0+/build/vmlinux for symbols
found inline addr: 0xffffffff8119bb70
Probe point found: getname_flags+0
found inline addr: 0xffffffff8119bcb6
Probe point found: getname+6
found inline addr: 0xffffffff811a06a6
Probe point found: user_path_at_empty+6
find 3 probe_trace_events.
Opening /sys/kernel/debug//tracing/kprobe_events write=1
Added new events:
Writing event: p:probe/getname_flags getname_flags+0
Failed to write event: No such file or directory
Error: Failed to add events. (-1)
Because the debuginfo knows only the original (non suffix) symbol name,
it uses the original symbol for probe address but the kernel (kallsyms)
knows only suffixed symbol. Then, the kernel rejects that original
symbol.
This patch uses dwfl_module_addrsym() to get the correct (suffixed)
symbol from symtab when a probe point is found.
Reported-by: Arnaldo Carvalho de Melo <acme@ghostprotocols.net>
Signed-off-by: Masami Hiramatsu <masami.hiramatsu.pt@hitachi.com>
Cc: Ingo Molnar <mingo@redhat.com>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl>
Link: http://lkml.kernel.org/r/20130925131616.31632.46658.stgit@udc4-manage.rcp.hitachi.co.jp
Signed-off-by: Arnaldo Carvalho de Melo <acme@redhat.com>
Solve the problems around the broken definition of perf_event_mmap_page::
cap_usr_time and cap_usr_rdpmc fields which used to overlap, partially
fixed by:
860f085b74 ("perf: Fix broken union in 'struct perf_event_mmap_page'")
The problem with the fix (merged in v3.12-rc1 and not yet released
officially), noticed by Vince Weaver is that the new behavior is
not detectable by new user-space, and that due to the reuse of the
field names it's easy to mis-compile a binary if old headers are used
on a new kernel or new headers are used on an old kernel.
To solve all that make this change explicit, detectable and self-contained,
by iterating the ABI the following way:
- Always clear bit 0, and rename it to usrpage->cap_bit0, to at least not
confuse old user-space binaries. RDPMC will be marked as unavailable
to old binaries but that's within the ABI, this is a capability bit.
- Rename bit 1 to ->cap_bit0_is_deprecated and always set it to 1, so new
libraries can reliably detect that bit 0 is deprecated and perma-zero
without having to check the kernel version.
- Use bits 2, 3, 4 for the newly defined, correct functionality:
cap_user_rdpmc : 1, /* The RDPMC instruction can be used to read counts */
cap_user_time : 1, /* The time_* fields are used */
cap_user_time_zero : 1, /* The time_zero field is used */
- Rename all the bitfield names in perf_event.h to be different from the
old names, to make sure it's not possible to mis-compile it
accidentally with old assumptions.
The 'size' field can then be used in the future to add new fields and it
will act as a natural ABI version indicator as well.
Also adjust tools/perf/ userspace for the new definitions, noticed by
Adrian Hunter.
Reported-by: Vince Weaver <vincent.weaver@maine.edu>
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl>
Also-Fixed-by: Adrian Hunter <adrian.hunter@intel.com>
Link: http://lkml.kernel.org/n/tip-zr03yxjrpXesOzzupszqglbv@git.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
