Cache the dont_correlate() result once per symbol at the start of
correlate_symbols(). This reduces klp diff time on an arm64 LTO
vmlinux.o from 2m51s to 35s.
Acked-by: Song Liu <song@kernel.org>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Add debugging messages to show how duplicate symbols get correlated, and
split the --debug feature into --debug-correlate and --debug-clone.
Acked-by: Song Liu <song@kernel.org>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Rewrite the symbol correlation code, using a tiered list of
deterministic strategies in a loop. For duplicately named symbols, each
tier applies a filter with the goal of finding a 1:1 deterministic
correlation between the original and patched version of the symbol.
The three matching strategies are:
find_twin(): A funnel of progressively tighter filters. Candidates
with the same demangled name are counted at four levels: name, scope
(local-vs-global), file (strict file association), and checksum
(unchanged functions). The widest level that yields a 1:1 match wins,
narrower levels are only tried when the wider level is ambiguous.
find_twin_suffixed(): Uses already-correlated LLVM symbol pairs to map
.llvm.<hash> suffixes from orig to patched. Because all promoted
symbols from the same TU share the same hash, one correlated pair
seeds the mapping for the entire TU.
find_twin_positional(): Last resort, matches symbols by position among
same-named candidates, similar to livepatch sympos. Used for data
objects like __quirk variables where no deterministic filter can
distinguish the candidates.
Overall this works much better than the existing algorithm, particularly
with LTO kernels.
Acked-by: Song Liu <song@kernel.org>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Start checksumming data objects in preparation for revamping the
correlation algorithm.
Acked-by: Song Liu <song@kernel.org>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Move the checksum functionality out of the main objtool command into a
new "objtool klp checksum" subcommand.
This has the benefit of making the code (and the patch generation
process itself) more modular.
For bisectability, both "objtool --checksum" and "objtool klp checksum"
work for now. The former will be removed after klp-build has been
converted to use the new subcommand.
Acked-by: Song Liu <song@kernel.org>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Clang generates anonymous data sections named .data..Lanon.<hash>.
These need section-symbol references in the same way as .data..Lubsan
(GCC) and .data..L__unnamed_ (Clang UBSAN) sections. Without this,
convert_reloc_sym() fails when processing relocations that reference
these sections.
Acked-by: Song Liu <song@kernel.org>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Use the section symbol's index instead of the old symbol's index when
updating the ELF relocation entry in convert_reloc_sym_to_secsym().
Found by Sashiko review.
Acked-by: Song Liu <song@kernel.org>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Clang aggregates UBSAN type descriptors into shared anonymous
.data..L__unnamed_* sections. This data is used by UBSAN trap handlers.
When a changed function has an UBSAN bounds check, klp-diff clones the
entire UBSAN data section associated with the TU. Relocations within
the cloned section that reference named rodata objects in .rodata.cst*
(like 'exponent', 'pirq_ali_set.irqmap') become KLP relocations because
those objects now get correlated.
That results in a .klp.rela.vmlinux..data section which can easily have
thousands of KLP relocs, most of which are completely superfluous, used
by functions which aren't cloned to the patch module.
The .rodata.cst* sections are SHF_MERGE constant pool sections
containing small fixed-size data (lookup tables, bitmasks) that is only
read by value. Pointer identity is never relevant for these objects, so
correlating them is unnecessary.
Exclude .rodata.cst* objects from correlation so they get cloned as
local data instead of generating KLP relocations.
It might be possible to someday treat UBSAN data sections as special
sections, and only extract the few needed entries. But this works for
now.
Acked-by: Song Liu <song@kernel.org>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
klp-diff treats all rodata as uncorrelated, so any reference to it uses
a duplicated copy rather than using a KLP reloc.
For the contents of the data itself, a duplicated copy is fine.
However, pointer comparisons (e.g., f->f_op == &foo_ops) are broken.
Fix it by correlating non-anonymous rodata objects.
Also, use a new find_symbol_containing_inclusive() helper for matching
the end of a symbol so bounds calculations don't get broken, for the
case where an array or other symbol's ending address is used as part of
a bounds calculation.
While these are really two distinct changes, they need to be done in the
same patch so as to avoid introducing bisection regressions.
Acked-by: Song Liu <song@kernel.org>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Inline section_reference_needed() and is_reloc_allowed() into
convert_reloc_sym() and remove the redundant is_reloc_allowed() check in
clone_reloc().
Move the is_sec_sym() checks into the convert callees so they become
no-ops when the reloc is already in the right format. This allows
convert_reloc_sym() to unconditionally dispatch to the right converter
based on section type.
Acked-by: Song Liu <song@kernel.org>
Reviewed-by: Miroslav Benes <mbenes@suse.cz>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Objtool has some hacks which NOP out certain calls/jumps and replace
their relocations with R_X86_64_NONE. The klp-diff relocation
extraction code will error out when trying to copy these relocations due
to their negative addend, which would only makes sense for a PC-relative
branch instruction. Just ignore them.
Fixes: dd590d4d57 ("objtool/klp: Introduce klp diff subcommand for diffing object files")
Acked-by: Song Liu <song@kernel.org>
Reviewed-by: Miroslav Benes <mbenes@suse.cz>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
.kcfi_traps contains references to kCFI trap instruction locations.
When a KCFI type check fails at an indirect call, the trap handler looks
up the faulting address in this section.
Add it to the special sections list so the entries get extracted for the
changed functions they reference.
Acked-by: Song Liu <song@kernel.org>
Reviewed-by: Miroslav Benes <mbenes@suse.cz>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Objtool is failing to extract text annotations which reference
.altinstr_replacement instructions:
1) Alternative replacement fake symbols are NOTYPE rather than FUNC,
and they don't have sym->included set, thus they aren't recognized
by should_keep_special_sym().
2) .discard.annotate_insn gets processed before .altinstr_replacement,
so the referenced (fake) symbols don't have clones yet.
Fix the first issue by checking for a valid clone instead of
sym->included and by accepting NOTYPE symbols when processing
.discard.annotate_insn.
Fix the second issue by deferring text annotation processing until after
the other special sections have been cloned.
Fixes: dd590d4d57 ("objtool/klp: Introduce klp diff subcommand for diffing object files")
Acked-by: Song Liu <song@kernel.org>
Reviewed-by: Miroslav Benes <mbenes@suse.cz>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Fix NULL dereference when cloning a symbol from an empty section.
sec->data is only populated for sections with non-zero size.
Fixes: dd590d4d57 ("objtool/klp: Introduce klp diff subcommand for diffing object files")
Acked-by: Song Liu <song@kernel.org>
Reviewed-by: Miroslav Benes <mbenes@suse.cz>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
When a section is empty (e.g. only zero-length alternative
replacements), there are no symbols to convert a section symbol
reference to. Skip the reloc instead of erroring out.
Fixes: dd590d4d57 ("objtool/klp: Introduce klp diff subcommand for diffing object files")
Acked-by: Song Liu <song@kernel.org>
Reviewed-by: Miroslav Benes <mbenes@suse.cz>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Convert the callback-based iterate_sym_by_demangled_name() with a new
for_each_sym_by_demangled_name() macro. This eliminates the callback
struct/function and makes the code more compact and readable.
Acked-by: Song Liu <song@kernel.org>
Reviewed-by: Miroslav Benes <mbenes@suse.cz>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
create_fake_symbols() has two phases: creating symbols from
ANNOTATE_DATA_SPECIAL entries, and a fallback that uses sh_entsize for
special sections like .static_call_sites.
When .discard.annotate_data is absent, the function returns early,
skipping the entsize fallback and silently allowing unsupported
module-local static call keys through.
Fix it by jumping to the entsize phase instead of returning early.
Fixes: dd590d4d57 ("objtool/klp: Introduce klp diff subcommand for diffing object files")
Assisted-by: Claude:claude-4-opus
Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
Acked-by: Song Liu <song@kernel.org>
Reviewed-by: Miroslav Benes <mbenes@suse.cz>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Clang emits various .L-prefixed local symbols beyond .Ltmp*, such as
.L__const.* for local constant data. These are assembler-local labels
not present in kallsyms, so they can never be resolved at module load
time.
Broaden the check from .Ltmp* to all .L* symbols so they get cloned into
the patch module instead.
Acked-by: Song Liu <song@kernel.org>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Clang LTO uses __UNIQUE_ID() to generate some uniquely named wrapper
functions, like initstubs. If they're uncorrelated, prevent them from
being reported as new functions and included unnecessarily.
Note that dont_correlate() already includes prefix functions, so prefix
functions are still being ignored here.
Acked-by: Song Liu <song@kernel.org>
Reviewed-by: Miroslav Benes <mbenes@suse.cz>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
With LTO, the initcall infrastructure generates __initstub__kmod_*
wrapper functions in .init.text. These are the LTO equivalent of
__initcall__kmod_* data pointers, which are already excluded from
correlation.
These are __init functions whose memory is freed after boot, so there's
no reason to include or reference them in a livepatch module.
Acked-by: Song Liu <song@kernel.org>
Reviewed-by: Miroslav Benes <mbenes@suse.cz>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Some arch/x86/crypto/*.S files define local .set/.equ constants that get
duplicated in vmlinux.o. This causes klp-diff to fail with "Multiple
correlation candidates" errors since it can't uniquely match these
between orig and patched builds.
Skip ABS symbols in dont_correlate(). They're purely compile-time
assembly constants that are never referenced by relocations, so they
don't need correlation.
Acked-by: Song Liu <song@kernel.org>
Reviewed-by: Miroslav Benes <mbenes@suse.cz>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Symbols created by __ADDRESSABLE() are only used to convince the
toolchain not to optimize out the referenced symbol.
Reviewed-by: Miroslav Benes <mbenes@suse.cz>
Acked-by: Song Liu <song@kernel.org>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
While there was once a section named .data.once, it has since been
renamed to .data..once with commit dbefa1f31a ("Rename .data.once to
.data..once to fix resetting WARN*_ONCE"). Fix it.
Fixes: dd590d4d57 ("objtool/klp: Introduce klp diff subcommand for diffing object files")
Acked-by: Song Liu <song@kernel.org>
Reviewed-by: Miroslav Benes <mbenes@suse.cz>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
For naming function-local static locals, GCC uses <var>.<id>, e.g.
__already_done.15, while Clang uses <func>.<var> with optional .<id>,
e.g. create_worker.__already_done.111
The existing is_uncorrelated_static_local() check only matches the GCC
convention where the variable name is a prefix. Handle both cases by
checking for a prefix match (GCC) and by checking after the first dot
separator (Clang).
Fixes: dd590d4d57 ("objtool/klp: Introduce klp diff subcommand for diffing object files")
Signed-off-by: Joe Lawrence <joe.lawrence@redhat.com>
Acked-by: Song Liu <song@kernel.org>
Reviewed-by: Miroslav Benes <mbenes@suse.cz>
Signed-off-by: Josh Poimboeuf <jpoimboe@kernel.org>
Josh Poimboeuf
Joe LawrenceandJosh Poimboeuf