Implement lock handoff to avoid lock starvation.
Lock starvation is possible because mutex_lock() allows lock stealing,
where a running (or optimistic spinning) task beats the woken waiter
to the acquire.
Lock stealing is an important performance optimization because waiting
for a waiter to wake up and get runtime can take a significant time,
during which everyboy would stall on the lock.
The down-side is of course that it allows for starvation.
This patch has the waiter requesting a handoff if it fails to acquire
the lock upon waking. This re-introduces some of the wait time,
because once we do a handoff we have to wait for the waiter to wake up
again.
A future patch will add a round of optimistic spinning to attempt to
alleviate this penalty, but if that turns out to not be enough, we can
add a counter and only request handoff after multiple failed wakeups.
There are a few tricky implementation details:
- accepting a handoff must only be done in the wait-loop. Since the
handoff condition is owner == current, it can easily cause
recursive locking trouble.
- accepting the handoff must be careful to provide the ACQUIRE
semantics.
- having the HANDOFF bit set on unlock requires care, we must not
clear the owner.
- we must be careful to not leave HANDOFF set after we've acquired
the lock. The tricky scenario is setting the HANDOFF bit on an
unlocked mutex.
Tested-by: Jason Low <jason.low2@hpe.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Waiman Long <Waiman.Long@hpe.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
The current mutex implementation has an atomic lock word and a
non-atomic owner field.
This disparity leads to a number of issues with the current mutex code
as it means that we can have a locked mutex without an explicit owner
(because the owner field has not been set, or already cleared).
This leads to a number of weird corner cases, esp. between the
optimistic spinning and debug code. Where the optimistic spinning
code needs the owner field updated inside the lock region, the debug
code is more relaxed because the whole lock is serialized by the
wait_lock.
Also, the spinning code itself has a few corner cases where we need to
deal with a held lock without an owner field.
Furthermore, it becomes even more of a problem when trying to fix
starvation cases in the current code. We end up stacking special case
on special case.
To solve this rework the basic mutex implementation to be a single
atomic word that contains the owner and uses the low bits for extra
state.
This matches how PI futexes and rt_mutex already work. By having the
owner an integral part of the lock state a lot of the problems
dissapear and we get a better option to deal with starvation cases,
direct owner handoff.
Changing the basic mutex does however invalidate all the arch specific
mutex code; this patch leaves that unused in-place, a later patch will
remove that.
Tested-by: Jason Low <jason.low2@hpe.com>
Signed-off-by: Peter Zijlstra (Intel) <peterz@infradead.org>
Reviewed-by: Will Deacon <will.deacon@arm.com>
Cc: Andrew Morton <akpm@linux-foundation.org>
Cc: Linus Torvalds <torvalds@linux-foundation.org>
Cc: Paul E. McKenney <paulmck@linux.vnet.ibm.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: linux-kernel@vger.kernel.org
Signed-off-by: Ingo Molnar <mingo@kernel.org>
Pull crypto fix from Herbert Xu:
"This fixes a regression caused by the stack vmalloc change"
* 'linus' of git://git.kernel.org/pub/scm/linux/kernel/git/herbert/crypto-2.6:
hwrng: core - Don't use a stack buffer in add_early_randomness()
Pull clk fixes from Stephen Boyd:
"This is the first batch of clk driver fixes for this release.
We have a handful of fixes for the uniphier clk driver that was
introduced recently, as well as Kconfig option hiding, module
autoloading markings, and a few fixes for clk_hw based registration
patches that went in this merge window"
* tag 'clk-fixes-for-linus' of git://git.kernel.org/pub/scm/linux/kernel/git/clk/linux:
clk: at91: Fix a return value in case of error
clk: uniphier: rename MIO clock to SD clock for Pro5, PXs2, LD20 SoCs
clk: uniphier: fix memory overrun bug
clk: hi6220: use CLK_OF_DECLARE_DRIVER for sysctrl and mediactrl clock init
clk: mvebu: armada-37xx-periph: Fix the clock gate flag
clk: bcm2835: Clamp the PLL's requested rate to the hardware limits.
clk: max77686: fix number of clocks setup for clk_hw based registration
clk: mvebu: armada-37xx-periph: Fix the clock provider registration
clk: core: add __init decoration for CLK_OF_DECLARE_DRIVER function
clk: mediatek: Add hardware dependency
clk: samsung: clk-exynos-audss: Fix module autoload
clk: uniphier: fix type of variable passed to regmap_read()
clk: uniphier: add system clock support for sLD3 SoC
Pull GPIO fixes from Linus Walleij:
"Here is a set of GPIO fixes for the v4.9 kernel series:
- Fix up off-by one and line offset validation, info leak to
userspace, and reject invalid flags. Those are especially valuable
hardening patches from Lars-Peter Clausen, all tagged for stable.
- Fix module autoload for TS4800 and ATH79.
- Correct the IRQ handler for MPC8xxx to use handle_level_irq() as it
(a) reacts to edges not levels and (b) even implements .irq_ack().
We were missing IRQs here.
- Fix the error path for acpi_dev_gpio_irq_get()
- Fix a memory leak in the MXS driver.
- Fix an annoying typo in the STMPE driver.
- Put a dependency on sysfs to the mockup driver"
* tag 'gpio-v4.9-2' of git://git.kernel.org/pub/scm/linux/kernel/git/linusw/linux-gpio:
gpio: mpc8xxx: Correct irq handler function
gpio: ath79: Fix module autoload
gpio: ts4800: Fix module autoload
gpio: GPIO_GET_LINEEVENT_IOCTL: Reject invalid line and event flags
gpio: GPIO_GET_LINEHANDLE_IOCTL: Reject invalid line flags
gpio: GPIOHANDLE_GET_LINE_VALUES_IOCTL: Fix information leak
gpio: GPIO_GET_LINEEVENT_IOCTL: Validate line offset
gpio: GPIOHANDLE_GET_LINE_VALUES_IOCTL: Fix information leak
gpio: GPIO_GET_LINEHANDLE_IOCTL: Validate line offset
gpio: GPIO_GET_CHIPINFO_IOCTL: Fix information leak
gpio: GPIO_GET_CHIPINFO_IOCTL: Fix line offset validation
gpio / ACPI: fix returned error from acpi_dev_gpio_irq_get()
gpio: mockup: add sysfs dependency
gpio: stmpe: || vs && typo
gpio: mxs: Unmap region obtained by of_iomap
gpio/board.txt: point to gpiod_set_value
Pull xen fixes from David Vrabel:
- advertise control feature flags in xenstore
- fix x86 build when XEN_PVHVM is disabled
* tag 'for-linus-4.9-rc2-tag' of git://git.kernel.org/pub/scm/linux/kernel/git/xen/tip:
xenbus: check return value of xenbus_scanf()
xenbus: prefer list_for_each()
x86: xen: move cpu_up functions out of ifdef
xenbus: advertise control feature flags
This patch unexports the low-level __get_user_pages() function.
Recent refactoring of the get_user_pages* functions allow flags to be
passed through get_user_pages() which eliminates the need for access to
this function from its one user, kvm.
We can see that the two calls to get_user_pages() which replace
__get_user_pages() in kvm_main.c are equivalent by examining their call
stacks:
get_user_page_nowait():
get_user_pages(start, 1, flags, page, NULL)
__get_user_pages_locked(current, current->mm, start, 1, page, NULL, NULL,
false, flags | FOLL_TOUCH)
__get_user_pages(current, current->mm, start, 1,
flags | FOLL_TOUCH | FOLL_GET, page, NULL, NULL)
check_user_page_hwpoison():
get_user_pages(addr, 1, flags, NULL, NULL)
__get_user_pages_locked(current, current->mm, addr, 1, NULL, NULL, NULL,
false, flags | FOLL_TOUCH)
__get_user_pages(current, current->mm, addr, 1, flags | FOLL_TOUCH, NULL,
NULL, NULL)
Signed-off-by: Lorenzo Stoakes <lstoakes@gmail.com>
Acked-by: Paolo Bonzini <pbonzini@redhat.com>
Acked-by: Michal Hocko <mhocko@suse.com>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Now that Lorenzo cleaned things up and made the FOLL_FORCE users
explicit, it becomes obvious how some of them don't really need
FOLL_FORCE at all.
So remove FOLL_FORCE from the proc code that reads the command line and
arguments from user space.
The mem_rw() function actually does want FOLL_FORCE, because gdd (and
possibly many other debuggers) use it as a much more convenient version
of PTRACE_PEEKDATA, but we should consider making the FOLL_FORCE part
conditional on actually being a ptracer. This does not actually do
that, just moves adds a comment to that effect and moves the gup_flags
settings next to each other.
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
Don't ignore errors here: Set backend state to unknown when
unsuccessful.
Signed-off-by: Jan Beulich <jbeulich@suse.com>
Signed-off-by: David Vrabel <david.vrabel@citrix.com>
This is more efficient than list_for_each_safe() when list modification
is accompanied by breaking out of the loop.
Signed-off-by: Jan Beulich <jbeulich@suse.com>
Reviewed-by: Juergen Gross <jgross@suse.com>
Signed-off-by: David Vrabel <david.vrabel@citrix.com>
Three newly introduced functions are not defined when CONFIG_XEN_PVHVM is
disabled, but are still being used:
arch/x86/xen/enlighten.c:141:12: warning: ‘xen_cpu_up_prepare’ used but never defined
arch/x86/xen/enlighten.c:142:12: warning: ‘xen_cpu_up_online’ used but never defined
arch/x86/xen/enlighten.c:143:12: warning: ‘xen_cpu_dead’ used but never defined
Fixes: 4d737042d6 ("xen/x86: Convert to hotplug state machine")
Signed-off-by: Arnd Bergmann <arnd@arndb.de>
Signed-off-by: David Vrabel <david.vrabel@citrix.com>
The Xen docs specify several flags which a guest can set to advertise
which values of the xenstore control/shutdown key it will recognize.
This patch adds code to write all the relevant feature-flag keys.
Based-on-patch-by: Paul Durrant <paul.durrant@citrix.com>
Signed-off-by: Juergen Gross <jgross@suse.com>
Reviewed-by: David Vrabel <david.vrabel@citrix.com>
Reviewed-by: Paul Durrant <paul.durrant@citrix.com>
Signed-off-by: David Vrabel <david.vrabel@citrix.com>
From the beginning of the gpio-mpc8xxx.c, the "handle_level_irq"
has being used to handle GPIO interrupts in the PowerPC/Layerscape
platforms. But actually, almost all PowerPC/Layerscape platforms
assert an interrupt request upon either a high-to-low change or
any change on the state of the signal.
So the "handle_level_irq" is not reasonable for PowerPC/Layerscape
GPIO interrupt, it should be "handle_edge_irq". Otherwise the system
may lost some interrupts from the PIN's state changes.
Signed-off-by: Liu Gang <Gang.Liu@nxp.com>
Signed-off-by: Linus Walleij <linus.walleij@linaro.org>
Pull UBI[FS] fixes from Richard Weinberger:
"This contains fixes for issues in both UBI and UBIFS:
- Fallout from the merge window, refactoring UBI code introduced some
issues.
- Fixes for an UBIFS readdir bug which can cause getdents() to busy
loop for ever and a bug in the UBIFS xattr code"
* tag 'upstream-4.9-rc2' of git://git.infradead.org/linux-ubifs:
ubifs: Abort readdir upon error
UBI: Fix crash in try_recover_peb()
ubi: fix swapped arguments to call to ubi_alloc_aeb
ubifs: Fix xattr_names length in exit paths
ubifs: Rename ubifs_rename2
Pull ext4 fixes from Ted Ts'o:
"A few bug fixes and add some missing KERN_CONT annotations"
* tag 'ext4_for_linus_stable' of git://git.kernel.org/pub/scm/linux/kernel/git/tytso/ext4:
ext4: add missing KERN_CONT to a few more debugging uses
fscrypto: lock inode while setting encryption policy
ext4: correct endianness conversion in __xattr_check_inode()
fscrypto: make XTS tweak initialization endian-independent
ext4: do not advertise encryption support when disabled
jbd2: fix incorrect unlock on j_list_lock
ext4: super.c: Update logging style using KERN_CONT
Pull SCSI target fixes from Nicholas Bellinger:
"Here are the outstanding target-pending fixes for v4.9-rc2.
This includes:
- Fix v4.1.y+ reference leak regression with concurrent TMR
ABORT_TASK + session shutdown. (Vaibhav Tandon)
- Enable tcm_fc w/ SCF_USE_CPUID to avoid host exchange timeouts
(Hannes)
- target/user error sense handling fixes. (Andy + MNC + HCH)
- Fix iscsi-target NOP_OUT error path iscsi_cmd descriptor leak
(Varun)
- Two EXTENDED_COPY SCSI status fixes for ESX VAAI (Dinesh Israni +
Nixon Vincent)
- Revert a v4.8 residual overflow change, that breaks sg_inq with
small allocation lengths.
There are a number of folks stress testing the v4.1.y regression fix
in their environments, and more folks doing iser-target I/O stress
testing atop recent v4.x.y code.
There is also one v4.2.y+ RCU conversion regression related to
explicit NodeACL configfs changes, that is still being tracked down"
* git://git.kernel.org/pub/scm/linux/kernel/git/nab/target-pending:
target/tcm_fc: use CPU affinity for responses
target/tcm_fc: Update debugging statements to match libfc usage
target/tcm_fc: return detailed error in ft_sess_create()
target/tcm_fc: print command pointer in debug message
target: fix potential race window in target_sess_cmd_list_waiting()
Revert "target: Fix residual overflow handling in target_complete_cmd_with_length"
target: Don't override EXTENDED_COPY xcopy_pt_cmd SCSI status code
target: Make EXTENDED_COPY 0xe4 failure return COPY TARGET DEVICE NOT REACHABLE
target: Re-add missing SCF_ACK_KREF assignment in v4.1.y
iscsi-target: fix iscsi cmd leak
iscsi-target: fix spelling mistake "Unsolicitied" -> "Unsolicited"
target/user: Fix comments to not refer to data ring
target/user: Return an error if cmd data size is too large
target/user: Use sense_reason_t in tcmu_queue_cmd_ring
Pull hwmon fixes from Guenter Roeck:
"Couple of hwmon fixes:
Fix a potential ERR_PTR dereference in max31790 driver, and handle
temperature readings below 0 in adm9240 driver"
* tag 'hwmon-for-linus-v4.9-rc2' of git://git.kernel.org/pub/scm/linux/kernel/git/groeck/linux-staging:
hwmon: (max31790) potential ERR_PTR dereference
hwmon: (adm9240) handle temperature readings below 0
Pull IPMI updates from Corey Minyard:
"A small bug fix and a new driver for acting as an IPMI device.
I was on vacation during the merge window (a long vacation) but this
is a bug fix that should go in and a new driver that shouldn't hurt
anything.
This has been in linux-next for a month or so"
* tag 'for-linus-4.9-2' of git://git.code.sf.net/p/openipmi/linux-ipmi:
ipmi: fix crash on reading version from proc after unregisted bmc
ipmi/bt-bmc: remove redundant return value check of platform_get_resource()
ipmi/bt-bmc: add a dependency on ARCH_ASPEED
ipmi: Fix ioremap error handling in bt-bmc
ipmi: add an Aspeed BT IPMI BMC driver
Waiman Long
Peter Zijlstra
Linus Torvalds
Lorenzo Stoakes
Jan Beulich
Arnd Bergmann
Juergen Gross
Liu Gang