db44fc017d
sched: Avoid going ahead if ->cpus_allowed is not changed
...
If cpumask_equal(&p->cpus_allowed, new_mask) is true, seems
there is no reason to prevent set_cpus_allowed_ptr() return
directly.
Signed-off-by: Yong Zhang <yong.zhang0@gmail.com >
Acked-by: Hillf Danton <dhillf@gmail.com >
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl >
Link: http://lkml.kernel.org/r/20110509140705.GA2219@zhy
Signed-off-by: Ingo Molnar <mingo@elte.hu >
2011-05-16 11:01:18 +02:00
61eadef6a9
sched, rt: Update rq clock when unthrottling of an otherwise idle CPU
...
If an RT task is awakened while it's rt_rq is throttled, the time between
wakeup/enqueue and unthrottle/selection may be accounted as rt_time
if the CPU is idle. Set rq->skip_clock_update negative upon throttle
release to tell put_prev_task() that we need a clock update.
Reported-by: Thomas Giesel <skoe@directbox.com >
Signed-off-by: Mike Galbraith <efault@gmx.de >
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl >
Link: http://lkml.kernel.org/r/1304059010.7472.1.camel@marge.simson.net
Signed-off-by: Ingo Molnar <mingo@elte.hu >
2011-05-16 11:01:17 +02:00
3e51e3edfd
sched: Remove unused parameters from sched_fork() and wake_up_new_task()
...
sched_fork() and wake_up_new_task() are defined with a parameter
'unsigned long clone_flags', which is unused.
This patch removes the parameters.
Signed-off-by: Samir Bellabes <sam@synack.fr >
Acked-by: Peter Zijlstra <a.p.zijlstra@chello.nl >
Link: http://lkml.kernel.org/r/1305130685-1047-1-git-send-email-sam@synack.fr
Signed-off-by: Ingo Molnar <mingo@elte.hu >
2011-05-12 09:36:37 +02:00
7142d17e8f
sched: Shorten the construction of the span cpu mask of sched domain
...
For a given node, when constructing the cpumask for its
sched_domain to span, if there is no best node available after
searching, further efforts could be saved, based on small change
in the return value of find_next_best_node().
Signed-off-by: Hillf Danton <dhillf@gmail.com >
Cc: Peter Zijlstra <peterz@infradead.org >
Cc: Mike Galbraith <efault@gmx.de >
Cc: Yong Zhang <yong.zhang0@gmail.com >
Link: http://lkml.kernel.org/r/BANLkTi%3DqPWxRAa6%2BdT3ohEP6Z%3D0v%2Be4EXA@mail.gmail.com
Signed-off-by: Ingo Molnar <mingo@elte.hu >
2011-05-06 09:13:05 +02:00
4934a4d3d3
sched: Wrap the 'cfs_rq->nr_spread_over' field with CONFIG_SCHED_DEBUG
...
cfs_rq->nr_spread_over is only used when CONFIG_SCHED_DEBUG is set.
So wrap it with CONFIG_SCHED_DEBUG.
Signed-off-by: Rakib Mullick <rakib.mullick@gmail.com >
Cc: Peter Zijlstra <peterz@infradead.org >
Link: http://lkml.kernel.org/r/1304528026.15681.3.camel@localhost.localdomain
Signed-off-by: Ingo Molnar <mingo@elte.hu >
2011-05-06 09:04:19 +02:00
1437f5bca3
sched: Remove noop in alloc_rt_sched_group()
...
The rq varible, though computed for each possible cpu, has nothing to
do in the function, so it can be removed.
This also eliminates a build warning.
Signed-off-by: Hillf Danton <dhillf@gmail.com >
Reviewed-by: Yong Zhang <yong.zhang0@gmail.com >
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl >
Link: http://lkml.kernel.org/r/BANLkTin-FfQfqW5ym1iuEmrk8s777Y1LAg@mail.gmail.com
Signed-off-by: Ingo Molnar <mingo@elte.hu >
2011-04-26 13:34:08 +02:00
625f2a378e
sched: Get rid of lock_depth
...
Neil Brown pointed out that lock_depth somehow escaped the BKL
removal work. Let's get rid of it now.
Note that the perf scripting utilities still have a bunch of
code for dealing with common_lock_depth in tracepoints; I have
left that in place in case anybody wants to use that code with
older kernels.
Suggested-by: Neil Brown <neilb@suse.de >
Signed-off-by: Jonathan Corbet <corbet@lwn.net >
Cc: Arnd Bergmann <arnd@arndb.de >
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl >
Cc: Thomas Gleixner <tglx@linutronix.de >
Cc: Linus Torvalds <torvalds@linux-foundation.org >
Cc: Andrew Morton <akpm@linux-foundation.org >
Link: http://lkml.kernel.org/r/20110422111910.456c0e84@bike.lwn.net
Signed-off-by: Ingo Molnar <mingo@elte.hu >
2011-04-24 13:18:38 +02:00
d3bf52e998
sched: Remove obsolete comment from scheduler_tick()
...
scheduler_tick() is no longer called by fork code - this got discarded
a long time ago by commit bc947631d1rakib.mullick@gmail.com >
Cc: Peter Zijlstra <a.p.zijlstra@chello.nl >
Link: http://lkml.kernel.org/r/BANLkTimO4iGP0QpaHO1HHF1QOnVcQpc0cw@mail.gmail.com
Signed-off-by: Ingo Molnar <mingo@elte.hu >
2011-04-21 11:41:36 +02:00
42ac9e87fd
Merge commit 'v2.6.39-rc4' into sched/core
...
Merge reason: Pick up upstream fixes.
Signed-off-by: Ingo Molnar <mingo@elte.hu >
2011-04-21 11:39:28 +02:00
057f3fadb3
sched: Fix sched_domain iterations vs. RCU
...
Vladis Kletnieks reported a new RCU debug warning in the scheduler.
Since commit dce840a087Valdis.Kletnieks@vt.edu
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl >
Link: http://lkml.kernel.org/r/1302882741.2388.241.camel@twins
Signed-off-by: Ingo Molnar <mingo@elte.hu >
2011-04-19 10:56:54 +02:00
6ddafdaab3
Merge branch 'sched/locking' into sched/core
...
Merge reason: the rq locking changes are stable,
propagate them into the .40 queue.
Signed-off-by: Ingo Molnar <mingo@elte.hu >
2011-04-18 14:53:33 +02:00
a237c1c5bc
block: let io_schedule() flush the plug inline
...
Linus correctly observes that the most important dispatch cases
are now done from kblockd, this isn't ideal for latency reasons.
The original reason for switching dispatches out-of-line was to
avoid too deep a stack, so by _only_ letting the "accidental"
flush directly in schedule() be guarded by offload to kblockd,
we should be able to get the best of both worlds.
So add a blk_schedule_flush_plug() that offloads to kblockd,
and only use that from the schedule() path.
Signed-off-by: Jens Axboe <jaxboe@fusionio.com >
2011-04-16 13:27:55 +02:00
bd8e7dded8
sched: Remove need_migrate_task()
...
Oleg noticed that need_migrate_task() doesn't need the ->on_cpu check
now that ttwu() doesn't do remote enqueues for !->on_rq && ->on_cpu,
so remove the helper and replace the single instance with a direct
->on_rq test.
Suggested-by: Oleg Nesterov <oleg@redhat.com >
Reviewed-by: Frank Rowand <frank.rowand@am.sony.com >
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl >
Cc: Mike Galbraith <efault@gmx.de >
Cc: Nick Piggin <npiggin@kernel.dk >
Cc: Linus Torvalds <torvalds@linux-foundation.org >
Cc: Andrew Morton <akpm@linux-foundation.org >
Link: http://lkml.kernel.org/r/20110405152729.556674812@chello.nl
Signed-off-by: Ingo Molnar <mingo@elte.hu >
2011-04-14 08:52:41 +02:00
317f394160
sched: Move the second half of ttwu() to the remote cpu
...
Now that we've removed the rq->lock requirement from the first part of
ttwu() and can compute placement without holding any rq->lock, ensure
we execute the second half of ttwu() on the actual cpu we want the
task to run on.
This avoids having to take rq->lock and doing the task enqueue
remotely, saving lots on cacheline transfers.
As measured using: http://oss.oracle.com/~mason/sembench.c
$ for i in /sys/devices/system/cpu/cpu*/cpufreq/scaling_governor ; do echo performance > $i; done
$ echo 4096 32000 64 128 > /proc/sys/kernel/sem
$ ./sembench -t 2048 -w 1900 -o 0
unpatched: run time 30 seconds 647278 worker burns per second
patched: run time 30 seconds 816715 worker burns per second
Reviewed-by: Frank Rowand <frank.rowand@am.sony.com >
Cc: Mike Galbraith <efault@gmx.de >
Cc: Nick Piggin <npiggin@kernel.dk >
Cc: Linus Torvalds <torvalds@linux-foundation.org >
Cc: Andrew Morton <akpm@linux-foundation.org >
Signed-off-by: Ingo Molnar <mingo@elte.hu >
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl >
Link: http://lkml.kernel.org/r/20110405152729.515897185@chello.nl
2011-04-14 08:52:41 +02:00
c05fbafba1
sched: Restructure ttwu() some more
...
Factor our helper functions to make the inner workings of try_to_wake_up()
more obvious, this also allows for adding remote queues.
Reviewed-by: Frank Rowand <frank.rowand@am.sony.com >
Cc: Mike Galbraith <efault@gmx.de >
Cc: Nick Piggin <npiggin@kernel.dk >
Cc: Linus Torvalds <torvalds@linux-foundation.org >
Cc: Andrew Morton <akpm@linux-foundation.org >
Signed-off-by: Ingo Molnar <mingo@elte.hu >
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl >
Link: http://lkml.kernel.org/r/20110405152729.475848012@chello.nl
2011-04-14 08:52:40 +02:00
23f41eeb42
sched: Rename ttwu_post_activation() to ttwu_do_wakeup()
...
The ttwu_post_activation() code does the core wakeup, it sets TASK_RUNNING
and performs wakeup-preemption, so give is a more descriptive name.
Reviewed-by: Frank Rowand <frank.rowand@am.sony.com >
Cc: Mike Galbraith <efault@gmx.de >
Cc: Nick Piggin <npiggin@kernel.dk >
Cc: Linus Torvalds <torvalds@linux-foundation.org >
Cc: Andrew Morton <akpm@linux-foundation.org >
Signed-off-by: Ingo Molnar <mingo@elte.hu >
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl >
Link: http://lkml.kernel.org/r/20110405152729.434609705@chello.nl
2011-04-14 08:52:40 +02:00
b84cb5df1f
sched: Remove rq argument from ttwu_stat()
...
In order to call ttwu_stat() without holding rq->lock we must remove
its rq argument. Since we need to change rq stats, account to the
local rq instead of the task rq, this is safe since we have IRQs
disabled.
Reviewed-by: Frank Rowand <frank.rowand@am.sony.com >
Cc: Mike Galbraith <efault@gmx.de >
Cc: Nick Piggin <npiggin@kernel.dk >
Cc: Linus Torvalds <torvalds@linux-foundation.org >
Cc: Andrew Morton <akpm@linux-foundation.org >
Signed-off-by: Ingo Molnar <mingo@elte.hu >
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl >
Link: http://lkml.kernel.org/r/20110405152729.394638826@chello.nl
2011-04-14 08:52:40 +02:00
e4a52bcb9a
sched: Remove rq->lock from the first half of ttwu()
...
Currently ttwu() does two rq->lock acquisitions, once on the task's
old rq, holding it over the p->state fiddling and load-balance pass.
Then it drops the old rq->lock to acquire the new rq->lock.
By having serialized ttwu(), p->sched_class, p->cpus_allowed with
p->pi_lock, we can now drop the whole first rq->lock acquisition.
The p->pi_lock serializing concurrent ttwu() calls protects p->state,
which we will set to TASK_WAKING to bridge possible p->pi_lock to
rq->lock gaps and serialize set_task_cpu() calls against
task_rq_lock().
The p->pi_lock serialization of p->sched_class allows us to call
scheduling class methods without holding the rq->lock, and the
serialization of p->cpus_allowed allows us to do the load-balancing
bits without races.
Reviewed-by: Frank Rowand <frank.rowand@am.sony.com >
Cc: Mike Galbraith <efault@gmx.de >
Cc: Nick Piggin <npiggin@kernel.dk >
Cc: Linus Torvalds <torvalds@linux-foundation.org >
Cc: Andrew Morton <akpm@linux-foundation.org >
Signed-off-by: Ingo Molnar <mingo@elte.hu >
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl >
Link: http://lkml.kernel.org/r/20110405152729.354401150@chello.nl
2011-04-14 08:52:39 +02:00
8f42ced974
sched: Drop rq->lock from sched_exec()
...
Since we can now call select_task_rq() and set_task_cpu() with only
p->pi_lock held, and sched_exec() load-balancing has always been
optimistic, drop all rq->lock usage.
Oleg also noted that need_migrate_task() will always be true for
current, so don't bother calling that at all.
Reviewed-by: Frank Rowand <frank.rowand@am.sony.com >
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl >
Cc: Mike Galbraith <efault@gmx.de >
Cc: Nick Piggin <npiggin@kernel.dk >
Cc: Linus Torvalds <torvalds@linux-foundation.org >
Cc: Andrew Morton <akpm@linux-foundation.org >
Link: http://lkml.kernel.org/r/20110405152729.314204889@chello.nl
Signed-off-by: Ingo Molnar <mingo@elte.hu >
2011-04-14 08:52:39 +02:00
ab2515c4b9
sched: Drop rq->lock from first part of wake_up_new_task()
...
Since p->pi_lock now protects all things needed to call
select_task_rq() avoid the double remote rq->lock acquisition and rely
on p->pi_lock.
Reviewed-by: Frank Rowand <frank.rowand@am.sony.com >
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl >
Cc: Mike Galbraith <efault@gmx.de >
Cc: Nick Piggin <npiggin@kernel.dk >
Cc: Linus Torvalds <torvalds@linux-foundation.org >
Cc: Andrew Morton <akpm@linux-foundation.org >
Link: http://lkml.kernel.org/r/20110405152729.273362517@chello.nl
Signed-off-by: Ingo Molnar <mingo@elte.hu >
2011-04-14 08:52:38 +02:00
0122ec5b02
sched: Add p->pi_lock to task_rq_lock()
...
In order to be able to call set_task_cpu() while either holding
p->pi_lock or task_rq(p)->lock we need to hold both locks in order to
stabilize task_rq().
This makes task_rq_lock() acquire both locks, and have
__task_rq_lock() validate that p->pi_lock is held. This increases the
locking overhead for most scheduler syscalls but allows reduction of
rq->lock contention for some scheduler hot paths (ttwu).
Reviewed-by: Frank Rowand <frank.rowand@am.sony.com >
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl >
Cc: Mike Galbraith <efault@gmx.de >
Cc: Nick Piggin <npiggin@kernel.dk >
Cc: Linus Torvalds <torvalds@linux-foundation.org >
Cc: Andrew Morton <akpm@linux-foundation.org >
Link: http://lkml.kernel.org/r/20110405152729.232781355@chello.nl
Signed-off-by: Ingo Molnar <mingo@elte.hu >
2011-04-14 08:52:38 +02:00
2acca55ed9
sched: Also serialize ttwu_local() with p->pi_lock
...
Since we now serialize ttwu() using p->pi_lock, we also need to
serialize ttwu_local() using that, otherwise, once we drop the
rq->lock from ttwu() it can race with ttwu_local().
Reviewed-by: Frank Rowand <frank.rowand@am.sony.com >
Cc: Mike Galbraith <efault@gmx.de >
Cc: Nick Piggin <npiggin@kernel.dk >
Cc: Linus Torvalds <torvalds@linux-foundation.org >
Cc: Andrew Morton <akpm@linux-foundation.org >
Signed-off-by: Ingo Molnar <mingo@elte.hu >
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl >
Link: http://lkml.kernel.org/r/20110405152729.192366907@chello.nl
2011-04-14 08:52:37 +02:00
a8e4f2eaec
sched: Delay task_contributes_to_load()
...
In prepratation of having to call task_contributes_to_load() without
holding rq->lock, we need to store the result until we do and can
update the rq accounting accordingly.
Reviewed-by: Frank Rowand <frank.rowand@am.sony.com >
Cc: Mike Galbraith <efault@gmx.de >
Cc: Nick Piggin <npiggin@kernel.dk >
Cc: Linus Torvalds <torvalds@linux-foundation.org >
Cc: Andrew Morton <akpm@linux-foundation.org >
Signed-off-by: Ingo Molnar <mingo@elte.hu >
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl >
Link: http://lkml.kernel.org/r/20110405152729.151523907@chello.nl
2011-04-14 08:52:37 +02:00
3fe1698b7f
sched: Deal with non-atomic min_vruntime reads on 32bits
...
In order to avoid reading partial updated min_vruntime values on 32bit
implement a seqcount like solution.
Reviewed-by: Frank Rowand <frank.rowand@am.sony.com >
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl >
Cc: Mike Galbraith <efault@gmx.de >
Cc: Nick Piggin <npiggin@kernel.dk >
Cc: Linus Torvalds <torvalds@linux-foundation.org >
Cc: Andrew Morton <akpm@linux-foundation.org >
Link: http://lkml.kernel.org/r/20110405152729.111378493@chello.nl
Signed-off-by: Ingo Molnar <mingo@elte.hu >
2011-04-14 08:52:37 +02:00
74f8e4b233
sched: Remove rq argument to sched_class::task_waking()
...
In preparation of calling this without rq->lock held, remove the
dependency on the rq argument.
Reviewed-by: Frank Rowand <frank.rowand@am.sony.com >
Signed-off-by: Peter Zijlstra <a.p.zijlstra@chello.nl >
Cc: Mike Galbraith <efault@gmx.de >
Cc: Nick Piggin <npiggin@kernel.dk >
Cc: Linus Torvalds <torvalds@linux-foundation.org >
Cc: Andrew Morton <akpm@linux-foundation.org >
Link: http://lkml.kernel.org/r/20110405152729.071474242@chello.nl
Signed-off-by: Ingo Molnar <mingo@elte.hu >
2011-04-14 08:52:36 +02:00
Yong Zhang
Mike Galbraith
Samir Bellabes
Hillf Danton
Rakib Mullick
Jonathan Corbet
Ingo Molnar
Peter Zijlstra
Jens Axboe