Avoid atomic overhead in slab_alloc and slab_free
SLUB needs to use the slab_lock for the per cpu slabs to synchronize with
potential kfree operations. This patch avoids that need by moving all free
objects onto a lockless_freelist. The regular freelist continues to exist
and will be used to free objects. So while we consume the
lockless_freelist the regular freelist may build up objects.
If we are out of objects on the lockless_freelist then we may check the
regular freelist. If it has objects then we move those over to the
lockless_freelist and do this again. There is a significant savings in
terms of atomic operations that have to be performed.
We can even free directly to the lockless_freelist if we know that we are
running on the same processor. So this speeds up short lived objects.
They may be allocated and freed without taking the slab_lock. This is
particular good for netperf.
In order to maximize the effect of the new faster hotpath we extract the
hottest performance pieces into inlined functions. These are then inlined
into kmem_cache_alloc and kmem_cache_free. So hotpath allocation and
freeing no longer requires a subroutine call within SLUB.
[I am not sure that it is worth doing this because it changes the easy to
read structure of slub just to reduce atomic ops. However, there is
someone out there with a benchmark on 4 way and 8 way processor systems
that seems to show a 5% regression vs. Slab. Seems that the regression is
due to increased atomic operations use vs. SLAB in SLUB). I wonder if
this is applicable or discernable at all in a real workload?]
Signed-off-by: Christoph Lameter <clameter@sgi.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
* 'for-linus' of master.kernel.org:/pub/scm/linux/kernel/git/roland/infiniband:
IB/mlx4: Add a driver Mellanox ConnectX InfiniBand adapters
IB: Put rlimit accounting struct in struct ib_umem
IB/uverbs: Export ib_umem_get()/ib_umem_release() to modules
This reverts commit 5b479c91da.
Quoth Neil Brown:
"It causes an oops when auto-detecting raid arrays, and it doesn't
seem easy to fix.
The array may not be 'open' when do_md_run is called, so
bdev->bd_disk might be NULL, so bd_set_size can oops.
This whole approach of opening an md device before it has been
assembled just seems to get more and more painful. I think I'm going
to have to come up with something clever to provide both backward
comparability with usage expectation, and sane integration into the
rest of the kernel."
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
* master.kernel.org:/pub/scm/linux/kernel/git/bart/ide-2.6:
ide: fix PIO setup on resume for ATAPI devices
ide: legacy PCI bus order probing fixes
ide: add ide_proc_register_port()
ide: add "initializing" argument to ide_register_hw()
ide: cable detection fixes (take 2)
ide: move IDE settings handling to ide-proc.c
ide: split off ioctl handling from IDE settings (v2)
ide: make /proc/ide/ optional
ide: add ide_tune_dma() helper
ide: rework the code for selecting the best DMA transfer mode (v3)
ide: fix UDMA/MWDMA/SWDMA masks (v3)
IDE PCI host drivers should register themselves with IDE core only when
IDE driver is built-in, otherwise (IDE driver is modular and thus IDE PCI
host drivers are also modular) the code has no effect and just complicates
the probing.
Fix it by adding new config option CONFIG_IDEPCI_PCIBUS (defined only when
needed and invisible to the user) and covering by #ifdef/#endif the code
in question. It turned out that "ide=reverse" was silently accepted but did
nothing in case when IDE driver was modular, this is fixed now.
Signed-off-by: Bartlomiej Zolnierkiewicz <bzolnier@gmail.com>
* create_proc_ide_interfaces() tries to add /proc entries for every probed
and initialized IDE port, replace it by ide_proc_register_port() which does
it only for the given port (also rename destroy_proc_ide_interface() to
ide_proc_unregister_port() for consistency)
* convert {create,destroy}_proc_ide_interface[s]() users to use new functions
* pmac driver depended on proc_ide_create() to add /proc port entries, fix it
* au1xxx-ide, swarm and cs5520 drivers depended indirectly on ide-generic
driver (CONFIG_IDE_GENERIC=y) to add port /proc entries, fix them
* there is now no need to add /proc entries for IDE ports in proc_ide_create()
so don't do it
* proc_ide_create() needs now to be called before drivers are probed - fix it,
while at it make proc_ide_create() create /proc "ide" directory
Signed-off-by: Bartlomiej Zolnierkiewicz <bzolnier@gmail.com>
Add "initializing" argument to ide_register_hw() and use it instead of ide.c
wide variable of the same name. Update all users of ide_register_hw()
accordingly.
Signed-off-by: Bartlomiej Zolnierkiewicz <bzolnier@gmail.com>
Tejun's recent eighty_ninty_three() fix has inspired me to do more thorough
review of the cable detection code...
* print user-friendly warning about limiting the maximum transfer speed
to UDMA33 (and the reason behind it) when 80-wire cable is not detected,
also while at it cleanup eighty_ninty_three() a bit
* use eighty_ninty_three() in ide_ata66_check(), this actually fixes 3 bugs:
- bit 14 (word 93 validity check) == 1 && bit 13 (80-wire cable test) == 1
were used as 80-wire cable present test for CONFIG_IDEDMA_IVB=n case
(please see FIXME comment in eighty_ninty_three() for more details)
- CONFIG_IDEDMA_IVB=y/n cases were interchanged
- check for SATA devices was missing
* remove private cable warnings from pdc_202xx{old,new} drivers now that core
code provides this functionality (plus, in pdc202xx_new case the test could
give false warnings for ATAPI devices because pdc202xx_new driver doesn't
even support ATAPI DMA)
Cc: Tejun Heo <htejun@gmail.com>
Signed-off-by: Bartlomiej Zolnierkiewicz <bzolnier@gmail.com>
* move
__ide_add_setting()
ide_add_setting()
__ide_remove_setting()
auto_remove_settings()
ide_find_setting_by_name()
ide_read_setting()
ide_write_setting()
set_xfer_rate()
ide_add_generic_settings()
ide_register_subdriver()
ide_unregister_subdriver()
from ide.c to ide-proc.c
* set_{io_32bit,pio_mode,using_dma}() cannot be marked static now, fix it
* rename ide_[un]register_subdriver() to ide_proc_[un]register_driver(),
update device drivers to use new names
* add CONFIG_IDE_PROC_FS=n versions of ide_proc_[un]register_driver()
and ide_add_generic_settings()
* make ide_find_setting_by_name(), ide_{read,write}_setting()
and ide_{add,remove}_proc_entries() static
* cover IDE settings code in device drivers with CONFIG_IDE_PROC_FS #ifdef,
also while at it cover with CONFIG_IDE_PROC_FS #ifdef ide_driver_t.proc
* remove bogus comment from ide.h
* cover with CONFIG_IDE_PROC_FS #ifdef .proc and .settings in ide_drive_t
Besides saner code this patch results in the IDE core smaller by ~2 kB
(on x86-32) and IDE disk driver by ~1 kB (ditto) when CONFIG_IDE_PROC_FS=n.
Signed-off-by: Bartlomiej Zolnierkiewicz <bzolnier@gmail.com>
* do write permission and min/max checks in ide_procset_t functions
* ide-disk.c: drive->id is always available so cleanup "multcount" setting
accordingly
* ide-disk.c: "address" setting was incorrectly defined as type TYPE_INTA,
fix it by using type TYPE_BYTE and updating ide_drive_t->adressing field,
the bug didn't trigger because this IDE setting uses custom ->set function
* ide.c: add set_ksettings() for handling HDIO_SET_KEEPSETTINGS ioctl
* ide.c: add set_unmaskirq() for handling HDIO_SET_UNMASKINTR ioctl
* handle ioctls directly in generic_ide_ioclt() and idedisk_ioctl()
instead of using IDE settings to deal with them
* remove no longer needed ide_find_setting_by_ioctl() and {read,write}_ioctl
fields from ide_settings_t, also remove now unused TYPE_INTA handling
v2:
* add missing EXPORT_SYMBOL_GPL(ide_setting_sem) needed now for ide-disk
Signed-off-by: Bartlomiej Zolnierkiewicz <bzolnier@gmail.com>
All important information/features should be already available through
sysfs and ioctl interfaces.
Add CONFIG_IDE_PROC_FS (CONFIG_SCSI_PROC_FS rip-off) config option,
disabling it makes IDE driver ~5 kB smaller (on x86-32).
While at it add CONFIG_PROC_FS=n versions of proc_ide_{create,destroy}()
and remove no longer needed #ifdefs.
Signed-off-by: Bartlomiej Zolnierkiewicz <bzolnier@gmail.com>
After reworking the code responsible for selecting the best DMA
transfer mode it is now possible to add generic ide_tune_dma() helper.
Convert some IDE PCI host drivers to use it (the ones left need more work).
Signed-off-by: Bartlomiej Zolnierkiewicz <bzolnier@gmail.com>
Depends on the "ide: fix UDMA/MWDMA/SWDMA masks" patch.
* add ide_hwif_t.udma_filter hook for filtering UDMA mask
(use it in alim15x3, hpt366, siimage and serverworks drivers)
* add ide_max_dma_mode() for finding best DMA mode for the device
(loosely based on some older libata-core.c code)
* convert ide_dma_speed() users to use ide_max_dma_mode()
* make ide_rate_filter() take "ide_drive_t *drive" as an argument instead
of "u8 mode" and teach it to how to use UDMA mask to do filtering
* use ide_rate_filter() in hpt366 driver
* remove no longer needed ide_dma_speed() and *_ratemask()
* unexport eighty_ninty_three()
v2:
* rename ->filter_udma_mask to ->udma_filter
[ Suggested by Sergei Shtylyov <sshtylyov@ru.mvista.com>. ]
v3:
* updated for scc_pata driver (fixes XFER_UDMA_6 filtering for user-space
originated transfer mode change requests when 100MHz clock is used)
Signed-off-by: Bartlomiej Zolnierkiewicz <bzolnier@gmail.com>
* use 0x00 instead of 0x80 to disable ->{ultra,mwdma,swdma}_mask
* add udma_mask field to ide_pci_device_t and use it to initialize
->ultra_mask in aec62xx, cmd64x, pdc202xx_{new,old} and piix drivers
* fix UDMA masks to match with chipset specific *_ratemask()
(alim15x3, hpt366, serverworks and siimage drivers need UDMA mask
filtering method - done in the next patch)
v2:
* piix: fix cable detection for 82801AA_1 and 82372FB_1
[ Noticed by Sergei Shtylyov <sshtylyov@ru.mvista.com>. ]
* cmd64x: use hwif->cds->udma_mask
[ Suggested by Sergei Shtylyov <sshtylyov@ru.mvista.com>. ]
* aec62xx: fix newly introduced bug - check DMA status not command register
[ Noticed by Sergei Shtylyov <sshtylyov@ru.mvista.com>. ]
v3:
* piix: use hwif->cds->udma_mask
[ Suggested by Sergei Shtylyov <sshtylyov@ru.mvista.com>. ]
Signed-off-by: Bartlomiej Zolnierkiewicz <bzolnier@gmail.com>
* git://git.infradead.org/mtd-2.6: (21 commits)
[MTD] [CHIPS] Remove MTD_OBSOLETE_CHIPS (jedec, amd_flash, sharp)
[MTD] Delete allegedly obsolete "bank_size" field of mtd_info.
[MTD] Remove unnecessary user space check from mtd.h.
[MTD] [MAPS] Remove flash maps for no longer supported 405LP boards
[MTD] [MAPS] Fix missing printk() parameter in physmap_of.c MTD driver
[MTD] [NAND] platform NAND driver: add driver
[MTD] [NAND] platform NAND driver: update header
[JFFS2] Simplify and clean up jffs2_add_tn_to_tree() some more.
[JFFS2] Remove another bogus optimisation in jffs2_add_tn_to_tree()
[JFFS2] Remove broken insert_point optimisation in jffs2_add_tn_to_tree()
[JFFS2] Remember to calculate overlap on nodes which replace older nodes
[JFFS2] Don't advance c->wbuf_ofs to next eraseblock after wbuf flush
[MTD] [NAND] at91_nand.c: CMDLINE_PARTS support
[MTD] [NAND] Tidy up handling of page number in nand_block_bad()
[MTD] block2mtd_paramline[] mustn't be __initdata
[MTD] [NAND] Support multiple chips in CAFÉ driver
[MTD] [NAND] Rename cafe.c to cafe_nand.c and remove the multi-obj magic
[MTD] [NAND] Use rslib for CAFÉ ECC
[RSLIB] Support non-canonical GF representations
[JFFS2] Remove dead file histo_mips.h
...
* 'master' of git://git.kernel.org/pub/scm/linux/kernel/git/paulus/powerpc:
[POWERPC] Further fixes for the removal of 4level-fixup hack from ppc32
[POWERPC] EEH: log all PCI-X and PCI-E AER registers
[POWERPC] EEH: capture and log pci state on error
[POWERPC] EEH: Split up long error msg
[POWERPC] EEH: log error only after driver notification.
[POWERPC] fsl_soc: Make mac_addr const in fs_enet_of_init().
[POWERPC] Don't use SLAB/SLUB for PTE pages
[POWERPC] Spufs support for 64K LS mappings on 4K kernels
[POWERPC] Add ability to 4K kernel to hash in 64K pages
[POWERPC] Introduce address space "slices"
[POWERPC] Small fixes & cleanups in segment page size demotion
[POWERPC] iSeries: Make HVC_ISERIES the default
[POWERPC] iSeries: suppress build warning in lparmap.c
[POWERPC] Mark pages that don't exist as nosave
[POWERPC] swsusp: Introduce register_nosave_region_late
* git://git.kernel.org/pub/scm/linux/kernel/git/bunk/trivial: (25 commits)
sound: convert "sound" subdirectory to UTF-8
MAINTAINERS: Add cxacru website/mailing list
include files: convert "include" subdirectory to UTF-8
general: convert "kernel" subdirectory to UTF-8
documentation: convert the Documentation directory to UTF-8
Convert the toplevel files CREDITS and MAINTAINERS to UTF-8.
remove broken URLs from net drivers' output
Magic number prefix consistency change to Documentation/magic-number.txt
trivial: s/i_sem /i_mutex/
fix file specification in comments
drivers/base/platform.c: fix small typo in doc
misc doc and kconfig typos
Remove obsolete fat_cvf help text
Fix occurrences of "the the "
Fix minor typoes in kernel/module.c
Kconfig: Remove reference to external mqueue library
Kconfig: A couple of grammatical fixes in arch/i386/Kconfig
Correct comments in genrtc.c to refer to correct /proc file.
Fix more "deprecated" spellos.
Fix "deprecated" typoes.
...
Fix trivial comment conflict in kernel/relay.c.
md currently uses ->media_changed to make sure rescan_partitions
is call on md array after they are assembled.
However that doesn't happen until the array is opened, which is later
than some people would like.
So use blkdev_ioctl to do the rescan immediately that the
array has been assembled.
This means we can remove all the ->change infrastructure as it was only used
to trigger a partition rescan.
Signed-off-by: Neil Brown <neilb@suse.de>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
__used is defined to be __attribute__((unused)) for all pre-3.3 gcc
compilers to suppress warnings for unused functions because perhaps they
are referenced only in inline assembly. It is defined to be
__attribute__((used)) for gcc 3.3 and later so that the code is still
emitted for such functions.
__maybe_unused is defined to be __attribute__((unused)) for both function
and variable use if it could possibly be unreferenced due to the evaluation
of preprocessor macros. Function prototypes shall be marked with
__maybe_unused if the actual definition of the function is dependant on
preprocessor macros.
No update to compiler-intel.h is necessary because ICC supports both
__attribute__((used)) and __attribute__((unused)) as specified by the gcc
manual.
__attribute_used__ is deprecated and will be removed once all current
code is converted to using __used.
Cc: Rusty Russell <rusty@rustcorp.com.au>
Cc: Adrian Bunk <bunk@stusta.de>
Signed-off-by: David Rientjes <rientjes@google.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
This finally renames the thread_info field in task structure to stack, so that
the assumptions about this field are gone and archs have more freedom about
placing the thread_info structure.
Nonbroken archs which have a proper thread pointer can do the access to both
current thread and task structure via a single pointer.
It'll allow for a few more cleanups of the fork code, from which e.g. ia64
could benefit.
Signed-off-by: Roman Zippel <zippel@linux-m68k.org>
[akpm@linux-foundation.org: build fix]
Cc: Richard Henderson <rth@twiddle.net>
Cc: Ivan Kokshaysky <ink@jurassic.park.msu.ru>
Cc: Russell King <rmk@arm.linux.org.uk>
Cc: Ian Molton <spyro@f2s.com>
Cc: Haavard Skinnemoen <hskinnemoen@atmel.com>
Cc: Mikael Starvik <starvik@axis.com>
Cc: David Howells <dhowells@redhat.com>
Cc: Yoshinori Sato <ysato@users.sourceforge.jp>
Cc: "Luck, Tony" <tony.luck@intel.com>
Cc: Hirokazu Takata <takata@linux-m32r.org>
Cc: Geert Uytterhoeven <geert@linux-m68k.org>
Cc: Roman Zippel <zippel@linux-m68k.org>
Cc: Greg Ungerer <gerg@uclinux.org>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Ralf Baechle <ralf@linux-mips.org>
Cc: Paul Mackerras <paulus@samba.org>
Cc: Benjamin Herrenschmidt <benh@kernel.crashing.org>
Cc: Heiko Carstens <heiko.carstens@de.ibm.com>
Cc: Martin Schwidefsky <schwidefsky@de.ibm.com>
Cc: Paul Mundt <lethal@linux-sh.org>
Cc: Kazumoto Kojima <kkojima@rr.iij4u.or.jp>
Cc: Richard Curnow <rc@rc0.org.uk>
Cc: William Lee Irwin III <wli@holomorphy.com>
Cc: "David S. Miller" <davem@davemloft.net>
Cc: Jeff Dike <jdike@addtoit.com>
Cc: Paolo 'Blaisorblade' Giarrusso <blaisorblade@yahoo.it>
Cc: Miles Bader <uclinux-v850@lsi.nec.co.jp>
Cc: Andi Kleen <ak@muc.de>
Cc: Chris Zankel <chris@zankel.net>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
We need to make sure that the clocksources are resumed, when timekeeping is
resumed. The current resume logic does not guarantee this.
Add a resume function pointer to the clocksource struct, so clocksource
drivers which need to reinitialize the clocksource can provide a resume
function.
Add a resume function, which calls the maybe available clocksource resume
functions and resets the watchdog function, so a stable TSC can be used
accross suspend/resume.
Signed-off-by: Thomas Gleixner <tglx@linutronix.de>
Cc: john stultz <johnstul@us.ibm.com>
Cc: Andi Kleen <ak@suse.de>
Cc: Ingo Molnar <mingo@elte.hu>
Cc: <stable@kernel.org>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
Signed-off-by: Linus Torvalds <torvalds@linux-foundation.org>
