linuxmsm/laptops-kernel
0
0
Fork 0
mirror of https://github.com/linux-msm/laptops-kernel.git synced 2026-08-13 14:19:53 -07:00
Code Issues Packages Projects Releases Wiki Activity

Merge tag 'for-6.19/block-20251201' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux

Browse Source 
Pull block updates from Jens Axboe:

 - Fix head insertion for mq-deadline, a regression from when priority
   support was added

 - Series simplifying and improving the ublk user copy code

 - Various ublk related cleanups

 - Fixup REQ_NOWAIT handling in loop/zloop, clearing NOWAIT when the
   request is punted to a thread for handling

 - Merge and then later revert loop dio nowait support, as it ended up
   causing excessive stack usage for when the inline issue code needs to
   dip back into the full file system code

 - Improve auto integrity code, making it less deadlock prone

 - Speedup polled IO handling, but manually managing the hctx lookups

 - Fixes for blk-throttle for SSD devices

 - Small series with fixes for the S390 dasd driver

 - Add support for caching zones, avoiding unnecessary report zone
   queries

 - MD pull requests via Yu:
      - fix null-ptr-dereference regression for dm-raid0
      - fix IO hang for raid5 when array is broken with IO inflight
      - remove legacy 1s delay to speed up system shutdown
      - change maintainer's email address
      - data can be lost if array is created with different lbs devices,
        fix this problem and record lbs of the array in metadata
      - fix rcu protection for md_thread
      - fix mddev kobject lifetime regression
      - enable atomic writes for md-linear
      - some cleanups

 - bcache updates via Coly
      - remove useless discard and cache device code
      - improve usage of per-cpu workqueues

 - Reorganize the IO scheduler switching code, fixing some lockdep
   reports as well

 - Improve the block layer P2P DMA support

 - Add support to the block tracing code for zoned devices

 - Segment calculation improves, and memory alignment flexibility
   improvements

 - Set of prep and cleanups patches for ublk batching support. The
   actual batching hasn't been added yet, but helps shrink down the
   workload of getting that patchset ready for 6.20

 - Fix for how the ps3 block driver handles segments offsets

 - Improve how block plugging handles batch tag allocations

 - nbd fixes for use-after-free of the configuration on device clear/put

 - Set of improvements and fixes for zloop

 - Add Damien as maintainer of the block zoned device code handling

 - Various other fixes and cleanups

* tag 'for-6.19/block-20251201' of git://git.kernel.org/pub/scm/linux/kernel/git/axboe/linux: (162 commits)
  block/rnbd: correct all kernel-doc complaints
  blk-mq: use queue_hctx in blk_mq_map_queue_type
  md: remove legacy 1s delay in md_notify_reboot
  md/raid5: fix IO hang when array is broken with IO inflight
  md: warn about updating super block failure
  md/raid0: fix NULL pointer dereference in create_strip_zones() for dm-raid
  sbitmap: fix all kernel-doc warnings
  ublk: add helper of __ublk_fetch()
  ublk: pass const pointer to ublk_queue_is_zoned()
  ublk: refactor auto buffer register in ublk_dispatch_req()
  ublk: add `union ublk_io_buf` with improved naming
  ublk: add parameter `struct io_uring_cmd *` to ublk_prep_auto_buf_reg()
  kfifo: add kfifo_alloc_node() helper for NUMA awareness
  blk-mq: fix potential uaf for 'queue_hw_ctx'
  blk-mq: use array manage hctx map instead of xarray
  ublk: prevent invalid access with DEBUG
  s390/dasd: Use scnprintf() instead of sprintf()
  s390/dasd: Move device name formatting into separate function
  s390/dasd: Remove unnecessary debugfs_create() return checks
  s390/dasd: Fix gendisk parent after copy pair swap
  ...
This commit is contained in:
Linus Torvalds
2025-12-03 19:26:18 -08:00
parent 0abcfd8983 d211a28035
commit cc25df3e2e
108 changed files with 3050 additions and 1588 deletions
Download Patch File Download Diff File Expand all files Collapse all files
Documentation/ABI/testing/sysfs-block-bcache
-7
View File
@@ -106,13 +106,6 @@ Description:
will be discarded from the cache. Should not be turned off with
writeback caching enabled.
What: /sys/block/<disk>/bcache/discard
Date: November 2010
Contact: Kent Overstreet <kent.overstreet@gmail.com>
Description:
For a cache, a boolean allowing discard/TRIM to be turned off
or back on if the device supports it.
What: /sys/block/<disk>/bcache/bucket_size
Date: November 2010
Contact: Kent Overstreet <kent.overstreet@gmail.com>
Documentation/admin-guide/bcache.rst
+2 -11
View File
@@ -17,8 +17,7 @@ The latest bcache kernel code can be found from mainline Linux kernel:
It's designed around the performance characteristics of SSDs - it only allocates
in erase block sized buckets, and it uses a hybrid btree/log to track cached
extents (which can be anywhere from a single sector to the bucket size). It's
designed to avoid random writes at all costs; it fills up an erase block
sequentially, then issues a discard before reusing it.
designed to avoid random writes at all costs.
Both writethrough and writeback caching are supported. Writeback defaults to
off, but can be switched on and off arbitrarily at runtime. Bcache goes to
@@ -618,19 +617,11 @@ bucket_size
cache_replacement_policy
One of either lru, fifo or random.
discard
Boolean; if on a discard/TRIM will be issued to each bucket before it is
reused. Defaults to off, since SATA TRIM is an unqueued command (and thus
slow).
freelist_percent
Size of the freelist as a percentage of nbuckets. Can be written to to
increase the number of buckets kept on the freelist, which lets you
artificially reduce the size of the cache at runtime. Mostly for testing
purposes (i.e. testing how different size caches affect your hit rate), but
since buckets are discarded when they move on to the freelist will also make
the SSD's garbage collection easier by effectively giving it more reserved
space.
purposes (i.e. testing how different size caches affect your hit rate).
io_errors
Number of errors that have occurred, decayed by io_error_halflife.
Documentation/admin-guide/blockdev/zoned_loop.rst
+37 -24
View File
@@ -68,30 +68,43 @@ The options available for the add command can be listed by reading the
In more details, the options that can be used with the "add" command are as
follows.
================ ===========================================================
id Device number (the X in /dev/zloopX).
Default: automatically assigned.
capacity_mb Device total capacity in MiB. This is always rounded up to
the nearest higher multiple of the zone size.
Default: 16384 MiB (16 GiB).
zone_size_mb Device zone size in MiB. Default: 256 MiB.
zone_capacity_mb Device zone capacity (must always be equal to or lower than
the zone size. Default: zone size.
conv_zones Total number of conventioanl zones starting from sector 0.
Default: 8.
base_dir Path to the base directory where to create the directory
containing the zone files of the device.
Default=/var/local/zloop.
The device directory containing the zone files is always
named with the device ID. E.g. the default zone file
directory for /dev/zloop0 is /var/local/zloop/0.
nr_queues Number of I/O queues of the zoned block device. This value is
always capped by the number of online CPUs
Default: 1
queue_depth Maximum I/O queue depth per I/O queue.
Default: 64
buffered_io Do buffered IOs instead of direct IOs (default: false)
================ ===========================================================
=================== =========================================================
id Device number (the X in /dev/zloopX).
Default: automatically assigned.
capacity_mb Device total capacity in MiB. This is always rounded up
to the nearest higher multiple of the zone size.
Default: 16384 MiB (16 GiB).
zone_size_mb Device zone size in MiB. Default: 256 MiB.
zone_capacity_mb Device zone capacity (must always be equal to or lower
than the zone size. Default: zone size.
conv_zones Total number of conventioanl zones starting from
sector 0
Default: 8
base_dir Path to the base directory where to create the directory
containing the zone files of the device.
Default=/var/local/zloop.
The device directory containing the zone files is always
named with the device ID. E.g. the default zone file
directory for /dev/zloop0 is /var/local/zloop/0.
nr_queues Number of I/O queues of the zoned block device. This
value is always capped by the number of online CPUs
Default: 1
queue_depth Maximum I/O queue depth per I/O queue.
Default: 64
buffered_io Do buffered IOs instead of direct IOs (default: false)
zone_append Enable or disable a zloop device native zone append
support.
Default: 1 (enabled).
If native zone append support is disabled, the block layer
will emulate this operation using regular write
operations.
ordered_zone_append Enable zloop mitigation of zone append reordering.
Default: disabled.
This is useful for testing file systems file data mapping
(extents), as when enabled, this can significantly reduce
the number of data extents needed to for a file data
mapping.
=================== =========================================================
3) Deleting a Zoned Device
--------------------------
Documentation/admin-guide/md.rst
+10
View File
@@ -238,6 +238,16 @@ All md devices contain:
the number of devices in a raid4/5/6, or to support external
metadata formats which mandate such clipping.
logical_block_size
Configure the array's logical block size in bytes. This attribute
is only supported for 1.x meta. Write the value before starting
array. The final array LBS uses the maximum between this
configuration and LBS of all combined devices. Note that
LBS cannot exceed PAGE_SIZE before RAID supports folio.
WARNING: Arrays created on new kernel cannot be assembled at old
kernel due to padding check, Set module parameter 'check_new_feature'
to false to bypass, but data loss may occur.
reshape_position
This is either ``none`` or a sector number within the devices of
the array where ``reshape`` is up to. If this is set, the three
MAINTAINERS
+11 -2
View File
@@ -4307,7 +4307,7 @@ F: Documentation/filesystems/befs.rst
F: fs/befs/
BFQ I/O SCHEDULER
M: Yu Kuai <yukuai3@huawei.com>
M: Yu Kuai <yukuai@fnnas.com>
L: linux-block@vger.kernel.org
S: Odd Fixes
F: Documentation/block/bfq-iosched.rst
@@ -4407,6 +4407,8 @@ F: block/
F: drivers/block/
F: include/linux/bio.h
F: include/linux/blk*
F: include/uapi/linux/blk*
F: include/uapi/linux/ioprio.h
F: kernel/trace/blktrace.c
F: lib/sbitmap.c
@@ -23908,7 +23910,7 @@ F: include/linux/property.h
SOFTWARE RAID (Multiple Disks) SUPPORT
M: Song Liu <song@kernel.org>
M: Yu Kuai <yukuai3@huawei.com>
M: Yu Kuai <yukuai@fnnas.com>
L: linux-raid@vger.kernel.org
S: Supported
Q: https://patchwork.kernel.org/project/linux-raid/list/
@@ -28371,6 +28373,13 @@ L: linux-kernel@vger.kernel.org
S: Maintained
F: arch/x86/kernel/cpu/zhaoxin.c
ZONED BLOCK DEVICE (BLOCK LAYER)
M: Damien Le Moal <dlemoal@kernel.org>
L: linux-block@vger.kernel.org
S: Maintained
F: block/blk-zoned.c
F: include/uapi/linux/blkzoned.h
ZONED LOOP DEVICE
M: Damien Le Moal <dlemoal@kernel.org>
R: Christoph Hellwig <hch@lst.de>
block/bio-integrity-auto.c
+3 -23
View File
@@ -29,7 +29,7 @@ static void bio_integrity_finish(struct bio_integrity_data *bid)
{
bid->bio->bi_integrity = NULL;
bid->bio->bi_opf &= ~REQ_INTEGRITY;
kfree(bvec_virt(bid->bip.bip_vec));
bio_integrity_free_buf(&bid->bip);
mempool_free(bid, &bid_pool);
}
@@ -110,8 +110,6 @@ bool bio_integrity_prep(struct bio *bio)
struct bio_integrity_data *bid;
bool set_flags = true;
gfp_t gfp = GFP_NOIO;
unsigned int len;
void *buf;
if (!bi)
return true;
@@ -152,19 +150,12 @@ bool bio_integrity_prep(struct bio *bio)
if (WARN_ON_ONCE(bio_has_crypt_ctx(bio)))
return true;
/* Allocate kernel buffer for protection data */
len = bio_integrity_bytes(bi, bio_sectors(bio));
buf = kmalloc(len, gfp);
if (!buf)
goto err_end_io;
bid = mempool_alloc(&bid_pool, GFP_NOIO);
if (!bid)
goto err_free_buf;
bio_integrity_init(bio, &bid->bip, &bid->bvec, 1);
bid->bio = bio;
bid->bip.bip_flags |= BIP_BLOCK_INTEGRITY;
bio_integrity_alloc_buf(bio, gfp & __GFP_ZERO);
bip_set_seed(&bid->bip, bio->bi_iter.bi_sector);
if (set_flags) {
@@ -176,23 +167,12 @@ bool bio_integrity_prep(struct bio *bio)
bid->bip.bip_flags |= BIP_CHECK_REFTAG;
}
if (bio_integrity_add_page(bio, virt_to_page(buf), len,
offset_in_page(buf)) < len)
goto err_end_io;
/* Auto-generate integrity metadata if this is a write */
if (bio_data_dir(bio) == WRITE && bip_should_check(&bid->bip))
blk_integrity_generate(bio);
else
bid->saved_bio_iter = bio->bi_iter;
return true;
err_free_buf:
kfree(buf);
err_end_io:
bio->bi_status = BLK_STS_RESOURCE;
bio_endio(bio);
return false;
}
EXPORT_SYMBOL(bio_integrity_prep);
block/bio-integrity.c
+48
View File
@@ -14,6 +14,45 @@ struct bio_integrity_alloc {
struct bio_vec bvecs[];
};
static mempool_t integrity_buf_pool;
void bio_integrity_alloc_buf(struct bio *bio, bool zero_buffer)
{
struct blk_integrity *bi = blk_get_integrity(bio->bi_bdev->bd_disk);
struct bio_integrity_payload *bip = bio_integrity(bio);
unsigned int len = bio_integrity_bytes(bi, bio_sectors(bio));
gfp_t gfp = GFP_NOIO | (zero_buffer ? __GFP_ZERO : 0);
void *buf;
buf = kmalloc(len, (gfp & ~__GFP_DIRECT_RECLAIM) |
__GFP_NOMEMALLOC | __GFP_NORETRY | __GFP_NOWARN);
if (unlikely(!buf)) {
struct page *page;
page = mempool_alloc(&integrity_buf_pool, GFP_NOFS);
if (zero_buffer)
memset(page_address(page), 0, len);
bvec_set_page(&bip->bip_vec[0], page, len, 0);
bip->bip_flags |= BIP_MEMPOOL;
} else {
bvec_set_page(&bip->bip_vec[0], virt_to_page(buf), len,
offset_in_page(buf));
}
bip->bip_vcnt = 1;
bip->bip_iter.bi_size = len;
}
void bio_integrity_free_buf(struct bio_integrity_payload *bip)
{
struct bio_vec *bv = &bip->bip_vec[0];
if (bip->bip_flags & BIP_MEMPOOL)
mempool_free(bv->bv_page, &integrity_buf_pool);
else
kfree(bvec_virt(bv));
}
/**
* bio_integrity_free - Free bio integrity payload
* @bio: bio containing bip to be freed
@@ -438,3 +477,12 @@ int bio_integrity_clone(struct bio *bio, struct bio *bio_src,
return 0;
}
static int __init bio_integrity_initfn(void)
{
if (mempool_init_page_pool(&integrity_buf_pool, BIO_POOL_SIZE,
get_order(BLK_INTEGRITY_MAX_SIZE)))
panic("bio: can't create integrity buf pool\n");
return 0;
}
subsys_initcall(bio_integrity_initfn);
block/bio.c
+1
View File
@@ -253,6 +253,7 @@ void bio_init(struct bio *bio, struct block_device *bdev, struct bio_vec *table,
bio->bi_write_hint = 0;
bio->bi_write_stream = 0;
bio->bi_status = 0;
bio->bi_bvec_gap_bit = 0;
bio->bi_iter.bi_sector = 0;
bio->bi_iter.bi_size = 0;
bio->bi_iter.bi_idx = 0;
block/blk-core.c
+6 -6
View File
@@ -662,13 +662,13 @@ static void __submit_bio(struct bio *bio)
* bio_list of new bios to be added. ->submit_bio() may indeed add some more
* bios through a recursive call to submit_bio_noacct. If it did, we find a
* non-NULL value in bio_list and re-enter the loop from the top.
* - In this case we really did just take the bio of the top of the list (no
* - In this case we really did just take the bio off the top of the list (no
* pretending) and so remove it from bio_list, and call into ->submit_bio()
* again.
*
* bio_list_on_stack[0] contains bios submitted by the current ->submit_bio.
* bio_list_on_stack[1] contains bios that were submitted before the current
* ->submit_bio, but that haven't been processed yet.
* ->submit_bio(), but that haven't been processed yet.
*/
static void __submit_bio_noacct(struct bio *bio)
{
@@ -743,8 +743,8 @@ void submit_bio_noacct_nocheck(struct bio *bio, bool split)
/*
* We only want one ->submit_bio to be active at a time, else stack
* usage with stacked devices could be a problem. Use current->bio_list
* to collect a list of requests submited by a ->submit_bio method while
* it is active, and then process them after it returned.
* to collect a list of requests submitted by a ->submit_bio method
* while it is active, and then process them after it returned.
*/
if (current->bio_list) {
if (split)
@@ -901,7 +901,7 @@ static void bio_set_ioprio(struct bio *bio)
*
* submit_bio() is used to submit I/O requests to block devices. It is passed a
* fully set up &struct bio that describes the I/O that needs to be done. The
* bio will be send to the device described by the bi_bdev field.
* bio will be sent to the device described by the bi_bdev field.
*
* The success/failure status of the request, along with notification of
* completion, is delivered asynchronously through the ->bi_end_io() callback
@@ -991,7 +991,7 @@ int iocb_bio_iopoll(struct kiocb *kiocb, struct io_comp_batch *iob,
* point to a freshly allocated bio at this point. If that happens
* we have a few cases to consider:
*
* 1) the bio is beeing initialized and bi_bdev is NULL. We can just
* 1) the bio is being initialized and bi_bdev is NULL. We can just
* simply nothing in this case
* 2) the bio points to a not poll enabled device. bio_poll will catch
* this and return 0
block/blk-iocost.c
+2 -4
View File
@@ -2334,10 +2334,8 @@ static void ioc_timer_fn(struct timer_list *timer)
else
usage_dur = max_t(u64, now.now - ioc->period_at, 1);
usage = clamp_t(u32,
DIV64_U64_ROUND_UP(usage_us * WEIGHT_ONE,
usage_dur),
1, WEIGHT_ONE);
usage = clamp(DIV64_U64_ROUND_UP(usage_us * WEIGHT_ONE, usage_dur),
1, WEIGHT_ONE);
/*
* Already donating or accumulated enough to start.
block/blk-lib.c
+3 -3
View File
@@ -87,11 +87,11 @@ int blkdev_issue_discard(struct block_device *bdev, sector_t sector,
{
struct bio *bio = NULL;
struct blk_plug plug;
int ret;
int ret = 0;
blk_start_plug(&plug);
ret = __blkdev_issue_discard(bdev, sector, nr_sects, gfp_mask, &bio);
if (!ret && bio) {
__blkdev_issue_discard(bdev, sector, nr_sects, gfp_mask, &bio);
if (bio) {
ret = submit_bio_wait(bio);
if (ret == -EOPNOTSUPP)
ret = 0;
block/blk-map.c
+3
View File
@@ -459,6 +459,8 @@ int blk_rq_append_bio(struct request *rq, struct bio *bio)
if (rq->bio) {
if (!ll_back_merge_fn(rq, bio, nr_segs))
return -EINVAL;
rq->phys_gap_bit = bio_seg_gap(rq->q, rq->biotail, bio,
rq->phys_gap_bit);
rq->biotail->bi_next = bio;
rq->biotail = bio;
rq->__data_len += bio->bi_iter.bi_size;
@@ -469,6 +471,7 @@ int blk_rq_append_bio(struct request *rq, struct bio *bio)
rq->nr_phys_segments = nr_segs;
rq->bio = rq->biotail = bio;
rq->__data_len = bio->bi_iter.bi_size;
rq->phys_gap_bit = bio->bi_bvec_gap_bit;
return 0;
}
EXPORT_SYMBOL(blk_rq_append_bio);
block/blk-merge.c
+40 -4
View File
@@ -302,6 +302,12 @@ static unsigned int bio_split_alignment(struct bio *bio,
return lim->logical_block_size;
}
static inline unsigned int bvec_seg_gap(struct bio_vec *bvprv,
struct bio_vec *bv)
{
return bv->bv_offset | (bvprv->bv_offset + bvprv->bv_len);
}
/**
* bio_split_io_at - check if and where to split a bio
* @bio: [in] bio to be split
@@ -319,8 +325,8 @@ int bio_split_io_at(struct bio *bio, const struct queue_limits *lim,
unsigned *segs, unsigned max_bytes, unsigned len_align_mask)
{
struct bio_vec bv, bvprv, *bvprvp = NULL;
unsigned nsegs = 0, bytes = 0, gaps = 0;
struct bvec_iter iter;
unsigned nsegs = 0, bytes = 0;
bio_for_each_bvec(bv, bio, iter) {
if (bv.bv_offset & lim->dma_alignment ||
@@ -331,12 +337,15 @@ int bio_split_io_at(struct bio *bio, const struct queue_limits *lim,
* If the queue doesn't support SG gaps and adding this
* offset would create a gap, disallow it.
*/
if (bvprvp && bvec_gap_to_prev(lim, bvprvp, bv.bv_offset))
goto split;
if (bvprvp) {
if (bvec_gap_to_prev(lim, bvprvp, bv.bv_offset))
goto split;
gaps |= bvec_seg_gap(bvprvp, &bv);
}
if (nsegs < lim->max_segments &&
bytes + bv.bv_len <= max_bytes &&
bv.bv_offset + bv.bv_len <= lim->min_segment_size) {
bv.bv_offset + bv.bv_len <= lim->max_fast_segment_size) {
nsegs++;
bytes += bv.bv_len;
} else {
@@ -350,6 +359,7 @@ int bio_split_io_at(struct bio *bio, const struct queue_limits *lim,
}
*segs = nsegs;
bio->bi_bvec_gap_bit = ffs(gaps);
return 0;
split:
if (bio->bi_opf & REQ_ATOMIC)
@@ -385,6 +395,7 @@ split:
* big IO can be trival, disable iopoll when split needed.
*/
bio_clear_polled(bio);
bio->bi_bvec_gap_bit = ffs(gaps);
return bytes >> SECTOR_SHIFT;
}
EXPORT_SYMBOL_GPL(bio_split_io_at);
@@ -721,6 +732,24 @@ static bool blk_atomic_write_mergeable_rqs(struct request *rq,
return (rq->cmd_flags & REQ_ATOMIC) == (next->cmd_flags & REQ_ATOMIC);
}
u8 bio_seg_gap(struct request_queue *q, struct bio *prev, struct bio *next,
u8 gaps_bit)
{
struct bio_vec pb, nb;
if (!bio_has_data(prev))
return 0;
gaps_bit = min_not_zero(gaps_bit, prev->bi_bvec_gap_bit);
gaps_bit = min_not_zero(gaps_bit, next->bi_bvec_gap_bit);
bio_get_last_bvec(prev, &pb);
bio_get_first_bvec(next, &nb);
if (!biovec_phys_mergeable(q, &pb, &nb))
gaps_bit = min_not_zero(gaps_bit, ffs(bvec_seg_gap(&pb, &nb)));
return gaps_bit;
}
/*
* For non-mq, this has to be called with the request spinlock acquired.
* For mq with scheduling, the appropriate queue wide lock should be held.
@@ -785,6 +814,9 @@ static struct request *attempt_merge(struct request_queue *q,
if (next->start_time_ns < req->start_time_ns)
req->start_time_ns = next->start_time_ns;
req->phys_gap_bit = bio_seg_gap(req->q, req->biotail, next->bio,
min_not_zero(next->phys_gap_bit,
req->phys_gap_bit));
req->biotail->bi_next = next->bio;
req->biotail = next->biotail;
@@ -908,6 +940,8 @@ enum bio_merge_status bio_attempt_back_merge(struct request *req,
if (req->rq_flags & RQF_ZONE_WRITE_PLUGGING)
blk_zone_write_plug_bio_merged(bio);
req->phys_gap_bit = bio_seg_gap(req->q, req->biotail, bio,
req->phys_gap_bit);
req->biotail->bi_next = bio;
req->biotail = bio;
req->__data_len += bio->bi_iter.bi_size;
@@ -942,6 +976,8 @@ static enum bio_merge_status bio_attempt_front_merge(struct request *req,
blk_update_mixed_merge(req, bio, true);
req->phys_gap_bit = bio_seg_gap(req->q, bio, req->bio,
req->phys_gap_bit);
bio->bi_next = req->bio;
req->bio = bio;
block/blk-mq-dma.c
+17 -12
View File
@@ -79,8 +79,7 @@ static bool blk_map_iter_next(struct request *req, struct blk_map_iter *iter,
static inline bool blk_can_dma_map_iova(struct request *req,
struct device *dma_dev)
{
return !((queue_virt_boundary(req->q) + 1) &
dma_get_merge_boundary(dma_dev));
return !(req_phys_gap_mask(req) & dma_get_merge_boundary(dma_dev));
}
static bool blk_dma_map_bus(struct blk_dma_iter *iter, struct phys_vec *vec)
@@ -93,8 +92,13 @@ static bool blk_dma_map_bus(struct blk_dma_iter *iter, struct phys_vec *vec)
static bool blk_dma_map_direct(struct request *req, struct device *dma_dev,
struct blk_dma_iter *iter, struct phys_vec *vec)
{
iter->addr = dma_map_page(dma_dev, phys_to_page(vec->paddr),
offset_in_page(vec->paddr), vec->len, rq_dma_dir(req));
unsigned int attrs = 0;
if (iter->p2pdma.map == PCI_P2PDMA_MAP_THRU_HOST_BRIDGE)
attrs |= DMA_ATTR_MMIO;
iter->addr = dma_map_phys(dma_dev, vec->paddr, vec->len,
rq_dma_dir(req), attrs);
if (dma_mapping_error(dma_dev, iter->addr)) {
iter->status = BLK_STS_RESOURCE;
return false;
@@ -109,14 +113,18 @@ static bool blk_rq_dma_map_iova(struct request *req, struct device *dma_dev,
{
enum dma_data_direction dir = rq_dma_dir(req);
unsigned int mapped = 0;
unsigned int attrs = 0;
int error;
iter->addr = state->addr;
iter->len = dma_iova_size(state);
if (iter->p2pdma.map == PCI_P2PDMA_MAP_THRU_HOST_BRIDGE)
attrs |= DMA_ATTR_MMIO;
do {
error = dma_iova_link(dma_dev, state, vec->paddr, mapped,
vec->len, dir, 0);
vec->len, dir, attrs);
if (error)
break;
mapped += vec->len;
@@ -143,7 +151,7 @@ static inline void blk_rq_map_iter_init(struct request *rq,
.bi_size = rq->special_vec.bv_len,
}
};
} else if (bio) {
} else if (bio) {
*iter = (struct blk_map_iter) {
.bio = bio,
.bvecs = bio->bi_io_vec,
@@ -151,7 +159,7 @@ static inline void blk_rq_map_iter_init(struct request *rq,
};
} else {
/* the internal flush request may not have bio attached */
*iter = (struct blk_map_iter) {};
*iter = (struct blk_map_iter) {};
}
}
@@ -163,6 +171,7 @@ static bool blk_dma_map_iter_start(struct request *req, struct device *dma_dev,
memset(&iter->p2pdma, 0, sizeof(iter->p2pdma));
iter->status = BLK_STS_OK;
iter->p2pdma.map = PCI_P2PDMA_MAP_NONE;
/*
* Grab the first segment ASAP because we'll need it to check for P2P
@@ -174,10 +183,6 @@ static bool blk_dma_map_iter_start(struct request *req, struct device *dma_dev,
switch (pci_p2pdma_state(&iter->p2pdma, dma_dev,
phys_to_page(vec.paddr))) {
case PCI_P2PDMA_MAP_BUS_ADDR:
if (iter->iter.is_integrity)
bio_integrity(req->bio)->bip_flags |= BIP_P2P_DMA;
else
req->cmd_flags |= REQ_P2PDMA;
return blk_dma_map_bus(iter, &vec);
case PCI_P2PDMA_MAP_THRU_HOST_BRIDGE:
/*
@@ -352,7 +357,7 @@ bool blk_rq_integrity_dma_map_iter_start(struct request *req,
EXPORT_SYMBOL_GPL(blk_rq_integrity_dma_map_iter_start);
/**
* blk_rq_integrity_dma_map_iter_start - map the next integrity DMA segment for
* blk_rq_integrity_dma_map_iter_next - map the next integrity DMA segment for
* a request
* @req: request to map
* @dma_dev: device to map to
block/blk-mq-sched.c
+96 -24
View File
@@ -427,11 +427,25 @@ void blk_mq_free_sched_tags(struct elevator_tags *et,
kfree(et);
}
void blk_mq_free_sched_tags_batch(struct xarray *et_table,
void blk_mq_free_sched_res(struct elevator_resources *res,
struct elevator_type *type,
struct blk_mq_tag_set *set)
{
if (res->et) {
blk_mq_free_sched_tags(res->et, set);
res->et = NULL;
}
if (res->data) {
blk_mq_free_sched_data(type, res->data);
res->data = NULL;
}
}
void blk_mq_free_sched_res_batch(struct xarray *elv_tbl,
struct blk_mq_tag_set *set)
{
struct request_queue *q;
struct elevator_tags *et;
struct elv_change_ctx *ctx;
lockdep_assert_held_write(&set->update_nr_hwq_lock);
@@ -444,15 +458,48 @@ void blk_mq_free_sched_tags_batch(struct xarray *et_table,
* concurrently.
*/
if (q->elevator) {
et = xa_load(et_table, q->id);
if (unlikely(!et))
ctx = xa_load(elv_tbl, q->id);
if (!ctx) {
WARN_ON_ONCE(1);
else
blk_mq_free_sched_tags(et, set);
continue;
}
blk_mq_free_sched_res(&ctx->res, ctx->type, set);
}
}
}
void blk_mq_free_sched_ctx_batch(struct xarray *elv_tbl)
{
unsigned long i;
struct elv_change_ctx *ctx;
xa_for_each(elv_tbl, i, ctx) {
xa_erase(elv_tbl, i);
kfree(ctx);
}
}
int blk_mq_alloc_sched_ctx_batch(struct xarray *elv_tbl,
struct blk_mq_tag_set *set)
{
struct request_queue *q;
struct elv_change_ctx *ctx;
lockdep_assert_held_write(&set->update_nr_hwq_lock);
list_for_each_entry(q, &set->tag_list, tag_set_list) {
ctx = kzalloc(sizeof(struct elv_change_ctx), GFP_KERNEL);
if (!ctx)
return -ENOMEM;
if (xa_insert(elv_tbl, q->id, ctx, GFP_KERNEL)) {
kfree(ctx);
return -ENOMEM;
}
}
return 0;
}
struct elevator_tags *blk_mq_alloc_sched_tags(struct blk_mq_tag_set *set,
unsigned int nr_hw_queues, unsigned int nr_requests)
{
@@ -466,8 +513,7 @@ struct elevator_tags *blk_mq_alloc_sched_tags(struct blk_mq_tag_set *set,
else
nr_tags = nr_hw_queues;
et = kmalloc(sizeof(struct elevator_tags) +
nr_tags * sizeof(struct blk_mq_tags *), gfp);
et = kmalloc(struct_size(et, tags, nr_tags), gfp);
if (!et)
return NULL;
@@ -498,12 +544,33 @@ out:
return NULL;
}
int blk_mq_alloc_sched_tags_batch(struct xarray *et_table,
int blk_mq_alloc_sched_res(struct request_queue *q,
struct elevator_type *type,
struct elevator_resources *res,
unsigned int nr_hw_queues)
{
struct blk_mq_tag_set *set = q->tag_set;
res->et = blk_mq_alloc_sched_tags(set, nr_hw_queues,
blk_mq_default_nr_requests(set));
if (!res->et)
return -ENOMEM;
res->data = blk_mq_alloc_sched_data(q, type);
if (IS_ERR(res->data)) {
blk_mq_free_sched_tags(res->et, set);
return -ENOMEM;
}
return 0;
}
int blk_mq_alloc_sched_res_batch(struct xarray *elv_tbl,
struct blk_mq_tag_set *set, unsigned int nr_hw_queues)
{
struct elv_change_ctx *ctx;
struct request_queue *q;
struct elevator_tags *et;
gfp_t gfp = GFP_NOIO | __GFP_ZERO | __GFP_NOWARN | __GFP_NORETRY;
int ret = -ENOMEM;
lockdep_assert_held_write(&set->update_nr_hwq_lock);
@@ -516,39 +583,44 @@ int blk_mq_alloc_sched_tags_batch(struct xarray *et_table,
* concurrently.
*/
if (q->elevator) {
et = blk_mq_alloc_sched_tags(set, nr_hw_queues,
blk_mq_default_nr_requests(set));
if (!et)
ctx = xa_load(elv_tbl, q->id);
if (WARN_ON_ONCE(!ctx)) {
ret = -ENOENT;
goto out_unwind;
}
ret = blk_mq_alloc_sched_res(q, q->elevator->type,
&ctx->res, nr_hw_queues);
if (ret)
goto out_unwind;
if (xa_insert(et_table, q->id, et, gfp))
goto out_free_tags;
}
}
return 0;
out_free_tags:
blk_mq_free_sched_tags(et, set);
out_unwind:
list_for_each_entry_continue_reverse(q, &set->tag_list, tag_set_list) {
if (q->elevator) {
et = xa_load(et_table, q->id);
if (et)
blk_mq_free_sched_tags(et, set);
ctx = xa_load(elv_tbl, q->id);
if (ctx)
blk_mq_free_sched_res(&ctx->res,
ctx->type, set);
}
}
return -ENOMEM;
return ret;
}
/* caller must have a reference to @e, will grab another one if successful */
int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e,
struct elevator_tags *et)
struct elevator_resources *res)
{
unsigned int flags = q->tag_set->flags;
struct elevator_tags *et = res->et;
struct blk_mq_hw_ctx *hctx;
struct elevator_queue *eq;
unsigned long i;
int ret;
eq = elevator_alloc(q, e, et);
eq = elevator_alloc(q, e, res);
if (!eq)
return -ENOMEM;
block/blk-mq-sched.h
+37 -3
View File
@@ -19,18 +19,52 @@ void __blk_mq_sched_restart(struct blk_mq_hw_ctx *hctx);
void blk_mq_sched_dispatch_requests(struct blk_mq_hw_ctx *hctx);
int blk_mq_init_sched(struct request_queue *q, struct elevator_type *e,
struct elevator_tags *et);
struct elevator_resources *res);
void blk_mq_exit_sched(struct request_queue *q, struct elevator_queue *e);
void blk_mq_sched_free_rqs(struct request_queue *q);
struct elevator_tags *blk_mq_alloc_sched_tags(struct blk_mq_tag_set *set,
unsigned int nr_hw_queues, unsigned int nr_requests);
int blk_mq_alloc_sched_tags_batch(struct xarray *et_table,
int blk_mq_alloc_sched_res(struct request_queue *q,
struct elevator_type *type,
struct elevator_resources *res,
unsigned int nr_hw_queues);
int blk_mq_alloc_sched_res_batch(struct xarray *elv_tbl,
struct blk_mq_tag_set *set, unsigned int nr_hw_queues);
int blk_mq_alloc_sched_ctx_batch(struct xarray *elv_tbl,
struct blk_mq_tag_set *set);
void blk_mq_free_sched_ctx_batch(struct xarray *elv_tbl);
void blk_mq_free_sched_tags(struct elevator_tags *et,
struct blk_mq_tag_set *set);
void blk_mq_free_sched_tags_batch(struct xarray *et_table,
void blk_mq_free_sched_res(struct elevator_resources *res,
struct elevator_type *type,
struct blk_mq_tag_set *set);
void blk_mq_free_sched_res_batch(struct xarray *et_table,
struct blk_mq_tag_set *set);
/*
* blk_mq_alloc_sched_data() - Allocates scheduler specific data
* Returns:
* - Pointer to allocated data on success
* - NULL if no allocation needed
* - ERR_PTR(-ENOMEM) in case of failure
*/
static inline void *blk_mq_alloc_sched_data(struct request_queue *q,
struct elevator_type *e)
{
void *sched_data;
if (!e || !e->ops.alloc_sched_data)
return NULL;
sched_data = e->ops.alloc_sched_data(q);
return (sched_data) ?: ERR_PTR(-ENOMEM);
}
static inline void blk_mq_free_sched_data(struct elevator_type *e, void *data)
{
if (e && e->ops.free_sched_data)
e->ops.free_sched_data(data);
}
static inline void blk_mq_sched_restart(struct blk_mq_hw_ctx *hctx)
{
block/blk-mq-tag.c
+1 -1
View File
@@ -499,7 +499,7 @@ void blk_mq_queue_tag_busy_iter(struct request_queue *q, busy_tag_iter_fn *fn,
int srcu_idx;
/*
* __blk_mq_update_nr_hw_queues() updates nr_hw_queues and hctx_table
* __blk_mq_update_nr_hw_queues() updates nr_hw_queues and queue_hw_ctx
* while the queue is frozen. So we can use q_usage_counter to avoid
* racing with it.
*/
block/blk-mq.c
+96 -56
View File
@@ -376,6 +376,7 @@ void blk_rq_init(struct request_queue *q, struct request *rq)
INIT_LIST_HEAD(&rq->queuelist);
rq->q = q;
rq->__sector = (sector_t) -1;
rq->phys_gap_bit = 0;
INIT_HLIST_NODE(&rq->hash);
RB_CLEAR_NODE(&rq->rb_node);
rq->tag = BLK_MQ_NO_TAG;
@@ -467,21 +468,26 @@ __blk_mq_alloc_requests_batch(struct blk_mq_alloc_data *data)
unsigned long tag_mask;
int i, nr = 0;
tag_mask = blk_mq_get_tags(data, data->nr_tags, &tag_offset);
if (unlikely(!tag_mask))
return NULL;
do {
tag_mask = blk_mq_get_tags(data, data->nr_tags - nr, &tag_offset);
if (unlikely(!tag_mask)) {
if (nr == 0)
return NULL;
break;
}
tags = blk_mq_tags_from_data(data);
for (i = 0; tag_mask; i++) {
if (!(tag_mask & (1UL << i)))
continue;
tag = tag_offset + i;
prefetch(tags->static_rqs[tag]);
tag_mask &= ~(1UL << i);
rq = blk_mq_rq_ctx_init(data, tags, tag);
rq_list_add_head(data->cached_rqs, rq);
nr++;
}
} while (data->nr_tags > nr);
tags = blk_mq_tags_from_data(data);
for (i = 0; tag_mask; i++) {
if (!(tag_mask & (1UL << i)))
continue;
tag = tag_offset + i;
prefetch(tags->static_rqs[tag]);
tag_mask &= ~(1UL << i);
rq = blk_mq_rq_ctx_init(data, tags, tag);
rq_list_add_head(data->cached_rqs, rq);
nr++;
}
if (!(data->rq_flags & RQF_SCHED_TAGS))
blk_mq_add_active_requests(data->hctx, nr);
/* caller already holds a reference, add for remainder */
@@ -668,6 +674,7 @@ struct request *blk_mq_alloc_request(struct request_queue *q, blk_opf_t opf,
goto out_queue_exit;
}
rq->__data_len = 0;
rq->phys_gap_bit = 0;
rq->__sector = (sector_t) -1;
rq->bio = rq->biotail = NULL;
return rq;
@@ -723,7 +730,7 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
* If not tell the caller that it should skip this queue.
*/
ret = -EXDEV;
data.hctx = xa_load(&q->hctx_table, hctx_idx);
data.hctx = q->queue_hw_ctx[hctx_idx];
if (!blk_mq_hw_queue_mapped(data.hctx))
goto out_queue_exit;
cpu = cpumask_first_and(data.hctx->cpumask, cpu_online_mask);
@@ -748,6 +755,7 @@ struct request *blk_mq_alloc_request_hctx(struct request_queue *q,
rq = blk_mq_rq_ctx_init(&data, blk_mq_tags_from_data(&data), tag);
blk_mq_rq_time_init(rq, alloc_time_ns);
rq->__data_len = 0;
rq->phys_gap_bit = 0;
rq->__sector = (sector_t) -1;
rq->bio = rq->biotail = NULL;
return rq;
@@ -2674,6 +2682,8 @@ static void blk_mq_bio_to_request(struct request *rq, struct bio *bio,
rq->bio = rq->biotail = bio;
rq->__sector = bio->bi_iter.bi_sector;
rq->__data_len = bio->bi_iter.bi_size;
rq->phys_gap_bit = bio->bi_bvec_gap_bit;
rq->nr_phys_segments = nr_segs;
if (bio_integrity(bio))
rq->nr_integrity_segments = blk_rq_count_integrity_sg(rq->q,
@@ -3380,6 +3390,7 @@ int blk_rq_prep_clone(struct request *rq, struct request *rq_src,
}
rq->nr_phys_segments = rq_src->nr_phys_segments;
rq->nr_integrity_segments = rq_src->nr_integrity_segments;
rq->phys_gap_bit = rq_src->phys_gap_bit;
if (rq->bio && blk_crypto_rq_bio_prep(rq, rq->bio, gfp_mask) < 0)
goto free_and_out;
@@ -3935,8 +3946,6 @@ static void blk_mq_exit_hctx(struct request_queue *q,
blk_free_flush_queue_callback);
hctx->fq = NULL;
xa_erase(&q->hctx_table, hctx_idx);
spin_lock(&q->unused_hctx_lock);
list_add(&hctx->hctx_list, &q->unused_hctx_list);
spin_unlock(&q->unused_hctx_lock);
@@ -3978,14 +3987,8 @@ static int blk_mq_init_hctx(struct request_queue *q,
hctx->numa_node))
goto exit_hctx;
if (xa_insert(&q->hctx_table, hctx_idx, hctx, GFP_KERNEL))
goto exit_flush_rq;
return 0;
exit_flush_rq:
if (set->ops->exit_request)
set->ops->exit_request(set, hctx->fq->flush_rq, hctx_idx);
exit_hctx:
if (set->ops->exit_hctx)
set->ops->exit_hctx(hctx, hctx_idx);
@@ -4374,7 +4377,7 @@ void blk_mq_release(struct request_queue *q)
kobject_put(&hctx->kobj);
}
xa_destroy(&q->hctx_table);
kfree(q->queue_hw_ctx);
/*
* release .mq_kobj and sw queue's kobject now because
@@ -4518,26 +4521,49 @@ static struct blk_mq_hw_ctx *blk_mq_alloc_and_init_hctx(
static void __blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
struct request_queue *q)
{
struct blk_mq_hw_ctx *hctx;
unsigned long i, j;
int i, j, end;
struct blk_mq_hw_ctx **hctxs = q->queue_hw_ctx;
if (q->nr_hw_queues < set->nr_hw_queues) {
struct blk_mq_hw_ctx **new_hctxs;
new_hctxs = kcalloc_node(set->nr_hw_queues,
sizeof(*new_hctxs), GFP_KERNEL,
set->numa_node);
if (!new_hctxs)
return;
if (hctxs)
memcpy(new_hctxs, hctxs, q->nr_hw_queues *
sizeof(*hctxs));
rcu_assign_pointer(q->queue_hw_ctx, new_hctxs);
/*
* Make sure reading the old queue_hw_ctx from other
* context concurrently won't trigger uaf.
*/
synchronize_rcu_expedited();
kfree(hctxs);
hctxs = new_hctxs;
}
for (i = 0; i < set->nr_hw_queues; i++) {
int old_node;
int node = blk_mq_get_hctx_node(set, i);
struct blk_mq_hw_ctx *old_hctx = xa_load(&q->hctx_table, i);
struct blk_mq_hw_ctx *old_hctx = hctxs[i];
if (old_hctx) {
old_node = old_hctx->numa_node;
blk_mq_exit_hctx(q, set, old_hctx, i);
}
if (!blk_mq_alloc_and_init_hctx(set, q, i, node)) {
hctxs[i] = blk_mq_alloc_and_init_hctx(set, q, i, node);
if (!hctxs[i]) {
if (!old_hctx)
break;
pr_warn("Allocate new hctx on node %d fails, fallback to previous one on node %d\n",
node, old_node);
hctx = blk_mq_alloc_and_init_hctx(set, q, i, old_node);
WARN_ON_ONCE(!hctx);
hctxs[i] = blk_mq_alloc_and_init_hctx(set, q, i,
old_node);
WARN_ON_ONCE(!hctxs[i]);
}
}
/*
@@ -4546,13 +4572,21 @@ static void __blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
*/
if (i != set->nr_hw_queues) {
j = q->nr_hw_queues;
end = i;
} else {
j = i;
end = q->nr_hw_queues;
q->nr_hw_queues = set->nr_hw_queues;
}
xa_for_each_start(&q->hctx_table, j, hctx, j)
blk_mq_exit_hctx(q, set, hctx, j);
for (; j < end; j++) {
struct blk_mq_hw_ctx *hctx = hctxs[j];
if (hctx) {
blk_mq_exit_hctx(q, set, hctx, j);
hctxs[j] = NULL;
}
}
}
static void blk_mq_realloc_hw_ctxs(struct blk_mq_tag_set *set,
@@ -4588,8 +4622,6 @@ int blk_mq_init_allocated_queue(struct blk_mq_tag_set *set,
INIT_LIST_HEAD(&q->unused_hctx_list);
spin_lock_init(&q->unused_hctx_lock);
xa_init(&q->hctx_table);
blk_mq_realloc_hw_ctxs(set, q);
if (!q->nr_hw_queues)
goto err_hctxs;
@@ -4983,27 +5015,28 @@ struct elevator_tags *blk_mq_update_nr_requests(struct request_queue *q,
* Switch back to the elevator type stored in the xarray.
*/
static void blk_mq_elv_switch_back(struct request_queue *q,
struct xarray *elv_tbl, struct xarray *et_tbl)
struct xarray *elv_tbl)
{
struct elevator_type *e = xa_load(elv_tbl, q->id);
struct elevator_tags *t = xa_load(et_tbl, q->id);
struct elv_change_ctx *ctx = xa_load(elv_tbl, q->id);
if (WARN_ON_ONCE(!ctx))
return;
/* The elv_update_nr_hw_queues unfreezes the queue. */
elv_update_nr_hw_queues(q, e, t);
elv_update_nr_hw_queues(q, ctx);
/* Drop the reference acquired in blk_mq_elv_switch_none. */
if (e)
elevator_put(e);
if (ctx->type)
elevator_put(ctx->type);
}
/*
* Stores elevator type in xarray and set current elevator to none. It uses
* q->id as an index to store the elevator type into the xarray.
* Stores elevator name and type in ctx and set current elevator to none.
*/
static int blk_mq_elv_switch_none(struct request_queue *q,
struct xarray *elv_tbl)
{
int ret = 0;
struct elv_change_ctx *ctx;
lockdep_assert_held_write(&q->tag_set->update_nr_hwq_lock);
@@ -5015,10 +5048,11 @@ static int blk_mq_elv_switch_none(struct request_queue *q,
* can't run concurrently.
*/
if (q->elevator) {
ctx = xa_load(elv_tbl, q->id);
if (WARN_ON_ONCE(!ctx))
return -ENOENT;
ret = xa_insert(elv_tbl, q->id, q->elevator->type, GFP_KERNEL);
if (WARN_ON_ONCE(ret))
return ret;
ctx->name = q->elevator->type->elevator_name;
/*
* Before we switch elevator to 'none', take a reference to
@@ -5029,9 +5063,14 @@ static int blk_mq_elv_switch_none(struct request_queue *q,
*/
__elevator_get(q->elevator->type);
/*
* Store elevator type so that we can release the reference
* taken above later.
*/
ctx->type = q->elevator->type;
elevator_set_none(q);
}
return ret;
return 0;
}
static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,
@@ -5041,7 +5080,7 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,
int prev_nr_hw_queues = set->nr_hw_queues;
unsigned int memflags;
int i;
struct xarray elv_tbl, et_tbl;
struct xarray elv_tbl;
bool queues_frozen = false;
lockdep_assert_held(&set->tag_list_lock);
@@ -5055,11 +5094,12 @@ static void __blk_mq_update_nr_hw_queues(struct blk_mq_tag_set *set,
memflags = memalloc_noio_save();
xa_init(&et_tbl);
if (blk_mq_alloc_sched_tags_batch(&et_tbl, set, nr_hw_queues) < 0)
goto out_memalloc_restore;
xa_init(&elv_tbl);
if (blk_mq_alloc_sched_ctx_batch(&elv_tbl, set) < 0)
goto out_free_ctx;
if (blk_mq_alloc_sched_res_batch(&elv_tbl, set, nr_hw_queues) < 0)
goto out_free_ctx;
list_for_each_entry(q, &set->tag_list, tag_set_list) {
blk_mq_debugfs_unregister_hctxs(q);
@@ -5105,7 +5145,7 @@ switch_back:
/* switch_back expects queue to be frozen */
if (!queues_frozen)
blk_mq_freeze_queue_nomemsave(q);
blk_mq_elv_switch_back(q, &elv_tbl, &et_tbl);
blk_mq_elv_switch_back(q, &elv_tbl);
}
list_for_each_entry(q, &set->tag_list, tag_set_list) {
@@ -5116,9 +5156,9 @@ switch_back:
blk_mq_add_hw_queues_cpuhp(q);
}
out_free_ctx:
blk_mq_free_sched_ctx_batch(&elv_tbl);
xa_destroy(&elv_tbl);
xa_destroy(&et_tbl);
out_memalloc_restore:
memalloc_noio_restore(memflags);
/* Free the excess tags when nr_hw_queues shrink. */
@@ -5168,7 +5208,7 @@ int blk_mq_poll(struct request_queue *q, blk_qc_t cookie,
{
if (!blk_mq_can_poll(q))
return 0;
return blk_hctx_poll(q, xa_load(&q->hctx_table, cookie), iob, flags);
return blk_hctx_poll(q, q->queue_hw_ctx[cookie], iob, flags);
}
int blk_rq_poll(struct request *rq, struct io_comp_batch *iob,
block/blk-mq.h
+1 -1
View File
@@ -84,7 +84,7 @@ static inline struct blk_mq_hw_ctx *blk_mq_map_queue_type(struct request_queue *
enum hctx_type type,
unsigned int cpu)
{
return xa_load(&q->hctx_table, q->tag_set->map[type].mq_map[cpu]);
return queue_hctx((q), (q->tag_set->map[type].mq_map[cpu]));
}
static inline enum hctx_type blk_mq_get_hctx_type(blk_opf_t opf)
block/blk-settings.c
+25 -2
View File
@@ -123,6 +123,19 @@ static int blk_validate_zoned_limits(struct queue_limits *lim)
return 0;
}
/*
* Maximum size of I/O that needs a block layer integrity buffer. Limited
* by the number of intervals for which we can fit the integrity buffer into
* the buffer size. Because the buffer is a single segment it is also limited
* by the maximum segment size.
*/
static inline unsigned int max_integrity_io_size(struct queue_limits *lim)
{
return min_t(unsigned int, lim->max_segment_size,
(BLK_INTEGRITY_MAX_SIZE / lim->integrity.metadata_size) <<
lim->integrity.interval_exp);
}
static int blk_validate_integrity_limits(struct queue_limits *lim)
{
struct blk_integrity *bi = &lim->integrity;
@@ -194,6 +207,14 @@ static int blk_validate_integrity_limits(struct queue_limits *lim)
(1U << bi->interval_exp) - 1);
}
/*
* The block layer automatically adds integrity data for bios that don't
* already have it. Limit the I/O size so that a single maximum size
* metadata segment can cover the integrity data for the entire I/O.
*/
lim->max_sectors = min(lim->max_sectors,
max_integrity_io_size(lim) >> SECTOR_SHIFT);
return 0;
}
@@ -467,12 +488,12 @@ int blk_validate_limits(struct queue_limits *lim)
return -EINVAL;
}
/* setup min segment size for building new segment in fast path */
/* setup max segment size for building new segment in fast path */
if (lim->seg_boundary_mask > lim->max_segment_size - 1)
seg_size = lim->max_segment_size;
else
seg_size = lim->seg_boundary_mask + 1;
lim->min_segment_size = min_t(unsigned int, seg_size, PAGE_SIZE);
lim->max_fast_segment_size = min_t(unsigned int, seg_size, PAGE_SIZE);
/*
* We require drivers to at least do logical block aligned I/O, but
@@ -535,6 +556,8 @@ int queue_limits_commit_update(struct request_queue *q,
{
int error;
lockdep_assert_held(&q->limits_lock);
error = blk_validate_limits(lim);
if (error)
goto out_unlock;

Some files were not shown because too many files have changed in this diff Show More