disk_release() undoes blk_mq_init_allocated_queue() for a disk whose
probe failed before add_disk(), but it only calls blk_mq_exit_queue().
Nothing there stops q->timeout, and that timer rolls forward: it stays
pending until it next expires, not until the last request completes.
So if the driver issued any I/O before adding the disk, the
request_queue is freed while still linked into a timer wheel bucket.
Commit 6f8191fdf4 ("block: simplify disk shutdown") dropped the
blk_cleanup_queue() call that used to stop it. __del_gendisk() and
blk_mq_destroy_queue() still do; only the probe failure path lost it.
nvme gets there because nvme_update_ns_info() submits Report Zones or
FDP io-mgmt-recv on ns->queue before the disk is added, so a later
failure - a concurrent reset setting NVME_CTRL_FROZEN, or
device_add_disk() failing - lands in put_disk() with the timer armed:
BUG: KASAN: slab-use-after-free in detach_if_pending+0x30c/0x340
Write of size 8 at addr ffff888004d71310 by task kworker/u8:2/37
__timer_delete_sync+0x156/0x240 kernel/time/timer.c:1621
blk_sync_queue+0x22/0x40 block/blk-core.c:222
nvme_sync_queues+0x100/0x150 drivers/nvme/host/core.c:5362
nvme_reset_work+0x138/0x930 drivers/nvme/host/pci.c:3264
Allocated by task 34:
__blk_mq_alloc_disk+0x33/0x100 block/blk-mq.c:4462
nvme_alloc_ns+0x290/0x3870 drivers/nvme/host/core.c:4146
Freed by task 0:
blk_free_queue_rcu+0x3a/0x50 block/blk-core.c:254
rcu_core+0xc10/0x1730 kernel/rcu/tree.c:2857
The queue being synced there is ctrl->admin_q, only a victim sharing a
timer wheel bucket with the freed queue's dangling entry; other runs
tripped in enqueue_timer(), __run_timers() or blk_mq_timeout_work().
Failing nvme_alloc_ns() with a debug patch makes it deterministic: one
leaked timer trips KASAN within seconds, while 1987 patched releases
produced no splat.
Stop the timer and the queue work items before blk_mq_exit_queue(), like
blk_mq_destroy_queue() does.
Found by FuzzNvme.
Fixes: 6f8191fdf4 ("block: simplify disk shutdown")
Acked-by: Weidong Zhu <weizhu@fiu.edu>
Signed-off-by: Chao Shi <coshi036@gmail.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/20260727201257.211635-1-coshi036@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
fscrypt uses inline encryption hardware only when the "inlinecrypt"
mount option is given. I'd like to keep that behavior even after
standardizing on the blk-crypto API for file contents encryption. That
is, the default should continue to be the well-tested CPU-based
encryption code, and the use of inline encryption hardware should
continue to be an opt-in feature for systems where it's beneficial and
has been fully validated (including verifying ciphertext correctness).
To support this use case, extend blk_crypto_config with a new flag
BLK_CRYPTO_CFG_ALLOW_HW.
For now it's always set. Later commits will change that.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/20260713023708.9245-4-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
Since blk-crypto-fallback supports all blk_crypto_keys except wrapped
keys, just check for that condition directly instead of using
__blk_crypto_cfg_supported(). With this done,
__blk_crypto_cfg_supported() is now used only for the hardware support.
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/20260713023708.9245-2-ebiggers@kernel.org
Signed-off-by: Eric Biggers <ebiggers@kernel.org>
bio_iov_iter_align_down expects the "normal" biovec layout from vector 0,
while bio_iov_iter_bounce_read abuses vector 0 for a bounce buffer
allocation. Pass an explicit bvec to bio_iov_iter_align_down to deal
with this case to avoid a double unpin.
Additionally we need to free the folio if no bio_vec could be added,
and adjust the size of the first bio_vec that contains the bounce buffer
when the I/O size is aligned down.
Fixes: e7b8b3c5b2 ("block: align down bounces bios")
Reported-by: 0wnerD1ed <l7z@0b1t.tech>
Signed-off-by: Christoph Hellwig <hch@lst.de>
Tested-by: 0wnerD1ed <l7z@0b1t.tech>
Reviewed-by: Keith Busch <kbusch@kernel.org>
Link: https://patch.msgid.link/20260716091306.316625-3-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
When CONFIG_PERSISTENT_HUGE_ZERO_FOLIO is enabled, iomap_dio_zero() can
add a huge zero folio to a zeroing bio, which needs special treatment
in bio_free_folios by also checking is_huge_zero_folio() in addition to
is_zero_folio().
Fixes: 8dd5e7c75d ("block: add helpers to bounce buffer an iov_iter into bios")
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Hannes Reinecke <hare@kernel.org>
Link: https://patch.msgid.link/20260716091306.316625-2-hch@lst.de
Signed-off-by: Jens Axboe <axboe@kernel.dk>
When the per-CPU bio cache is enabled but empty, bio_alloc_percpu_cache()
returns NULL and bio_alloc_bioset() falls straight through to the mempool
fallback:
if (unlikely(!bio)) {
if (!(saved_gfp & __GFP_DIRECT_RECLAIM))
return NULL;
...
}
For non-sleeping allocations (no __GFP_DIRECT_RECLAIM) this returns NULL
without ever attempting a slab allocation, even when there is plenty of
free memory.
Commit b520c4eef8 ("block: split bio_alloc_bioset more clearly into a
fast and slowpath") introduced this. Before it, a percpu cache miss fell
through to mempool_alloc(), which attempted the underlying slab allocation
first and only failed when that slab allocation failed. The restructuring
dropped the slab attempt that non-sleeping callers of a cache-enabled
bioset (such as the default fs_bio_set used by bio_alloc()) relied on.
Try a slab allocation with optimistic GFP_ flags before falling back to
the mempool whenever the bio is still NULL, so both the cache-empty and
non-cache paths share the same slab attempt. This restores the previous
behavior for non-sleeping allocations.
Fixes: b520c4eef8 ("block: split bio_alloc_bioset more clearly into a fast and slowpath")
Suggested-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Joseph Qi <joseph.qi@linux.alibaba.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/20260709020145.4011533-1-joseph.qi@linux.alibaba.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
When elevator_change() is called concurrently for the same queue, the
elevator_change_done() function runs concurrently as well. This function
adds or deletes kobjects for the debugfs entry of the queue. Then the
concurrent calls cause memory corruption of the kobjects and result in a
process hang. The core part of the elevator switch is protected by queue
freeze and q->elevator_lock. However, since the commit 559dc11143
("block: move elv_register[unregister]_queue out of elevator_lock"), the
elevator_change_done() is not serialized. Hence the memory corruption
and the hang.
The failures are observed when udev-worker writes to a sysfs
queue/scheduler attribute file while the blktests test case block/005
writes to the same attribute file. The failure also can be recreated by
running two processes that write to the same queue/scheduler file
concurrently. The failure is observed since another commit 370ac285f2
("block: avoid cpu_hotplug_lock depedency on freeze_lock"). This commit
changed the behavior of queue freeze and it unveiled the failure.
Fix the failure by changing elv_iosched_store() to acquire
update_nr_hwq_lock as the writer lock instead of the reader lock. This
serializes the whole elevator switch steps, including the
elevator_change_done() call.
Fixes: 559dc11143 ("block: move elv_register[unregister]_queue out of elevator_lock")
Signed-off-by: Shin'ichiro Kawasaki <shinichiro.kawasaki@wdc.com>
Reviewed-by: Nilay Shroff <nilay@linux.ibm.com>
Reviewed-by: Ming Lei <tom.leiming@gmail.com>
Link: https://patch.msgid.link/20260716092237.1305030-1-shinichiro.kawasaki@wdc.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
bio_copy_kern() allocates pages that are normally freed by the bio
completion callback. If blk_rq_append_bio() rejects the bio, however,
blk_rq_map_kern() only drops the bio reference. Since bio_put() does not
free pages referenced by the bio vectors, those pages leak.
This can happen when the bio exceeds the queue segment constraints or
when a later mapping cannot be merged into a request built by earlier
calls. Track whether the buffer was copied and free those pages before
dropping the rejected bio.
Fixes: 3a5a39276d ("block: allow blk_rq_map_kern to append to requests")
Assisted-by: Codex:gpt-5.6-sol
Signed-off-by: Jackie Liu <liuyun01@kylinos.cn>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/20260715073518.96042-1-liu.yun@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
During one of my local btrfs fstests runs, folio_alloc() inside
folio_alloc_greedy() triggered an allocation failure report when trying
to allocate an order-4 folio.
The kernel is from the latest development branch, which is utilizing
the IOMAP_DIO_BOUNCE flag for direct writes when the inode requires
checksum.
Unfortunately I didn't save the full log, only the function and the
order.
When the IOMAP_DIO_BOUNCE flag is utilized, we will hit the following call
chain:
bio_iov_iter_bounce_write()
|- folio_alloc_greedy()
|- folio_alloc(gfp | __GFP_NORETRY, get_order(*size));
However __GFP_NORETRY will still emit an allocation failure report
when it fails.
And folio_alloc_greedy() will retry with a smaller order anyway, there
is no point in emitting that allocation failure report.
Append the __GFP_NOWARN flag to folio_alloc() for the larger-order folio
attempts.
Fixes: 8dd5e7c75d ("block: add helpers to bounce buffer an iov_iter into bios")
Signed-off-by: Qu Wenruo <wqu@suse.com>
Link: https://patch.msgid.link/d10571445ee505d95ba6eaad7558fc1f556d2921.1784020005.git.wqu@suse.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
aix_partition() reads the logical-volume descriptor array as a single
sector and then scans it:
if (numlvs && (d = read_part_sector(state, vgda_sector + 1, §))) {
struct lvd *p = (struct lvd *)d;
...
for (i = 0; foundlvs < numlvs && i < state->limit; i++) {
lvip[i].pps_per_lv = be16_to_cpu(p[i].num_lps);
p points at a single 512-byte sector, which holds SECTOR_SIZE /
sizeof(struct lvd) = 16 entries, but the loop runs until foundlvs reaches
the on-disk numlvs or i reaches state->limit (DISK_MAX_PARTS, 256).
numlvs is an on-disk __be16 read straight from the volume group
descriptor and is not validated, so a crafted AIX image with numlvs
larger than 16 and lvd entries whose num_lps fields are zero (so foundlvs
never advances) drives the loop to read p[i] well past the end of the
read sector buffer.
Commit d97a86c170 ("partitions: aix.c: off by one bug") hardened the
matching write of lvip[lv_ix] in 2014 but left this read loop unbounded.
Bound the scan to the number of struct lvd entries that fit in the
sector that was actually read.
Fixes: 6ceea22bbb ("partitions: add aix lvm partition support files")
Assisted-by: Claude:claude-opus-4-8
Signed-off-by: Michael Bommarito <michael.bommarito@gmail.com>
Link: https://patch.msgid.link/20260714114806.3761553-1-michael.bommarito@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
When radix_tree_insert() fails in blkg_create(), the error path has two
issues:
1. blkg->online is set to true unconditionally, even when the blkg was
never fully inserted. Move the assignment inside the success block.
2. The error path calls blkg_put() without first calling
percpu_ref_kill(). Because the refcount is still in percpu mode,
percpu_ref_put() only does this_cpu_sub() without checking for zero,
so blkg_release() is never triggered. This permanently leaks the
blkg memory, its percpu iostat, policy data, the parent blkg
reference, and the cgroup css reference — the latter preventing the
cgroup from ever being destroyed.
Fix by replacing blkg_put() with percpu_ref_kill(), matching the pattern
used in blkg_destroy().
Acked-by: Tejun Heo <tj@kernel.org>
Signed-off-by: Tao Cui <cuitao@kylinos.cn>
Link: https://patch.msgid.link/20260715132407.1469777-1-cui.tao@linux.dev
Signed-off-by: Jens Axboe <axboe@kernel.dk>
blk_time_get_ns() populates the per-plug cached timestamp and then
returns it by re-reading the field:
if (!plug->cur_ktime) {
plug->cur_ktime = ktime_get_ns();
current->flags |= PF_BLOCK_TS;
}
return plug->cur_ktime;
This is problematic when the compiler emits the final
"return plug->cur_ktime" as a reload from memory, after PF_BLOCK_TS has
already been set.
Since the cached timestamp is now invalidated from finish_task_switch()
(fad156c2af "block: invalidate cached plug timestamp after task
switch"), a task preempted between setting PF_BLOCK_TS and that reload
has plug->cur_ktime zeroed by blk_plug_invalidate_ts() when it is
scheduled back in. The reload then returns 0.
A 0 handed back here is stored as a start timestamp -- e.g.
blk_account_io_start() writes it to rq->start_time_ns -- and later
subtracted from "now". blk_account_io_done() then adds (now - 0), i.e.
roughly the system uptime, to the per-group nsecs[] counters. On an
otherwise idle, healthy device this appears as sudden ~uptime-sized jumps
in the diskstats time fields (write_ticks/discard_ticks/time_in_queue).
The solution is to be explicit in our reads and writes to this field
that is preemption volatile. We also add a barrier() to ensure that any
setting of PF_BLOCK_TS is ordered to happen after the cur_ktime update.
This issue was discovered using AI-assisted kprobes looking for paths
that were leaking zeroed timestamps in a live system, based on the
observation that we were sometimes seeing uptime-sized jumps in kernel
exported counters. This was flagged by NodeDiskIOSaturation
prometheus alerts that started firing on all hosts post 7.1.3 kernel
upgrade, due to node-exporter now exporting a nonsensical
node_disk_io_time_weighted_seconds_total.
Fixes: fad156c2af ("block: invalidate cached plug timestamp after task switch")
Cc: stable@vger.kernel.org
Signed-off-by: Mike Waychison <mike@waychison.com>
Assisted-by: Claude:claude-opus-4.8
Link: https://patch.msgid.link/20260715192950.2488921-1-mike@waychison.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
add_disk_final() sets GD_NEED_PART_SCAN before calling bdev_add(),
then calls disk_scan_partitions() which sets the flag itself. The
early set is redundant and introduces a race.
Between bdev_add() and disk_scan_partitions(), concurrent openers
(multipathd, blkid, LVM) see the flag in blkdev_get_whole() and
trigger bdev_disk_changed(). When disk_scan_partitions() then runs,
it calls bdev_disk_changed() again, dropping the partitions the
concurrent opener already created before re-adding them, which can
result in transient partition disappearances.
The race is observable by inserting an msleep() between bdev_add()
and disk_scan_partitions() while running concurrent open() calls
during device bind. Without artificial delay, it manifests under
scheduling pressure during boot on systems with aggressive device
scanners (multipathd, systemd-udevd).
Therefore, do not set GD_NEED_PART_SCAN in add_disk_final(). Other
GD_NEED_PART_SCAN consumers (blkdev_get_whole(),
sd_need_revalidate()) should not be affected as the flag
is set internally by disk_scan_partitions().
The retry-on-next-open intention from commit e5cfefa97b
("block: fix scan partition for exclusively open device again")
should also not be affected as the early return paths in
disk_scan_partitions() should be unreachable at device registration
time (bd_holder is NULL and open_partitions is zero).
Fixes: e5cfefa97b ("block: fix scan partition for exclusively open device again")
Cc: stable@vger.kernel.org
Signed-off-by: Connor Williamson <connordw@amazon.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Link: https://patch.msgid.link/20260615130715.53693-1-connordw@amazon.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
queue/wbt_lat_usec controls both the stored WBT latency target and the
effective WBT enable state.
The old no-op check skipped updates whenever the converted latency
matched the stored min_lat_nsec. That check ignored whether the current
WBT state already matched the state requested by the write. For a queue
disabled by default, attempting to enable WBT by writing the default
value through sysfs could return success while the enable state was left
unchanged.
Treat a write as a no-op only when both the stored latency and the
effective WBT enabled state already match the converted value.
Signed-off-by: Guzebing <guzebing1612@gmail.com>
Link: https://patch.msgid.link/20260621014030.1625306-1-guzebing1612@gmail.com
Signed-off-by: Jens Axboe <axboe@kernel.dk>
bdev_fput() yields the holder claim and then closes the file, which is a
deferred operation. Split the yield half into bdev_yield_claim() so a caller
can give up the holder while the file - and therefore the block device - is
still open, act on the device, and only then bdev_fput().
A filesystem that made a device unfreezable for a membership change with
bdev_deny_freeze() undoes the deny on release with
bdev_yield_claim(bdev_file);
bdev_allow_freeze(file_bdev(bdev_file));
bdev_fput(bdev_file);
Re-allowing only after the holder is yielded avoids stranding the filesystem
on a racing freeze, and doing it while the file is still open avoids touching
the block device after bdev_fput(). bdev_fput() yields again, which is a
no-op once the claim has already been given up.
Link: https://patch.msgid.link/20260616-work-super-freeze_deny_upstream-v2-2-b3567c7f994b@kernel.org
Reviewed-by: Jan Kara <jack@suse.cz>
Reviewd-by: Johannes Thumshirn <johannes.thumshirn@wdc.com>
Signed-off-by: Christian Brauner (Amutable) <brauner@kernel.org>
Mark Brown
Christian Brauner
Joanne KoongandChristian Brauner
Chao ShiandJens Axboe
Eric Biggers
Christoph HellwigandJens Axboe
Joseph QiandJens Axboe
Jackie LiuandJens Axboe
Shin'ichiro KawasakiandJens Axboe
Qu WenruoandJens Axboe
Michael BommaritoandJens Axboe
Tao CuiandJens Axboe
Mike WaychisonandJens Axboe
Connor WilliamsonandJens Axboe
Yun ZhouandChristian Brauner
GuzebingandJens Axboe
Damien Le MoalandJens Axboe