Merge tag 'rust-7.2' of gitolite.kernel.org:pub/scm/linux/kernel/git/ojeda/linux

Pull Rust updates from Miguel Ojeda: "This one is big due to the vendoring of the `zerocopy` library, which allows us to replace a bunch of `unsafe` code dealing with conversions between byte sequences and other types with safe alternatives. More details on that below (and in its merge commit). Toolchain and infrastructure: - Introduce support for the 'zerocopy' library [1][2]: Fast, safe, compile error. Pick two. Zerocopy makes zero-cost memory manipulation effortless. We write `unsafe` so you don't have to. It essentially provides derivable traits (e.g. 'FromBytes') and macros (e.g. 'transmute!') for safely converting between byte sequences and other types. Having such support allows us to remove some 'unsafe' code. It is among the most downloaded Rust crates and it is also used by the Rust compiler itself. It is licensed under "BSD-2-Clause OR Apache-2.0 OR MIT". The crates are imported essentially as-is (only +2/-3 lines needed to be adapted), plus SPDX identifiers. Upstream has since added the SPDX identifiers as well as one of the tweaks at my request, thus reducing our future diffs on updates -- I keep the details in one of our usual live lists [3]. In total, it is about ~39k lines added, ~32k without counting 'benches/' which are just for documentation purposes. The series includes a few Kbuild and rust-analyzer improvements and an example patch using it in Nova, removing one 'unsafe impl'. I checked that the codegen of an isolated example function (similar to the Nova patch on top) is essentially identical. It also turns out that (for that particular case) the 'zerocopy' version, even with 'debug-assertions' enabled, has no remaining panics, unlike a few in the current code (since the compiler can prove the remaining 'ub_checks' statically). So their "fast, safe" does indeed check out -- at least in that case. - Support AutoFDO. This allows Rust code to be profiled and optimized based on the profile. Tested with Rust Binder: ~13% slower without AutoFDO in the binderAddInts benchmark (using an app-launch benchmark for the profile). - Support Software Tag-Based KASAN. In addition, fix KASAN Kconfig by requiring Clang. - Add Kconfig options for each existing Rust KUnit test suite, such as 'CONFIG_RUST_BITMAP_KUNIT_TEST'. They are placed within a new menu, 'CONFIG_RUST_KUNIT_TESTS', in the new 'rust/kernel/Kconfig.test' file. - Support the upcoming Rust 1.98.0 release (expected 2026-08-20): lint cleanups and an unstable flag rename. - Disable 'rustdoc' documentation inlining for all prelude items, which bloats the generated documentation. - Ignore (in Git) and clean (in Kbuild) the (rarely) 'rustc'-generated '*.long-type-*.txt' files. 'kernel' crate: - Add new 'bitfield' module with the 'bitfield!' macro (extracted from the existing 'register!' one), which declares integer types that are split into distinct bit fields of arbitrary length. Each field is a 'Bounded' of the appropriate bit width (ensuring values are properly validated and avoiding implicit data loss) and gets several generated getters and setters (infallible, 'const' and fallible) as well as associated constants ('_MASK', '_SHIFT' and '_RANGE'). It also supports fields that can be converted from/to custom types, either fallibly ('?=>') or infallibly ('=>'). For instance: bitfield! { struct Rgb(u16) { 15:11 blue; 10:5 green; 4:0 red; } } // Compile-time checks. let color = Rgb::zeroed().with_const_green::<0x1f>(); assert_eq!(color.green(), 0x1f); assert_eq!(color.into_raw(), 0x1f << Rgb::GREEN_SHIFT); Add as well documentation and a test suite for it, as usual; and update the 'register!' macro to use it. It will be maintained by Alexandre Courbot (with Yury Norov as reviewer) under a new 'MAINTAINERS' entry: 'RUST [BITFIELD]'. - 'ptr' module: rework index projection syntax into keyworded syntax and introduce panicking variant. The keyword syntax ('build:', 'try:', 'panic:') is more explicit and paves the way of perhaps adding more flavors in the future, e.g. an 'unsafe' index projection. For instance, projections now look like this: fn f(p: *const [u8; 32]) -> Result { // Ok, within bounds, checked at build time. project!(p, [build: 1]); // Build error. project!(p, [build: 128]); // `OutOfBound` runtime error (convertible to `ERANGE`). project!(p, [try: 128]); // Runtime panic. project!(p, [panic: 128]); Ok(()) } Update as well the users, which now look like e.g. // Pointer to the first entry of the GSP message queue. let data = project!(self.0.as_ptr(), .gspq.msgq.data[build: 0]); - 'build_assert' module: make the module the home of its macros instead of rendering them twice. - 'sync' module: add 'UniqueArc::as_ptr()' associated function. - 'alloc' module: - Fix the 'Vec::reserve()' doctest to properly account for the existing vector length in the capacity assertion. - Fix an incorrect operator in the 'Vec::extend_with()' 'SAFETY' comment; add a doc test demonstrating basic usage and the zero-length case. - Clean imports across several modules to follow the "kernel vertical" import style in order to minimize conflicts. 'pin-init' crate: - User visible changes: - Do not generate 'non_snake_case' warnings for identifiers that are syntactically just users of a field name. This would allow all '#[allow(non_snake_case)]' in nova-core to be removed, which Gary will send to the nova tree next cycle. - Filter non-cfg attributes out properly in derived structs. This improves pin-init compatibility with other derive macros. - Insert projection types' where clause properly. - Other changes: - Bump MSRV to 1.82, plus associated cleanups. - Overhaul how init slots are projected. The new approach is easier to justify with safety comments. - Mark more functions as inline, which should help mitigate the super-long symbol name issue due to lack of inlining. rust-analyzer: - Support '--envs' for passing env vars for crates like 'zerocopy'. 'MAINTAINERS': - Add the following reviewers to the 'RUST' entry: - Daniel Almeida - Tamir Duberstein - Alexandre Courbot - Onur Özkan They have been involved in the Rust for Linux project for about 7 collective years and bring expertise across several domains, which will be very useful to have around in the future. Thanks everyone for stepping up! And some other fixes, cleanups and improvements" Link: https://github.com/google/zerocopy [1] Link: https://docs.rs/zerocopy [2] Link: https://github.com/Rust-for-Linux/linux/issues/1239 [3] * tag 'rust-7.2' of gitolite.kernel.org:pub/scm/linux/kernel/git/ojeda/linux: (86 commits) MAINTAINERS: add Onur Özkan as Rust reviewer MAINTAINERS: add Alexandre Courbot as Rust reviewer MAINTAINERS: add Tamir Duberstein as Rust reviewer MAINTAINERS: add Daniel Almeida as Rust reviewer kbuild: rust: clean `zerocopy-derive` in `mrproper` rust: make `build_assert` module the home of related macros rust: str: clean unused import for Rust >= 1.98 rust: str: use the "kernel vertical" imports style rust: aref: use the "kernel vertical" imports style rust: page: use the "kernel vertical" imports style gpu: nova-core: firmware: parse `FalconUCodeDescV2` via `zerocopy` rust: prelude: add `zerocopy{,_derive}::FromBytes` rust: zerocopy-derive: enable support in kbuild rust: zerocopy-derive: add `README.md` rust: zerocopy-derive: avoid generating non-ASCII identifiers rust: zerocopy-derive: add SPDX License Identifiers rust: zerocopy-derive: import crate rust: zerocopy: enable support in kbuild rust: zerocopy: add `README.md` rust: zerocopy: remove float `Display` support ...
2026-08-13 14:19:53 -07:00 · 2026-06-15 09:25:48 +05:30
parent b8b674748f 48b375e482
commit b079329b86
310 changed files with 40556 additions and 844 deletions
@@ -188,5 +188,8 @@ sphinx_*/
 # Rust analyzer configuration
 /rust-project.json

+# rustc error message long types
+*.long-type-*.txt
+
 # bc language scripts (not LLVM bitcode)
 !kernel/time/timeconst.bc
@@ -140,11 +140,15 @@ are also mapped to KUnit.
 These tests are introduced by the ``kunit_tests`` procedural macro, which takes
 the name of the test suite as an argument.

+Each test suite should be guarded by a Kconfig option in
+``rust/kernel/Kconfig.test``.
+
 For instance, assume we want to test the function ``f`` from the documentation
 tests section. We could write, in the same file where we have our function:

 .. code-block:: rust

+	#[cfg(CONFIG_RUST_MYMOD_KUNIT_TEST)]
 	#[kunit_tests(rust_kernel_mymod)]
 	mod tests {
 	    use super::*;
@@ -173,6 +177,7 @@ the unit type ``()``) or ``Result`` (i.e. any ``Result<T, E>``). For instance:

 .. code-block:: rust

+	#[cfg(CONFIG_RUST_MYMOD_KUNIT_TEST)]
 	#[kunit_tests(rust_kernel_mymod)]
 	mod tests {
 	    use super::*;
@@ -23400,6 +23400,10 @@ R:	Andreas Hindborg <a.hindborg@kernel.org>
 R:	Alice Ryhl <aliceryhl@google.com>
 R:	Trevor Gross <tmgross@umich.edu>
 R:	Danilo Krummrich <dakr@kernel.org>
+R:	Daniel Almeida <daniel.almeida@collabora.com>
+R:	Tamir Duberstein <tamird@kernel.org>
+R:	Alexandre Courbot <acourbot@nvidia.com>
+R:	Onur Özkan <work@onurozkan.dev>
 L:	rust-for-linux@vger.kernel.org
 S:	Supported
 W:	https://rust-for-linux.com
@@ -23429,6 +23433,13 @@ T:	git https://github.com/Rust-for-Linux/linux.git alloc-next
 F:	rust/kernel/alloc.rs
 F:	rust/kernel/alloc/

+RUST [BITFIELD]
+M:	Alexandre Courbot <acourbot@nvidia.com>
+R:	Yury Norov <yury.norov@gmail.com>
+L:	rust-for-linux@vger.kernel.org
+S:	Maintained
+F:	rust/kernel/bitfield.rs
+
 RUST [INTEROP]
 M:	Joel Fernandes <joelagnelf@nvidia.com>
 M:	Alexandre Courbot <acourbot@nvidia.com>
@@ -1709,7 +1709,8 @@ MRPROPER_FILES += include/config include/generated          \
 		  vmlinux-gdb.py \
 		  rpmbuild \
 		  rust/libmacros.so rust/libmacros.dylib \
-		  rust/libpin_init_internal.so rust/libpin_init_internal.dylib
+		  rust/libpin_init_internal.so rust/libpin_init_internal.dylib \
+		  rust/libzerocopy_derive.so rust/libzerocopy_derive.dylib

 # clean - Delete most, but leave enough to build external modules
 #
@@ -1961,6 +1962,8 @@ rustfmt:
 			-path $(srctree)/rust/proc-macro2 \
 			-o -path $(srctree)/rust/quote \
 			-o -path $(srctree)/rust/syn \
+			-o -path $(srctree)/rust/zerocopy \
+			-o -path $(srctree)/rust/zerocopy-derive \
 		\) -prune -o \
 		-type f -a -name '*.rs' -a ! -name '*generated*' -print \
 		| xargs $(RUSTFMT) $(rustfmt_flags)
@@ -2169,6 +2172,7 @@ clean: $(clean-dirs)
 		-o -name '*.c.[012]*.*' \
 		-o -name '*.ll' \
 		-o -name '*.gcno' \
+		-o -name '*.long-type-*.txt' \
 		\) -type f -print \
 		-o -name '.tmp_*' -print \
 		| xargs rm -rf
@@ -170,7 +170,7 @@ macro_rules! bitfield {
    (@check_field_bounds $hi:tt:$lo:tt $field:ident as bool) => {
        #[allow(clippy::eq_op)]
        const _: () = {
-            ::kernel::build_assert!(
+            ::kernel::build_assert::build_assert!(
                $hi == $lo,
                concat!("boolean field `", stringify!($field), "` covers more than one bit")
            );
@@ -181,7 +181,7 @@ macro_rules! bitfield {
    (@check_field_bounds $hi:tt:$lo:tt $field:ident as $type:tt) => {
        #[allow(clippy::eq_op)]
        const _: () = {
-            ::kernel::build_assert!(
+            ::kernel::build_assert::build_assert!(
                $hi >= $lo,
                concat!("field `", stringify!($field), "`'s MSB is smaller than its LSB")
            );
@@ -48,7 +48,7 @@ fn request_firmware(

 /// Structure used to describe some firmwares, notably FWSEC-FRTS.
 #[repr(C)]
-#[derive(Debug, Clone)]
+#[derive(Debug, Clone, FromBytes)]
 pub(crate) struct FalconUCodeDescV2 {
    /// Header defined by 'NV_BIT_FALCON_UCODE_DESC_HEADER_VDESC*' in OpenRM.
    hdr: u32,
@@ -84,9 +84,6 @@ pub(crate) struct FalconUCodeDescV2 {
    pub(crate) alt_dmem_load_size: u32,
 }

-// SAFETY: all bit patterns are valid for this type, and it doesn't use interior mutability.
-unsafe impl FromBytes for FalconUCodeDescV2 {}
-
 /// Structure used to describe some firmwares, notably FWSEC-FRTS.
 #[repr(C)]
 #[derive(Debug, Clone)]
@@ -237,7 +237,7 @@ impl DmaGspMem {
        let start = gsp_mem.dma_handle();
        // Write values one by one to avoid an on-stack instance of `PteArray`.
        for i in 0..GspMem::PTE_ARRAY_SIZE {
-            dma_write!(gsp_mem, .ptes.0[i], PteArray::<0>::entry(start, i)?);
+            dma_write!(gsp_mem, .ptes.0[build: i], PteArray::<0>::entry(start, i)?);
        }

        dma_write!(
@@ -260,7 +260,7 @@ impl DmaGspMem {
        let rx = self.gsp_read_ptr();

        // Pointer to the first entry of the CPU message queue.
-        let data = ptr::project!(mut self.0.as_mut_ptr(), .cpuq.msgq.data[0]);
+        let data = ptr::project!(mut self.0.as_mut_ptr(), .cpuq.msgq.data[build: 0]);

        let (tail_end, wrap_end) = if rx == 0 {
            // The write area is non-wrapping, and stops at the second-to-last entry of the command
@@ -322,7 +322,7 @@ impl DmaGspMem {
        let rx = self.cpu_read_ptr();

        // Pointer to the first entry of the GSP message queue.
-        let data = ptr::project!(self.0.as_ptr(), .gspq.msgq.data[0]);
+        let data = ptr::project!(self.0.as_ptr(), .gspq.msgq.data[build: 0]);

        let (tail_end, wrap_end) = if rx <= tx {
            // Read area is non-wrapping and stops right before `tx`.
@@ -49,7 +49,7 @@ macro_rules! impl_safe_as {
            #[allow(unused)]
            #[inline(always)]
            pub(crate) const fn [<$from _as_ $into>](value: $from) -> $into {
-                kernel::static_assert!(size_of::<$into>() >= size_of::<$from>());
+                ::kernel::build_assert::static_assert!(size_of::<$into>() >= size_of::<$from>());

                value as $into
            }
@@ -16,6 +16,8 @@ use kernel::{
    transmute::FromBytes,
 };

+use zerocopy::FromBytes as _;
+
 use crate::{
    driver::Bar0,
    firmware::{
@@ -1011,8 +1013,8 @@ impl FwSecBiosImage {
        let data = self.base.data.get(falcon_ucode_offset..).ok_or(EINVAL)?;
        match ver {
            2 => {
-                let v2 = FalconUCodeDescV2::from_bytes_copy_prefix(data)
-                    .ok_or(EINVAL)?
+                let v2 = FalconUCodeDescV2::read_from_prefix(data)
+                    .map_err(|_| EINVAL)?
                    .0;
                Ok(FalconUCodeDesc::V2(v2))
            }
@@ -2195,7 +2195,8 @@ config RUST
 	depends on !DEBUG_INFO_BTF || (PAHOLE_HAS_LANG_EXCLUDE && !LTO)
 	depends on !CFI || HAVE_CFI_ICALL_NORMALIZE_INTEGERS_RUSTC
 	select CFI_ICALL_NORMALIZE_INTEGERS if CFI
-	depends on !KASAN_SW_TAGS
+	depends on !KASAN || CC_IS_CLANG
+	depends on !KASAN_SW_TAGS || RUSTC_VERSION >= 109600
 	help
 	  Enables Rust support in the kernel.

@@ -2209,6 +2210,8 @@ config RUST

 	  If unsure, say N.

+source "rust/kernel/Kconfig.test"
+
 config RUSTC_VERSION_TEXT
 	string
 	depends on RUST
@@ -6,6 +6,8 @@ rustdoc_output := $(objtree)/Documentation/output/rust/rustdoc
 obj-$(CONFIG_RUST) += core.o compiler_builtins.o ffi.o
 always-$(CONFIG_RUST) += exports_core_generated.h

+obj-$(CONFIG_RUST) += zerocopy.o
+
 ifdef CONFIG_RUST_INLINE_HELPERS
 always-$(CONFIG_RUST) += helpers/helpers.bc helpers/helpers_module.bc
 else
@@ -47,13 +49,14 @@ endif
 # Avoids running `$(RUSTC)` when it may not be available.
 ifdef CONFIG_RUST

-libmacros_name := $(shell MAKEFLAGS= $(RUSTC) --print file-names --crate-name macros --crate-type proc-macro - </dev/null)
-libmacros_extension := $(patsubst libmacros.%,%,$(libmacros_name))
+procmacro-name = $(shell MAKEFLAGS= $(RUSTC) --print file-names --crate-name $(1) --crate-type proc-macro - </dev/null)
+procmacro-extension := $(patsubst libname.%,%,$(call procmacro-name,name))

-libpin_init_internal_name := $(shell MAKEFLAGS= $(RUSTC) --print file-names --crate-name pin_init_internal --crate-type proc-macro - </dev/null)
-libpin_init_internal_extension := $(patsubst libpin_init_internal.%,%,$(libpin_init_internal_name))
+libzerocopy_derive_name := $(call procmacro-name,zerocopy_derive)
+libmacros_name := $(call procmacro-name,macros)
+libpin_init_internal_name := $(call procmacro-name,pin_init_internal)

-always-$(CONFIG_RUST) += $(libmacros_name) $(libpin_init_internal_name)
+always-$(CONFIG_RUST) += $(libzerocopy_derive_name) $(libmacros_name) $(libpin_init_internal_name)

 # `$(rust_flags)` is passed in case the user added `--sysroot`.
 rustc_sysroot := $(shell MAKEFLAGS= $(RUSTC) $(rust_flags) --print sysroot)
@@ -81,6 +84,12 @@ core-flags := \
    --edition=$(core-edition) \
    $(call cfgs-to-flags,$(core-cfgs))

+zerocopy-flags := \
+    --cap-lints=allow
+
+zerocopy-envs := \
+    CARGO_PKG_VERSION=0.8.50
+
 proc_macro2-cfgs := \
    feature="proc-macro" \
    wrap_proc_macro \
@@ -118,6 +127,12 @@ syn-flags := \
    --extern quote \
    $(call cfgs-to-flags,$(syn-cfgs))

+zerocopy_derive-flags := \
+    --cap-lints=allow \
+    --extern proc_macro2 \
+    --extern quote \
+    --extern syn
+
 pin_init_internal-cfgs := \
    kernel USE_RUSTC_FEATURES

@@ -144,6 +159,7 @@ doctests_modifiers_workaround := $(rustdoc_modifiers_workaround)$(if $(call rust

 quiet_cmd_rustdoc = RUSTDOC $(if $(rustdoc_host),H, ) $<
      cmd_rustdoc = \
+	$(rustc_target_envs) \
 	OBJTREE=$(abspath $(objtree)) \
 	$(RUSTDOC) $(filter-out $(skip_flags) --remap-path-scope=%,$(if $(rustdoc_host),$(rust_common_flags),$(rust_flags))) \
 		$(rustc_target_flags) -L$(objtree)/$(obj) \
@@ -166,7 +182,7 @@ quiet_cmd_rustdoc = RUSTDOC $(if $(rustdoc_host),H, ) $<
 # command-like flags to solve the issue. Meanwhile, we use the non-custom case
 # and then retouch the generated files.
 rustdoc: rustdoc-core rustdoc-macros rustdoc-compiler_builtins \
-    rustdoc-kernel rustdoc-pin_init
+    rustdoc-kernel rustdoc-pin_init rustdoc-zerocopy rustdoc-zerocopy_derive
 	$(Q)grep -Ehro '<a href="srctree/([^"]+)"' $(rustdoc_output) | \
 		cut -d'"' -f2 | cut -d/ -f2- | while read f; do \
 			if [ ! -e "$(srctree)/$$f" ]; then \
@@ -199,6 +215,12 @@ rustdoc-syn: private rustc_target_flags = $(syn-flags)
 rustdoc-syn: $(src)/syn/lib.rs rustdoc-clean rustdoc-quote FORCE
 	+$(call if_changed,rustdoc)

+rustdoc-zerocopy_derive: private rustdoc_host = yes
+rustdoc-zerocopy_derive: private rustc_target_flags = $(zerocopy_derive-flags) \
+    --extern proc_macro --crate-type proc-macro
+rustdoc-zerocopy_derive: $(src)/zerocopy-derive/lib.rs rustdoc-clean rustdoc-syn FORCE
+	+$(call if_changed,rustdoc)
+
 rustdoc-macros: private rustdoc_host = yes
 rustdoc-macros: private rustc_target_flags = --crate-type proc-macro \
    --extern proc_macro --extern proc_macro2 --extern quote --extern syn
@@ -218,6 +240,13 @@ rustdoc-compiler_builtins: private is-kernel-object := y
 rustdoc-compiler_builtins: $(src)/compiler_builtins.rs rustdoc-core FORCE
 	+$(call if_changed,rustdoc)

+rustdoc-zerocopy: private rustc_target_envs := $(zerocopy-envs)
+rustdoc-zerocopy: private is-kernel-object := y
+rustdoc-zerocopy: private rustc_target_flags = $(zerocopy-flags) \
+    --extend-css $(src)/zerocopy/rustdoc/style.css
+rustdoc-zerocopy: $(src)/zerocopy/src/lib.rs rustdoc-clean rustdoc-core FORCE
+	+$(call if_changed,rustdoc)
+
 rustdoc-ffi: private is-kernel-object := y
 rustdoc-ffi: $(src)/ffi.rs rustdoc-core FORCE
 	+$(call if_changed,rustdoc)
@@ -239,7 +268,8 @@ rustdoc-pin_init: $(src)/pin-init/src/lib.rs rustdoc-pin_init_internal \
 rustdoc-kernel: private is-kernel-object := y
 rustdoc-kernel: private rustc_target_flags = --extern ffi --extern pin_init \
    --extern build_error --extern macros \
-    --extern bindings --extern uapi
+    --extern bindings --extern uapi \
+    --extern zerocopy --extern zerocopy_derive
 rustdoc-kernel: $(src)/kernel/lib.rs rustdoc-core rustdoc-ffi rustdoc-macros \
    rustdoc-pin_init rustdoc-compiler_builtins $(obj)/$(libmacros_name) \
    $(obj)/bindings.o FORCE
@@ -250,6 +280,7 @@ rustdoc-clean: FORCE

 quiet_cmd_rustc_test_library = $(RUSTC_OR_CLIPPY_QUIET) TL $<
      cmd_rustc_test_library = \
+	$(rustc_target_envs) \
 	OBJTREE=$(abspath $(objtree)) \
 	$(RUSTC_OR_CLIPPY) $(filter-out $(skip_flags),$(rust_common_flags) $(rustc_target_flags)) \
 		@$(objtree)/include/generated/rustc_cfg \
@@ -258,6 +289,11 @@ quiet_cmd_rustc_test_library = $(RUSTC_OR_CLIPPY_QUIET) TL $<
 		-L$(objtree)/$(obj)/test \
 		--crate-name $(subst rusttest-,,$(subst rusttestlib-,,$@)) $<

+rusttestlib-zerocopy: private rustc_target_envs := $(zerocopy-envs)
+rusttestlib-zerocopy: private rustc_target_flags = $(zerocopy-flags)
+rusttestlib-zerocopy: $(src)/zerocopy/src/lib.rs FORCE
+	+$(call if_changed,rustc_test_library)
+
 rusttestlib-build_error: $(src)/build_error.rs FORCE
 	+$(call if_changed,rustc_test_library)

@@ -277,6 +313,12 @@ rusttestlib-syn: private rustc_target_flags = $(syn-flags)
 rusttestlib-syn: $(src)/syn/lib.rs rusttestlib-quote FORCE
 	+$(call if_changed,rustc_test_library)

+rusttestlib-zerocopy_derive: private rustc_target_flags = $(zerocopy_derive-flags) \
+    --extern proc_macro
+rusttestlib-zerocopy_derive: private rustc_test_library_proc = yes
+rusttestlib-zerocopy_derive: $(src)/zerocopy-derive/lib.rs rusttestlib-syn FORCE
+	+$(call if_changed,rustc_test_library)
+
 rusttestlib-macros: private rustc_target_flags = --extern proc_macro \
    --extern proc_macro2 --extern quote --extern syn
 rusttestlib-macros: private rustc_test_library_proc = yes
@@ -298,10 +340,11 @@ rusttestlib-pin_init: $(src)/pin-init/src/lib.rs rusttestlib-macros \

 rusttestlib-kernel: private rustc_target_flags = --extern ffi \
    --extern build_error --extern macros --extern pin_init \
-    --extern bindings --extern uapi
+    --extern bindings --extern uapi \
+    --extern zerocopy --extern zerocopy_derive
 rusttestlib-kernel: $(src)/kernel/lib.rs rusttestlib-bindings rusttestlib-uapi \
    rusttestlib-build_error rusttestlib-pin_init $(obj)/$(libmacros_name) \
-    $(obj)/bindings.o FORCE
+    $(obj)/bindings.o rusttestlib-zerocopy rusttestlib-zerocopy_derive FORCE
 	+$(call if_changed,rustc_test_library)

 rusttestlib-bindings: private rustc_target_flags = --extern ffi --extern pin_init
@@ -314,6 +357,7 @@ rusttestlib-uapi: $(src)/uapi/lib.rs rusttestlib-ffi rusttestlib-pin_init FORCE

 quiet_cmd_rustdoc_test = RUSTDOC T $<
      cmd_rustdoc_test = \
+	$(rustc_target_envs) \
 	RUST_MODFILE=test.rs \
 	OBJTREE=$(abspath $(objtree)) \
 	$(RUSTDOC) --test $(rust_common_flags) \
@@ -328,11 +372,13 @@ quiet_cmd_rustdoc_test_kernel = RUSTDOC TK $<
      cmd_rustdoc_test_kernel = \
 	rm -rf $(objtree)/$(obj)/test/doctests/kernel; \
 	mkdir -p $(objtree)/$(obj)/test/doctests/kernel; \
+	$(rustc_target_envs) \
 	OBJTREE=$(abspath $(objtree)) \
 	$(RUSTDOC) --test $(filter-out --remap-path-scope=%,$(rust_flags)) \
 		-L$(objtree)/$(obj) --extern ffi --extern pin_init \
 		--extern kernel --extern build_error --extern macros \
 		--extern bindings --extern uapi \
+		--extern zerocopy --extern zerocopy_derive \
 		--no-run --crate-name kernel -Zunstable-options \
 		--sysroot=/dev/null \
 		$(doctests_modifiers_workaround) \
@@ -350,6 +396,7 @@ quiet_cmd_rustdoc_test_kernel = RUSTDOC TK $<
 # so for the moment we skip `-Cpanic=abort`.
 quiet_cmd_rustc_test = $(RUSTC_OR_CLIPPY_QUIET) T  $<
      cmd_rustc_test = \
+	$(rustc_target_envs) \
 	OBJTREE=$(abspath $(objtree)) \
 	$(RUSTC_OR_CLIPPY) --test $(rust_common_flags) \
 		@$(objtree)/include/generated/rustc_cfg \
@@ -517,8 +564,9 @@ $(obj)/exports_bindings_generated.h: $(obj)/bindings.o FORCE
 $(obj)/exports_kernel_generated.h: $(obj)/kernel.o FORCE
 	$(call if_changed,exports)

-quiet_cmd_rustc_procmacrolibrary = $(RUSTC_OR_CLIPPY_QUIET) PL $@
+quiet_cmd_rustc_procmacrolibrary = $(if $(skip_clippy),RUSTC,$(RUSTC_OR_CLIPPY_QUIET)) PL $@
      cmd_rustc_procmacrolibrary = \
+	$(rustc_target_envs) \
 	$(if $(skip_clippy),$(RUSTC),$(RUSTC_OR_CLIPPY)) \
 		$(filter-out $(skip_flags),$(rust_common_flags) $(rustc_target_flags)) \
 		--emit=dep-info=$(depfile) --emit=link=$@ --crate-type rlib -O \
@@ -541,17 +589,24 @@ $(obj)/libsyn.rlib: private rustc_target_flags = $(syn-flags)
 $(obj)/libsyn.rlib: $(src)/syn/lib.rs $(obj)/libquote.rlib FORCE
 	+$(call if_changed_dep,rustc_procmacrolibrary)

-quiet_cmd_rustc_procmacro = $(RUSTC_OR_CLIPPY_QUIET) P $@
+quiet_cmd_rustc_procmacro = $(if $(skip_clippy),RUSTC,$(RUSTC_OR_CLIPPY_QUIET)) P $@
      cmd_rustc_procmacro = \
-	$(RUSTC_OR_CLIPPY) $(rust_common_flags) $(rustc_target_flags) \
+	$(rustc_target_envs) \
+	$(if $(skip_clippy),$(RUSTC),$(RUSTC_OR_CLIPPY)) $(rust_common_flags) $(rustc_target_flags) \
 		-Clinker-flavor=gcc -Clinker=$(HOSTCC) \
 		-Clink-args='$(call escsq,$(KBUILD_PROCMACROLDFLAGS))' \
 		--emit=dep-info=$(depfile) --emit=link=$@ --extern proc_macro \
 		--crate-type proc-macro -L$(objtree)/$(obj) \
-		--crate-name $(patsubst lib%.$(libmacros_extension),%,$(notdir $@)) \
+		--crate-name $(patsubst lib%.$(procmacro-extension),%,$(notdir $@)) \
 		@$(objtree)/include/generated/rustc_cfg $<

 # Procedural macros can only be used with the `rustc` that compiled it.
+$(obj)/$(libzerocopy_derive_name): private skip_clippy = 1
+$(obj)/$(libzerocopy_derive_name): private rustc_target_flags = $(zerocopy_derive-flags)
+$(obj)/$(libzerocopy_derive_name): $(src)/zerocopy-derive/lib.rs $(obj)/libproc_macro2.rlib \
+    $(obj)/libquote.rlib $(obj)/libsyn.rlib FORCE
+	+$(call if_changed_dep,rustc_procmacro)
+
 $(obj)/$(libmacros_name): private rustc_target_flags = \
    --extern proc_macro2 --extern quote --extern syn
 $(obj)/$(libmacros_name): $(src)/macros/lib.rs $(obj)/libproc_macro2.rlib \
@@ -567,6 +622,7 @@ $(obj)/$(libpin_init_internal_name): $(src)/pin-init/internal/src/lib.rs \
 # since Rust 1.95.0 (https://github.com/rust-lang/rust/pull/151534).
 quiet_cmd_rustc_library = $(if $(skip_clippy),RUSTC,$(RUSTC_OR_CLIPPY_QUIET)) L $@
      cmd_rustc_library = \
+	$(rustc_target_envs) \
 	OBJTREE=$(abspath $(objtree)) \
 	$(if $(skip_clippy),$(RUSTC),$(RUSTC_OR_CLIPPY)) \
 		$(filter-out $(skip_flags),$(rust_flags)) $(rustc_target_flags) \
@@ -591,6 +647,7 @@ rust-analyzer:
 		--cfgs='syn=$(syn-cfgs)' \
 		--cfgs='pin_init_internal=$(pin_init_internal-cfgs)' \
 		--cfgs='pin_init=$(pin_init-cfgs)' \
+		--envs='zerocopy=$(zerocopy-envs)' \
 		$(realpath $(srctree)) $(realpath $(objtree)) \
 		$(rustc_sysroot) $(RUST_LIB_SRC) $(if $(KBUILD_EXTMOD),$(srcroot)) \
 		> rust-project.json
@@ -662,6 +719,13 @@ $(obj)/compiler_builtins.o: private rustc_objcopy = -w -W '__*'
 $(obj)/compiler_builtins.o: $(src)/compiler_builtins.rs $(obj)/core.o FORCE
 	+$(call if_changed_rule,rustc_library)

+$(obj)/zerocopy.o: private skip_clippy = 1
+$(obj)/zerocopy.o: private skip_gendwarfksyms = 1
+$(obj)/zerocopy.o: private rustc_target_envs := $(zerocopy-envs)
+$(obj)/zerocopy.o: private rustc_target_flags = $(zerocopy-flags)
+$(obj)/zerocopy.o: $(src)/zerocopy/src/lib.rs $(obj)/compiler_builtins.o FORCE
+	+$(call if_changed_rule,rustc_library)
+
 $(obj)/pin_init.o: private skip_gendwarfksyms = 1
 $(obj)/pin_init.o: private rustc_target_flags = $(pin_init-flags)
 $(obj)/pin_init.o: $(src)/pin-init/src/lib.rs $(obj)/compiler_builtins.o \
@@ -697,9 +761,11 @@ $(obj)/uapi.o: $(src)/uapi/lib.rs \
 	+$(call if_changed_rule,rustc_library)

 $(obj)/kernel.o: private rustc_target_flags = --extern ffi --extern pin_init \
-    --extern build_error --extern macros --extern bindings --extern uapi
+    --extern build_error --extern macros --extern bindings --extern uapi \
+    --extern zerocopy --extern zerocopy_derive
 $(obj)/kernel.o: $(src)/kernel/lib.rs $(obj)/build_error.o $(obj)/pin_init.o \
-    $(obj)/$(libmacros_name) $(obj)/bindings.o $(obj)/uapi.o FORCE
+    $(obj)/$(libmacros_name) $(obj)/bindings.o $(obj)/uapi.o \
+    $(obj)/zerocopy.o $(obj)/$(libzerocopy_derive_name) FORCE
 	+$(call if_changed_rule,rustc_library)

 ifdef CONFIG_JUMP_LABEL
@@ -0,0 +1,86 @@
+# SPDX-License-Identifier: GPL-2.0-only
+menuconfig RUST_KUNIT_TESTS
+	bool "Rust KUnit tests"
+	depends on KUNIT && RUST
+	default KUNIT_ALL_TESTS
+	help
+	  This menu collects all options for Rust KUnit tests.
+	  See Documentation/rust/testing.rst for how to protect
+	  unit tests with these options.
+
+	  Say Y here to enable Rust KUnit tests.
+
+	  If unsure, say N.
+
+if RUST_KUNIT_TESTS
+config RUST_ALLOCATOR_KUNIT_TEST
+	bool "KUnit tests for Rust allocator API" if !KUNIT_ALL_TESTS
+	default KUNIT_ALL_TESTS
+	help
+	  This option enables KUnit tests for the Rust allocator API.
+	  These are only for development and testing, not for regular
+	  kernel use cases.
+
+	  If unsure, say N.
+
+config RUST_KVEC_KUNIT_TEST
+	bool "KUnit tests for Rust KVec API" if !KUNIT_ALL_TESTS
+	default KUNIT_ALL_TESTS
+	help
+	  This option enables KUnit tests for the Rust KVec API.
+	  These are only for development and testing, not for
+	  regular kernel use cases.
+
+	  If unsure, say N.
+
+config RUST_BITMAP_KUNIT_TEST
+	bool "KUnit tests for Rust bitmap API" if !KUNIT_ALL_TESTS
+	default KUNIT_ALL_TESTS
+	help
+	  This option enables KUnit tests for the Rust bitmap API.
+	  These are only for development and testing, not for regular
+	  kernel use cases.
+
+	  If unsure, say N.
+
+config RUST_KUNIT_SELFTEST
+	bool "KUnit selftests for Rust" if !KUNIT_ALL_TESTS
+	default KUNIT_ALL_TESTS
+	help
+	  This option enables KUnit selftests. These are only
+	  for development and testing, not for regular kernel
+	  use cases.
+
+	  If unsure, say N.
+
+config RUST_STR_KUNIT_TEST
+	bool "KUnit tests for Rust strings API" if !KUNIT_ALL_TESTS
+	default KUNIT_ALL_TESTS
+	help
+	  This option enables KUnit tests for the Rust strings API.
+	  These are only for development and testing, not for regular
+	  kernel use cases.
+
+	  If unsure, say N.
+
+config RUST_ATOMICS_KUNIT_TEST
+	bool "KUnit tests for Rust atomics API" if !KUNIT_ALL_TESTS
+	default KUNIT_ALL_TESTS
+	help
+	  This option enables KUnit tests for the Rust atomics API.
+	  These are only for development and testing, not for regular
+	  kernel use cases.
+
+	  If unsure, say N.
+
+config RUST_BITFIELD_KUNIT_TEST
+	bool "KUnit tests for the Rust `bitfield!` macro" if !KUNIT_ALL_TESTS
+	default KUNIT_ALL_TESTS
+	help
+	  This option enables KUnit tests for the Rust `bitfield!` macro.
+	  These are only for development and testing, not for regular
+	  kernel use cases.
+
+	  If unsure, say N.
+
+endif
@@ -22,8 +22,12 @@ pub use self::kvec::Vec;
 #[derive(Copy, Clone, PartialEq, Eq, Debug)]
 pub struct AllocError;

-use crate::error::{code::EINVAL, Result};
-use core::{alloc::Layout, ptr::NonNull};
+use crate::prelude::*;
+
+use core::{
+    alloc::Layout,
+    ptr::NonNull, //
+};

 /// Flags to be used when allocating memory.
 ///
@@ -8,14 +8,25 @@
 //!
 //! Reference: <https://docs.kernel.org/core-api/memory-allocation.html>

-use super::Flags;
-use core::alloc::Layout;
-use core::ptr;
-use core::ptr::NonNull;
+use super::{
+    AllocError,
+    Allocator,
+    Flags,
+    NumaNode, //
+};

-use crate::alloc::{AllocError, Allocator, NumaNode};
-use crate::bindings;
-use crate::page;
+use crate::{
+    bindings,
+    page, //
+};
+
+use core::{
+    alloc::Layout,
+    ptr::{
+        self,
+        NonNull, //
+    }, //
+};

 const ARCH_KMALLOC_MINALIGN: usize = bindings::ARCH_KMALLOC_MINALIGN;

@@ -163,8 +174,11 @@ impl Vmalloc {
    /// # Examples
    ///
    /// ```
-    /// # use core::ptr::{NonNull, from_mut};
-    /// # use kernel::{page, prelude::*};
+    /// # use core::ptr::{
+    /// #     from_mut,
+    /// #     NonNull, //
+    /// # };
+    /// # use kernel::page;
    /// use kernel::alloc::allocator::Vmalloc;
    ///
    /// let mut vbox = VBox::<[u8; page::PAGE_SIZE]>::new_uninit(GFP_KERNEL)?;
@@ -251,6 +265,7 @@ unsafe impl Allocator for KVmalloc {
    }
 }

+#[cfg(CONFIG_RUST_ALLOCATOR_KUNIT_TEST)]
 #[macros::kunit_tests(rust_allocator)]
 mod tests {
    use super::*;
@@ -1,9 +1,13 @@
 // SPDX-License-Identifier: GPL-2.0

 use super::Vmalloc;
+
 use crate::page;
-use core::marker::PhantomData;
-use core::ptr::NonNull;
+
+use core::{
+    marker::PhantomData,
+    ptr::NonNull, //
+};

 /// An [`Iterator`] of [`page::BorrowedPage`] items owned by a [`Vmalloc`] allocation.
 ///
@@ -3,24 +3,47 @@
 //! Implementation of [`Box`].

 #[allow(unused_imports)] // Used in doc comments.
-use super::allocator::{KVmalloc, Kmalloc, Vmalloc, VmallocPageIter};
-use super::{AllocError, Allocator, Flags, NumaNode};
-use core::alloc::Layout;
-use core::borrow::{Borrow, BorrowMut};
-use core::marker::PhantomData;
-use core::mem::ManuallyDrop;
-use core::mem::MaybeUninit;
-use core::ops::{Deref, DerefMut};
-use core::pin::Pin;
-use core::ptr::NonNull;
-use core::result::Result;
+use super::allocator::{
+    KVmalloc,
+    Kmalloc,
+    Vmalloc,
+    VmallocPageIter, //
+};

-use crate::ffi::c_void;
-use crate::fmt;
-use crate::init::InPlaceInit;
-use crate::page::AsPageIter;
-use crate::types::ForeignOwnable;
-use pin_init::{InPlaceWrite, Init, PinInit, ZeroableOption};
+use super::{
+    AllocError,
+    Allocator,
+    Flags,
+    NumaNode, //
+};
+
+use crate::{
+    fmt,
+    page::AsPageIter,
+    prelude::*,
+    types::ForeignOwnable, //
+};
+
+use core::{
+    alloc::Layout,
+    borrow::{
+        Borrow,
+        BorrowMut, //
+    },
+    marker::PhantomData,
+    mem::{
+        ManuallyDrop,
+        MaybeUninit, //
+    },
+    ops::{
+        Deref,
+        DerefMut, //
+    },
+    ptr::NonNull,
+    result::Result, //
+};
+
+use pin_init::ZeroableOption;

 /// The kernel's [`Box`] type -- a heap allocation for a single value of type `T`.
 ///
@@ -274,7 +297,10 @@ where
    /// # Examples
    ///
    /// ```
-    /// use kernel::sync::{new_spinlock, SpinLock};
+    /// use kernel::sync::{
+    ///     new_spinlock,
+    ///     SpinLock, //
+    /// };
    ///
    /// struct Inner {
    ///     a: u32,
@@ -411,6 +437,7 @@ where
 {
    type Initialized = Box<T, A>;

+    #[inline]
    fn write_init<E>(mut self, init: impl Init<T, E>) -> Result<Self::Initialized, E> {
        let slot = self.as_mut_ptr();
        // SAFETY: When init errors/panics, slot will get deallocated but not dropped,
@@ -420,6 +447,7 @@ where
        Ok(unsafe { Box::assume_init(self) })
    }

+    #[inline]
    fn write_pin_init<E>(mut self, init: impl PinInit<T, E>) -> Result<Pin<Self::Initialized>, E> {
        let slot = self.as_mut_ptr();
        // SAFETY: When init errors/panics, slot will get deallocated but not dropped,
@@ -567,7 +595,6 @@ where
 ///
 /// ```
 /// # use core::borrow::Borrow;
-/// # use kernel::alloc::KBox;
 /// struct Foo<B: Borrow<u32>>(B);
 ///
 /// // Owned instance.
@@ -595,7 +622,6 @@ where
 ///
 /// ```
 /// # use core::borrow::BorrowMut;
-/// # use kernel::alloc::KBox;
 /// struct Foo<B: BorrowMut<u32>>(B);
 ///
 /// // Owned instance.
@@ -660,9 +686,13 @@ where
 /// # Examples
 ///
 /// ```
-/// # use kernel::prelude::*;
-/// use kernel::alloc::allocator::VmallocPageIter;
-/// use kernel::page::{AsPageIter, PAGE_SIZE};
+/// use kernel::{
+///     alloc::allocator::VmallocPageIter,
+///     page::{
+///         AsPageIter,
+///         PAGE_SIZE, //
+///     }, //
+/// };
 ///
 /// let mut vbox = VBox::new((), GFP_KERNEL)?;
 ///
@@ -3,29 +3,52 @@
 //! Implementation of [`Vec`].

 use super::{
-    allocator::{KVmalloc, Kmalloc, Vmalloc, VmallocPageIter},
+    allocator::{
+        KVmalloc,
+        Kmalloc,
+        Vmalloc,
+        VmallocPageIter, //
+    },
    layout::ArrayLayout,
-    AllocError, Allocator, Box, Flags, NumaNode,
+    AllocError,
+    Allocator,
+    Box,
+    Flags,
+    NumaNode, //
 };
+
 use crate::{
    fmt,
    page::{
        AsPageIter,
        PAGE_SIZE, //
-    },
+    }, //
 };
+
 use core::{
-    borrow::{Borrow, BorrowMut},
+    borrow::{
+        Borrow,
+        BorrowMut, //
+    },
    marker::PhantomData,
-    mem::{ManuallyDrop, MaybeUninit},
-    ops::Deref,
-    ops::DerefMut,
-    ops::Index,
-    ops::IndexMut,
-    ptr,
-    ptr::NonNull,
-    slice,
-    slice::SliceIndex,
+    mem::{
+        ManuallyDrop,
+        MaybeUninit, //
+    },
+    ops::{
+        Deref,
+        DerefMut,
+        Index,
+        IndexMut, //
+    },
+    ptr::{
+        self,
+        NonNull, //
+    },
+    slice::{
+        self,
+        SliceIndex, //
+    }, //
 };

 mod errors;
@@ -614,7 +637,7 @@ where
    ///
    /// v.reserve(10, GFP_KERNEL)?;
    /// let cap = v.capacity();
-    /// assert!(cap >= 10);
+    /// assert!(cap >= v.len() + 10);
    ///
    /// v.reserve(10, GFP_KERNEL)?;
    /// let new_cap = v.capacity();
@@ -849,6 +872,24 @@ impl<T> Vec<T, KVmalloc> {

 impl<T: Clone, A: Allocator> Vec<T, A> {
    /// Extend the vector by `n` clones of `value`.
+    ///
+    /// # Examples
+    ///
+    /// ```
+    /// let mut v = KVec::new();
+    /// v.push(1, GFP_KERNEL)?;
+    ///
+    /// v.extend_with(3, 5, GFP_KERNEL)?;
+    /// assert_eq!(&v, &[1, 5, 5, 5]);
+    ///
+    /// v.extend_with(2, 8, GFP_KERNEL)?;
+    /// assert_eq!(&v, &[1, 5, 5, 5, 8, 8]);
+    ///
+    /// v.extend_with(0, 3, GFP_KERNEL)?;
+    /// assert_eq!(&v, &[1, 5, 5, 5, 8, 8]);
+    ///
+    /// # Ok::<(), Error>(())
+    /// ```
    pub fn extend_with(&mut self, n: usize, value: T, flags: Flags) -> Result<(), AllocError> {
        if n == 0 {
            return Ok(());
@@ -866,7 +907,7 @@ impl<T: Clone, A: Allocator> Vec<T, A> {
        spare[n - 1].write(value);

        // SAFETY:
-        // - `self.len() + n < self.capacity()` due to the call to reserve above,
+        // - `self.len() + n <= self.capacity()` due to the call to reserve above,
        // - the loop and the line above initialized the next `n` elements.
        unsafe { self.inc_len(n) };

@@ -1146,9 +1187,13 @@ where
 /// # Examples
 ///
 /// ```
-/// # use kernel::prelude::*;
-/// use kernel::alloc::allocator::VmallocPageIter;
-/// use kernel::page::{AsPageIter, PAGE_SIZE};
+/// use kernel::{
+///     alloc::allocator::VmallocPageIter,
+///     page::{
+///         AsPageIter,
+///         PAGE_SIZE, //
+///     }, //
+/// };
 ///
 /// let mut vec = VVec::<u8>::new();
 ///
@@ -1463,6 +1508,7 @@ impl<'vec, T> Drop for DrainAll<'vec, T> {
    }
 }

+#[cfg(CONFIG_RUST_KVEC_KUNIT_TEST)]
 #[macros::kunit_tests(rust_kvec)]
 mod tests {
    use super::*;
@@ -2,8 +2,10 @@

 //! Errors for the [`Vec`] type.

-use kernel::fmt;
-use kernel::prelude::*;
+use crate::{
+    fmt,
+    prelude::*, //
+};

 /// Error type for [`Vec::push_within_capacity`].
 pub struct PushError<T>(pub T);
@@ -4,7 +4,10 @@
 //!
 //! Custom layout types extending or improving [`Layout`].

-use core::{alloc::Layout, marker::PhantomData};
+use core::{
+    alloc::Layout,
+    marker::PhantomData, //
+};

 /// Error when constructing an [`ArrayLayout`].
 pub struct LayoutError;
@@ -47,7 +50,10 @@ impl<T> ArrayLayout<T> {
    /// # Examples
    ///
    /// ```
-    /// # use kernel::alloc::layout::{ArrayLayout, LayoutError};
+    /// # use kernel::alloc::layout::{
+    /// #     ArrayLayout,
+    /// #     LayoutError, //
+    /// # };
    /// let layout = ArrayLayout::<i32>::new(15)?;
    /// assert_eq!(layout.len(), 15);
    ///
--- a/Show More
+++ b/Show More