dasharo/linux
0
0
Fork 0
mirror of https://github.com/Dasharo/linux.git synced 2026-03-06 15:25:10 -08:00
Code Issues Packages Projects Releases Wiki Activity

Merge tag 'cxl-for-6.5' of git://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl

Browse Source 
Pull CXL updates from Dan Williams:
 "The highlights in terms of new functionality are support for the
  standard CXL Performance Monitor definition that appeared in CXL 3.0,
  support for device sanitization (wiping all data from a device),
  secure-erase (re-keying encryption of user data), and support for
  firmware update. The firmware update support is notable as it reuses
  the simple sysfs_upload interface to just cat(1) a blob to a sysfs
  file and pipe that to the device.

  Additionally there are a substantial number of cleanups and
  reorganizations to get ready for RCH error handling (RCH == Restricted
  CXL Host == current shipping hardware generation / pre CXL-2.0
  topologies) and type-2 (accelerator / vendor specific) devices.

  For vendor specific devices they implement a subset of what the
  generic type-3 (generic memory expander) driver expects. As a result
  the rework decouples optional infrastructure from the core driver
  context.

  For RCH topologies, where the specification working group did not want
  to confuse pre-CXL-aware operating systems, many of the standard
  registers are hidden which makes support standard bus features like
  AER (PCIe Advanced Error Reporting) difficult. The rework arranges for
  the driver to help the PCI-AER core. Bjorn is on board with this
  direction but a late regression disocvery means the completion of this
  functionality needs to cook a bit longer, so it is code
  reorganizations only for now.

  Summary:

   - Add infrastructure for supporting background commands along with
     support for device sanitization and firmware update

   - Introduce a CXL performance monitoring unit driver based on the
     common definition in the specification.

   - Land some preparatory cleanup and refactoring for the anticipated
     arrival of CXL type-2 (accelerator devices) and CXL RCH (CXL-v1.1
     topology) error handling.

   - Rework CPU cache management with respect to region configuration
     (device hotplug or other dynamic changes to memory interleaving)

   - Fix region reconfiguration vs CXL decoder ordering rules"

* tag 'cxl-for-6.5' of git://git.kernel.org/pub/scm/linux/kernel/git/cxl/cxl: (51 commits)
  cxl: Fix one kernel-doc comment
  cxl/pci: Use correct flag for sanitize polling
  docs: perf: Minimal introduction the the CXL PMU device and driver
  perf: CXL Performance Monitoring Unit driver
  tools/testing/cxl: add firmware update emulation to CXL memdevs
  tools/testing/cxl: Use named effects for the Command Effect Log
  tools/testing/cxl: Fix command effects for inject/clear poison
  cxl: add a firmware update mechanism using the sysfs firmware loader
  cxl/test: Add Secure Erase opcode support
  cxl/mem: Support Secure Erase
  cxl/test: Add Sanitize opcode support
  cxl/mem: Wire up Sanitization support
  cxl/mbox: Add sanitization handling machinery
  cxl/mem: Introduce security state sysfs file
  cxl/mbox: Allow for IRQ_NONE case in the isr
  Revert "cxl/port: Enable the HDM decoder capability for switch ports"
  cxl/memdev: Formalize endpoint port linkage
  cxl/pci: Unconditionally unmask 256B Flit errors
  cxl/region: Manage decoder target_type at decoder-attach time
  cxl/hdm: Default CXL_DEVTYPE_DEVMEM decoders to CXL_DECODER_DEVMEM
  ...
This commit is contained in:
Linus Torvalds
2023-07-01 08:58:41 -07:00
parent 0a1c979c6b fe77cc2e5a
commit d25f002575
37 changed files with 3516 additions and 884 deletions
Download Patch File Download Diff File Expand all files Collapse all files

48
Documentation/ABI/testing/sysfs-bus-cxl
View File

@@ -58,6 +58,54 @@ Description:
affinity for this device.
What: /sys/bus/cxl/devices/memX/security/state
Date: June, 2023
KernelVersion: v6.5
Contact: linux-cxl@vger.kernel.org
Description:
(RO) Reading this file will display the CXL security state for
that device. Such states can be: 'disabled', 'sanitize', when
a sanitization is currently underway; or those available only
for persistent memory: 'locked', 'unlocked' or 'frozen'. This
sysfs entry is select/poll capable from userspace to notify
upon completion of a sanitize operation.
What: /sys/bus/cxl/devices/memX/security/sanitize
Date: June, 2023
KernelVersion: v6.5
Contact: linux-cxl@vger.kernel.org
Description:
(WO) Write a boolean 'true' string value to this attribute to
sanitize the device to securely re-purpose or decommission it.
This is done by ensuring that all user data and meta-data,
whether it resides in persistent capacity, volatile capacity,
or the LSA, is made permanently unavailable by whatever means
is appropriate for the media type. This functionality requires
the device to be not be actively decoding any HPA ranges.
What /sys/bus/cxl/devices/memX/security/erase
Date: June, 2023
KernelVersion: v6.5
Contact: linux-cxl@vger.kernel.org
Description:
(WO) Write a boolean 'true' string value to this attribute to
secure erase user data by changing the media encryption keys for
all user data areas of the device.
What: /sys/bus/cxl/devices/memX/firmware/
Date: April, 2023
KernelVersion: v6.5
Contact: linux-cxl@vger.kernel.org
Description:
(RW) Firmware uploader mechanism. The different files under
this directory can be used to upload and activate new
firmware for CXL devices. The interfaces under this are
documented in sysfs-class-firmware.
What: /sys/bus/cxl/devices/*/devtype
Date: June, 2021
KernelVersion: v5.14

68
Documentation/admin-guide/perf/cxl.rst Normal file
View File

@@ -0,0 +1,68 @@
.. SPDX-License-Identifier: GPL-2.0
======================================
CXL Performance Monitoring Unit (CPMU)
======================================
The CXL rev 3.0 specification provides a definition of CXL Performance
Monitoring Unit in section 13.2: Performance Monitoring.
CXL components (e.g. Root Port, Switch Upstream Port, End Point) may have
any number of CPMU instances. CPMU capabilities are fully discoverable from
the devices. The specification provides event definitions for all CXL protocol
message types and a set of additional events for things commonly counted on
CXL devices (e.g. DRAM events).
CPMU driver
===========
The CPMU driver registers a perf PMU with the name pmu_mem<X>.<Y> on the CXL bus
representing the Yth CPMU for memX.
/sys/bus/cxl/device/pmu_mem<X>.<Y>
The associated PMU is registered as
/sys/bus/event_sources/devices/cxl_pmu_mem<X>.<Y>
In common with other CXL bus devices, the id has no specific meaning and the
relationship to specific CXL device should be established via the device parent
of the device on the CXL bus.
PMU driver provides description of available events and filter options in sysfs.
The "format" directory describes all formats of the config (event vendor id,
group id and mask) config1 (threshold, filter enables) and config2 (filter
parameters) fields of the perf_event_attr structure. The "events" directory
describes all documented events show in perf list.
The events shown in perf list are the most fine grained events with a single
bit of the event mask set. More general events may be enable by setting
multiple mask bits in config. For example, all Device to Host Read Requests
may be captured on a single counter by setting the bits for all of
* d2h_req_rdcurr
* d2h_req_rdown
* d2h_req_rdshared
* d2h_req_rdany
* d2h_req_rdownnodata
Example of usage::
$#perf list
cxl_pmu_mem0.0/clock_ticks/ [Kernel PMU event]
cxl_pmu_mem0.0/d2h_req_rdshared/ [Kernel PMU event]
cxl_pmu_mem0.0/h2d_req_snpcur/ [Kernel PMU event]
cxl_pmu_mem0.0/h2d_req_snpdata/ [Kernel PMU event]
cxl_pmu_mem0.0/h2d_req_snpinv/ [Kernel PMU event]
-----------------------------------------------------------
$# perf stat -a -e cxl_pmu_mem0.0/clock_ticks/ -e cxl_pmu_mem0.0/d2h_req_rdshared/
Vendor specific events may also be available and if so can be used via
$# perf stat -a -e cxl_pmu_mem0.0/vid=VID,gid=GID,mask=MASK/
The driver does not support sampling so "perf record" is unsupported.
It only supports system-wide counting so attaching to a task is
unsupported.

1
Documentation/admin-guide/perf/index.rst
View File

@@ -21,3 +21,4 @@ Performance monitor support
alibaba_pmu
nvidia-pmu
meson-ddr-pmu
cxl

7
MAINTAINERS
View File

@@ -5203,6 +5203,13 @@ S: Maintained
F: drivers/cxl/
F: include/uapi/linux/cxl_mem.h
COMPUTE EXPRESS LINK PMU (CPMU)
M: Jonathan Cameron <jonathan.cameron@huawei.com>
L: linux-cxl@vger.kernel.org
S: Maintained
F: Documentation/admin-guide/perf/cxl.rst
F: drivers/perf/cxl_pmu.c
CONEXANT ACCESSRUNNER USB DRIVER
L: accessrunner-general@lists.sourceforge.net
S: Orphan

14
drivers/cxl/Kconfig
View File

@@ -82,6 +82,7 @@ config CXL_PMEM
config CXL_MEM
tristate "CXL: Memory Expansion"
depends on CXL_PCI
select FW_UPLOAD
default CXL_BUS
help
The CXL.mem protocol allows a device to act as a provider of "System
@@ -139,4 +140,17 @@ config CXL_REGION_INVALIDATION_TEST
If unsure, or if this kernel is meant for production environments,
say N.
config CXL_PMU
tristate "CXL Performance Monitoring Unit"
default CXL_BUS
depends on PERF_EVENTS
help
Support performance monitoring as defined in CXL rev 3.0
section 13.2: Performance Monitoring. CXL components may have
one or more CXL Performance Monitoring Units (CPMUs).
Say 'y/m' to enable a driver that will attach to performance
monitoring units and provide standard perf based interfaces.
If unsure say 'm'.
endif

240
drivers/cxl/acpi.c
View File

@@ -258,7 +258,7 @@ static int cxl_parse_cfmws(union acpi_subtable_headers *header, void *arg,
cxld = &cxlrd->cxlsd.cxld;
cxld->flags = cfmws_to_decoder_flags(cfmws->restrictions);
cxld->target_type = CXL_DECODER_EXPANDER;
cxld->target_type = CXL_DECODER_HOSTONLYMEM;
cxld->hpa_range = (struct range) {
.start = res->start,
.end = res->end,
@@ -327,6 +327,120 @@ __mock struct acpi_device *to_cxl_host_bridge(struct device *host,
return NULL;
}
/* Note, @dev is used by mock_acpi_table_parse_cedt() */
struct cxl_chbs_context {
struct device *dev;
unsigned long long uid;
resource_size_t base;
u32 cxl_version;
};
static int cxl_get_chbs_iter(union acpi_subtable_headers *header, void *arg,
const unsigned long end)
{
struct cxl_chbs_context *ctx = arg;
struct acpi_cedt_chbs *chbs;
if (ctx->base != CXL_RESOURCE_NONE)
return 0;
chbs = (struct acpi_cedt_chbs *) header;
if (ctx->uid != chbs->uid)
return 0;
ctx->cxl_version = chbs->cxl_version;
if (!chbs->base)
return 0;
if (chbs->cxl_version == ACPI_CEDT_CHBS_VERSION_CXL11 &&
chbs->length != CXL_RCRB_SIZE)
return 0;
ctx->base = chbs->base;
return 0;
}
static int cxl_get_chbs(struct device *dev, struct acpi_device *hb,
struct cxl_chbs_context *ctx)
{
unsigned long long uid;
int rc;
rc = acpi_evaluate_integer(hb->handle, METHOD_NAME__UID, NULL, &uid);
if (rc != AE_OK) {
dev_err(dev, "unable to retrieve _UID\n");
return -ENOENT;
}
dev_dbg(dev, "UID found: %lld\n", uid);
*ctx = (struct cxl_chbs_context) {
.dev = dev,
.uid = uid,
.base = CXL_RESOURCE_NONE,
.cxl_version = UINT_MAX,
};
acpi_table_parse_cedt(ACPI_CEDT_TYPE_CHBS, cxl_get_chbs_iter, ctx);
return 0;
}
static int add_host_bridge_dport(struct device *match, void *arg)
{
acpi_status rc;
struct device *bridge;
struct cxl_dport *dport;
struct cxl_chbs_context ctx;
struct acpi_pci_root *pci_root;
struct cxl_port *root_port = arg;
struct device *host = root_port->dev.parent;
struct acpi_device *hb = to_cxl_host_bridge(host, match);
if (!hb)
return 0;
rc = cxl_get_chbs(match, hb, &ctx);
if (rc)
return rc;
if (ctx.cxl_version == UINT_MAX) {
dev_warn(match, "No CHBS found for Host Bridge (UID %lld)\n",
ctx.uid);
return 0;
}
if (ctx.base == CXL_RESOURCE_NONE) {
dev_warn(match, "CHBS invalid for Host Bridge (UID %lld)\n",
ctx.uid);
return 0;
}
pci_root = acpi_pci_find_root(hb->handle);
bridge = pci_root->bus->bridge;
/*
* In RCH mode, bind the component regs base to the dport. In
* VH mode it will be bound to the CXL host bridge's port
* object later in add_host_bridge_uport().
*/
if (ctx.cxl_version == ACPI_CEDT_CHBS_VERSION_CXL11) {
dev_dbg(match, "RCRB found for UID %lld: %pa\n", ctx.uid,
&ctx.base);
dport = devm_cxl_add_rch_dport(root_port, bridge, ctx.uid,
ctx.base);
} else {
dport = devm_cxl_add_dport(root_port, bridge, ctx.uid,
CXL_RESOURCE_NONE);
}
if (IS_ERR(dport))
return PTR_ERR(dport);
return 0;
}
/*
* A host bridge is a dport to a CFMWS decode and it is a uport to the
* dport (PCIe Root Ports) in the host bridge.
@@ -340,6 +454,8 @@ static int add_host_bridge_uport(struct device *match, void *arg)
struct cxl_dport *dport;
struct cxl_port *port;
struct device *bridge;
struct cxl_chbs_context ctx;
resource_size_t component_reg_phys;
int rc;
if (!hb)
@@ -358,12 +474,26 @@ static int add_host_bridge_uport(struct device *match, void *arg)
return 0;
}
rc = cxl_get_chbs(match, hb, &ctx);
if (rc)
return rc;
if (ctx.cxl_version == ACPI_CEDT_CHBS_VERSION_CXL11) {
dev_warn(bridge,
"CXL CHBS version mismatch, skip port registration\n");
return 0;
}
component_reg_phys = ctx.base;
if (component_reg_phys != CXL_RESOURCE_NONE)
dev_dbg(match, "CHBCR found for UID %lld: %pa\n",
ctx.uid, &component_reg_phys);
rc = devm_cxl_register_pci_bus(host, bridge, pci_root->bus);
if (rc)
return rc;
port = devm_cxl_add_port(host, bridge, dport->component_reg_phys,
dport);
port = devm_cxl_add_port(host, bridge, component_reg_phys, dport);
if (IS_ERR(port))
return PTR_ERR(port);
@@ -372,110 +502,6 @@ static int add_host_bridge_uport(struct device *match, void *arg)
return 0;
}
struct cxl_chbs_context {
struct device *dev;
unsigned long long uid;
resource_size_t rcrb;
resource_size_t chbcr;
u32 cxl_version;
};
static int cxl_get_chbcr(union acpi_subtable_headers *header, void *arg,
const unsigned long end)
{
struct cxl_chbs_context *ctx = arg;
struct acpi_cedt_chbs *chbs;
if (ctx->chbcr)
return 0;
chbs = (struct acpi_cedt_chbs *) header;
if (ctx->uid != chbs->uid)
return 0;
ctx->cxl_version = chbs->cxl_version;
ctx->rcrb = CXL_RESOURCE_NONE;
ctx->chbcr = CXL_RESOURCE_NONE;
if (!chbs->base)
return 0;
if (chbs->cxl_version != ACPI_CEDT_CHBS_VERSION_CXL11) {
ctx->chbcr = chbs->base;
return 0;
}
if (chbs->length != CXL_RCRB_SIZE)
return 0;
ctx->rcrb = chbs->base;
ctx->chbcr = cxl_rcrb_to_component(ctx->dev, chbs->base,
CXL_RCRB_DOWNSTREAM);
return 0;
}
static int add_host_bridge_dport(struct device *match, void *arg)
{
acpi_status rc;
struct device *bridge;
unsigned long long uid;
struct cxl_dport *dport;
struct cxl_chbs_context ctx;
struct acpi_pci_root *pci_root;
struct cxl_port *root_port = arg;
struct device *host = root_port->dev.parent;
struct acpi_device *hb = to_cxl_host_bridge(host, match);
if (!hb)
return 0;
rc = acpi_evaluate_integer(hb->handle, METHOD_NAME__UID, NULL, &uid);
if (rc != AE_OK) {
dev_err(match, "unable to retrieve _UID\n");
return -ENODEV;
}
dev_dbg(match, "UID found: %lld\n", uid);
ctx = (struct cxl_chbs_context) {
.dev = match,
.uid = uid,
};
acpi_table_parse_cedt(ACPI_CEDT_TYPE_CHBS, cxl_get_chbcr, &ctx);
if (!ctx.chbcr) {
dev_warn(match, "No CHBS found for Host Bridge (UID %lld)\n",
uid);
return 0;
}
if (ctx.rcrb != CXL_RESOURCE_NONE)
dev_dbg(match, "RCRB found for UID %lld: %pa\n", uid, &ctx.rcrb);
if (ctx.chbcr == CXL_RESOURCE_NONE) {
dev_warn(match, "CHBCR invalid for Host Bridge (UID %lld)\n",
uid);
return 0;
}
dev_dbg(match, "CHBCR found: %pa\n", &ctx.chbcr);
pci_root = acpi_pci_find_root(hb->handle);
bridge = pci_root->bus->bridge;
if (ctx.cxl_version == ACPI_CEDT_CHBS_VERSION_CXL11)
dport = devm_cxl_add_rch_dport(root_port, bridge, uid,
ctx.chbcr, ctx.rcrb);
else
dport = devm_cxl_add_dport(root_port, bridge, uid,
ctx.chbcr);
if (IS_ERR(dport))
return PTR_ERR(dport);
return 0;
}
static int add_root_nvdimm_bridge(struct device *match, void *data)
{
struct cxl_decoder *cxld;

1
drivers/cxl/core/Makefile
View File

@@ -12,5 +12,6 @@ cxl_core-y += memdev.o
cxl_core-y += mbox.o
cxl_core-y += pci.o
cxl_core-y += hdm.o
cxl_core-y += pmu.o
cxl_core-$(CONFIG_TRACING) += trace.o
cxl_core-$(CONFIG_CXL_REGION) += region.o

11
drivers/cxl/core/core.h
View File

@@ -6,6 +6,7 @@
extern const struct device_type cxl_nvdimm_bridge_type;
extern const struct device_type cxl_nvdimm_type;
extern const struct device_type cxl_pmu_type;
extern struct attribute_group cxl_base_attribute_group;
@@ -63,6 +64,16 @@ int cxl_dpa_alloc(struct cxl_endpoint_decoder *cxled, unsigned long long size);
int cxl_dpa_free(struct cxl_endpoint_decoder *cxled);
resource_size_t cxl_dpa_size(struct cxl_endpoint_decoder *cxled);
resource_size_t cxl_dpa_resource_start(struct cxl_endpoint_decoder *cxled);
enum cxl_rcrb {
CXL_RCRB_DOWNSTREAM,
CXL_RCRB_UPSTREAM,
};
struct cxl_rcrb_info;
resource_size_t __rcrb_to_component(struct device *dev,
struct cxl_rcrb_info *ri,
enum cxl_rcrb which);
extern struct rw_semaphore cxl_dpa_rwsem;
int cxl_memdev_init(void);

48
drivers/cxl/core/hdm.c
View File

@@ -85,6 +85,7 @@ static int map_hdm_decoder_regs(struct cxl_port *port, void __iomem *crb,
struct cxl_component_regs *regs)
{
struct cxl_register_map map = {
.dev = &port->dev,
.resource = port->component_reg_phys,
.base = crb,
.max_size = CXL_COMPONENT_REG_BLOCK_SIZE,
@@ -97,8 +98,7 @@ static int map_hdm_decoder_regs(struct cxl_port *port, void __iomem *crb,
return -ENODEV;
}
return cxl_map_component_regs(&port->dev, regs, &map,
BIT(CXL_CM_CAP_CAP_ID_HDM));
return cxl_map_component_regs(&map, regs, BIT(CXL_CM_CAP_CAP_ID_HDM));
}
static bool should_emulate_decoders(struct cxl_endpoint_dvsec_info *info)
@@ -570,8 +570,9 @@ static void cxld_set_interleave(struct cxl_decoder *cxld, u32 *ctrl)
static void cxld_set_type(struct cxl_decoder *cxld, u32 *ctrl)
{
u32p_replace_bits(ctrl, !!(cxld->target_type == 3),
CXL_HDM_DECODER0_CTRL_TYPE);
u32p_replace_bits(ctrl,
!!(cxld->target_type == CXL_DECODER_HOSTONLYMEM),
CXL_HDM_DECODER0_CTRL_HOSTONLY);
}
static int cxlsd_set_targets(struct cxl_switch_decoder *cxlsd, u64 *tgt)
@@ -764,7 +765,7 @@ static int cxl_setup_hdm_decoder_from_dvsec(
if (!len)
return -ENOENT;
cxld->target_type = CXL_DECODER_EXPANDER;
cxld->target_type = CXL_DECODER_HOSTONLYMEM;
cxld->commit = NULL;
cxld->reset = NULL;
cxld->hpa_range = info->dvsec_range[which];
@@ -793,8 +794,8 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,
int *target_map, void __iomem *hdm, int which,
u64 *dpa_base, struct cxl_endpoint_dvsec_info *info)
{
struct cxl_endpoint_decoder *cxled = NULL;
u64 size, base, skip, dpa_size, lo, hi;
struct cxl_endpoint_decoder *cxled;
bool committed;
u32 remainder;
int i, rc;
@@ -827,6 +828,8 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,
return -ENXIO;
}
if (info)
cxled = to_cxl_endpoint_decoder(&cxld->dev);
cxld->hpa_range = (struct range) {
.start = base,
.end = base + size - 1,
@@ -837,10 +840,10 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,
cxld->flags |= CXL_DECODER_F_ENABLE;
if (ctrl & CXL_HDM_DECODER0_CTRL_LOCK)
cxld->flags |= CXL_DECODER_F_LOCK;
if (FIELD_GET(CXL_HDM_DECODER0_CTRL_TYPE, ctrl))
cxld->target_type = CXL_DECODER_EXPANDER;
if (FIELD_GET(CXL_HDM_DECODER0_CTRL_HOSTONLY, ctrl))
cxld->target_type = CXL_DECODER_HOSTONLYMEM;
else
cxld->target_type = CXL_DECODER_ACCELERATOR;
cxld->target_type = CXL_DECODER_DEVMEM;
if (cxld->id != port->commit_end + 1) {
dev_warn(&port->dev,
"decoder%d.%d: Committed out of order\n",
@@ -856,12 +859,28 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,
}
port->commit_end = cxld->id;
} else {
/* unless / until type-2 drivers arrive, assume type-3 */
if (FIELD_GET(CXL_HDM_DECODER0_CTRL_TYPE, ctrl) == 0) {
ctrl |= CXL_HDM_DECODER0_CTRL_TYPE;
if (cxled) {
struct cxl_memdev *cxlmd = cxled_to_memdev(cxled);
struct cxl_dev_state *cxlds = cxlmd->cxlds;
/*
* Default by devtype until a device arrives that needs
* more precision.
*/
if (cxlds->type == CXL_DEVTYPE_CLASSMEM)
cxld->target_type = CXL_DECODER_HOSTONLYMEM;
else
cxld->target_type = CXL_DECODER_DEVMEM;
} else {
/* To be overridden by region type at commit time */
cxld->target_type = CXL_DECODER_HOSTONLYMEM;
}
if (!FIELD_GET(CXL_HDM_DECODER0_CTRL_HOSTONLY, ctrl) &&
cxld->target_type == CXL_DECODER_HOSTONLYMEM) {
ctrl |= CXL_HDM_DECODER0_CTRL_HOSTONLY;
writel(ctrl, hdm + CXL_HDM_DECODER0_CTRL_OFFSET(which));
}
cxld->target_type = CXL_DECODER_EXPANDER;
}
rc = eiw_to_ways(FIELD_GET(CXL_HDM_DECODER0_CTRL_IW_MASK, ctrl),
&cxld->interleave_ways);
@@ -880,7 +899,7 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,
port->id, cxld->id, cxld->hpa_range.start, cxld->hpa_range.end,
cxld->interleave_ways, cxld->interleave_granularity);
if (!info) {
if (!cxled) {
lo = readl(hdm + CXL_HDM_DECODER0_TL_LOW(which));
hi = readl(hdm + CXL_HDM_DECODER0_TL_HIGH(which));
target_list.value = (hi << 32) + lo;
@@ -903,7 +922,6 @@ static int init_hdm_decoder(struct cxl_port *port, struct cxl_decoder *cxld,
lo = readl(hdm + CXL_HDM_DECODER0_SKIP_LOW(which));
hi = readl(hdm + CXL_HDM_DECODER0_SKIP_HIGH(which));
skip = (hi << 32) + lo;
cxled = to_cxl_endpoint_decoder(&cxld->dev);
rc = devm_cxl_dpa_reserve(cxled, *dpa_base + skip, dpa_size, skip);
if (rc) {
dev_err(&port->dev,

339
drivers/cxl/core/mbox.c
View File

File diff suppressed because it is too large Load Diff

503
drivers/cxl/core/memdev.c
View File

File diff suppressed because it is too large Load Diff

31
drivers/cxl/core/pci.c
View File

@@ -67,7 +67,7 @@ static int match_add_dports(struct pci_dev *pdev, void *data)
/**
* devm_cxl_port_enumerate_dports - enumerate downstream ports of the upstream port
* @port: cxl_port whose ->uport is the upstream of dports to be enumerated
* @port: cxl_port whose ->uport_dev is the upstream of dports to be enumerated
*
* Returns a positive number of dports enumerated or a negative error
* code.
@@ -308,36 +308,17 @@ static void disable_hdm(void *_cxlhdm)
hdm + CXL_HDM_DECODER_CTRL_OFFSET);
}
int devm_cxl_enable_hdm(struct cxl_port *port, struct cxl_hdm *cxlhdm)
static int devm_cxl_enable_hdm(struct device *host, struct cxl_hdm *cxlhdm)
{
void __iomem *hdm;
void __iomem *hdm = cxlhdm->regs.hdm_decoder;
u32 global_ctrl;
/*
* If the hdm capability was not mapped there is nothing to enable and
* the caller is responsible for what happens next. For example,
* emulate a passthrough decoder.
*/
if (IS_ERR(cxlhdm))
return 0;
hdm = cxlhdm->regs.hdm_decoder;
global_ctrl = readl(hdm + CXL_HDM_DECODER_CTRL_OFFSET);
/*
* If the HDM decoder capability was enabled on entry, skip
* registering disable_hdm() since this decode capability may be
* owned by platform firmware.
*/
if (global_ctrl & CXL_HDM_DECODER_ENABLE)
return 0;
writel(global_ctrl | CXL_HDM_DECODER_ENABLE,
hdm + CXL_HDM_DECODER_CTRL_OFFSET);
return devm_add_action_or_reset(&port->dev, disable_hdm, cxlhdm);
return devm_add_action_or_reset(host, disable_hdm, cxlhdm);
}
EXPORT_SYMBOL_NS_GPL(devm_cxl_enable_hdm, CXL);
int cxl_dvsec_rr_decode(struct device *dev, int d,
struct cxl_endpoint_dvsec_info *info)
@@ -511,7 +492,7 @@ int cxl_hdm_decode_init(struct cxl_dev_state *cxlds, struct cxl_hdm *cxlhdm,
if (info->mem_enabled)
return 0;
rc = devm_cxl_enable_hdm(port, cxlhdm);
rc = devm_cxl_enable_hdm(&port->dev, cxlhdm);
if (rc)
return rc;
@@ -622,7 +603,7 @@ static int cxl_cdat_read_table(struct device *dev,
*/
void read_cdat_data(struct cxl_port *port)
{
struct cxl_memdev *cxlmd = to_cxl_memdev(port->uport);
struct cxl_memdev *cxlmd = to_cxl_memdev(port->uport_dev);
struct device *host = cxlmd->dev.parent;
struct device *dev = &port->dev;
struct pci_doe_mb *cdat_doe;

2
drivers/cxl/core/pmem.c
View File

@@ -64,7 +64,7 @@ static int match_nvdimm_bridge(struct device *dev, void *data)
struct cxl_nvdimm_bridge *cxl_find_nvdimm_bridge(struct cxl_memdev *cxlmd)
{
struct cxl_port *port = find_cxl_root(dev_get_drvdata(&cxlmd->dev));
struct cxl_port *port = find_cxl_root(cxlmd->endpoint);
struct device *dev;
if (!port)

68
drivers/cxl/core/pmu.c Normal file
View File

@@ -0,0 +1,68 @@
// SPDX-License-Identifier: GPL-2.0-only
/* Copyright(c) 2023 Huawei. All rights reserved. */
#include <linux/device.h>
#include <linux/slab.h>
#include <linux/idr.h>
#include <cxlmem.h>
#include <pmu.h>
#include <cxl.h>
#include "core.h"
static void cxl_pmu_release(struct device *dev)
{
struct cxl_pmu *pmu = to_cxl_pmu(dev);
kfree(pmu);
}
const struct device_type cxl_pmu_type = {
.name = "cxl_pmu",
.release = cxl_pmu_release,
};
static void remove_dev(void *dev)
{
device_del(dev);
}
int devm_cxl_pmu_add(struct device *parent, struct cxl_pmu_regs *regs,
int assoc_id, int index, enum cxl_pmu_type type)
{
struct cxl_pmu *pmu;
struct device *dev;
int rc;
pmu = kzalloc(sizeof(*pmu), GFP_KERNEL);
if (!pmu)
return -ENOMEM;
pmu->assoc_id = assoc_id;
pmu->index = index;
pmu->type = type;
pmu->base = regs->pmu;
dev = &pmu->dev;
device_initialize(dev);
device_set_pm_not_required(dev);
dev->parent = parent;
dev->bus = &cxl_bus_type;
dev->type = &cxl_pmu_type;
switch (pmu->type) {
case CXL_PMU_MEMDEV:
rc = dev_set_name(dev, "pmu_mem%d.%d", assoc_id, index);
break;
}
if (rc)
goto err;
rc = device_add(dev);
if (rc)
goto err;
return devm_add_action_or_reset(parent, remove_dev, dev);
err:
put_device(&pmu->dev);
return rc;
}
EXPORT_SYMBOL_NS_GPL(devm_cxl_pmu_add, CXL);

165
drivers/cxl/core/port.c
View File

@@ -56,6 +56,8 @@ static int cxl_device_id(const struct device *dev)
return CXL_DEVICE_MEMORY_EXPANDER;
if (dev->type == CXL_REGION_TYPE())
return CXL_DEVICE_REGION;
if (dev->type == &cxl_pmu_type)
return CXL_DEVICE_PMU;
return 0;
}
@@ -117,9 +119,9 @@ static ssize_t target_type_show(struct device *dev,
struct cxl_decoder *cxld = to_cxl_decoder(dev);
switch (cxld->target_type) {
case CXL_DECODER_ACCELERATOR:
case CXL_DECODER_DEVMEM:
return sysfs_emit(buf, "accelerator\n");
case CXL_DECODER_EXPANDER:
case CXL_DECODER_HOSTONLYMEM:
return sysfs_emit(buf, "expander\n");
}
return -ENXIO;
@@ -561,9 +563,9 @@ static void unregister_port(void *_port)
* unregistered while holding their parent port lock.
*/
if (!parent)
lock_dev = port->uport;
lock_dev = port->uport_dev;
else if (is_cxl_root(parent))
lock_dev = parent->uport;
lock_dev = parent->uport_dev;
else
lock_dev = &parent->dev;
@@ -583,7 +585,8 @@ static int devm_cxl_link_uport(struct device *host, struct cxl_port *port)
{
int rc;
rc = sysfs_create_link(&port->dev.kobj, &port->uport->kobj, "uport");
rc = sysfs_create_link(&port->dev.kobj, &port->uport_dev->kobj,
"uport");
if (rc)
return rc;
return devm_add_action_or_reset(host, cxl_unlink_uport, port);
@@ -605,7 +608,7 @@ static int devm_cxl_link_parent_dport(struct device *host,
if (!parent_dport)
return 0;
rc = sysfs_create_link(&port->dev.kobj, &parent_dport->dport->kobj,
rc = sysfs_create_link(&port->dev.kobj, &parent_dport->dport_dev->kobj,
"parent_dport");
if (rc)
return rc;
@@ -614,7 +617,7 @@ static int devm_cxl_link_parent_dport(struct device *host,
static struct lock_class_key cxl_port_key;
static struct cxl_port *cxl_port_alloc(struct device *uport,
static struct cxl_port *cxl_port_alloc(struct device *uport_dev,
resource_size_t component_reg_phys,
struct cxl_dport *parent_dport)
{
@@ -630,7 +633,7 @@ static struct cxl_port *cxl_port_alloc(struct device *uport,
if (rc < 0)
goto err;
port->id = rc;
port->uport = uport;
port->uport_dev = uport_dev;
/*
* The top-level cxl_port "cxl_root" does not have a cxl_port as
@@ -658,12 +661,13 @@ static struct cxl_port *cxl_port_alloc(struct device *uport,
if (iter->host_bridge)
port->host_bridge = iter->host_bridge;
else if (parent_dport->rch)
port->host_bridge = parent_dport->dport;
port->host_bridge = parent_dport->dport_dev;
else
port->host_bridge = iter->uport;
dev_dbg(uport, "host-bridge: %s\n", dev_name(port->host_bridge));
port->host_bridge = iter->uport_dev;
dev_dbg(uport_dev, "host-bridge: %s\n",
dev_name(port->host_bridge));
} else
dev->parent = uport;
dev->parent = uport_dev;
port->component_reg_phys = component_reg_phys;
ida_init(&port->decoder_ida);
@@ -686,8 +690,38 @@ err:
return ERR_PTR(rc);
}
static int cxl_setup_comp_regs(struct device *dev, struct cxl_register_map *map,
resource_size_t component_reg_phys)
{
if (component_reg_phys == CXL_RESOURCE_NONE)
return 0;
*map = (struct cxl_register_map) {
.dev = dev,
.reg_type = CXL_REGLOC_RBI_COMPONENT,
.resource = component_reg_phys,
.max_size = CXL_COMPONENT_REG_BLOCK_SIZE,
};
return cxl_setup_regs(map);
}
static inline int cxl_port_setup_regs(struct cxl_port *port,
resource_size_t component_reg_phys)
{
return cxl_setup_comp_regs(&port->dev, &port->comp_map,
component_reg_phys);
}
static inline int cxl_dport_setup_regs(struct cxl_dport *dport,
resource_size_t component_reg_phys)
{
return cxl_setup_comp_regs(dport->dport_dev, &dport->comp_map,
component_reg_phys);
}
static struct cxl_port *__devm_cxl_add_port(struct device *host,
struct device *uport,
struct device *uport_dev,
resource_size_t component_reg_phys,
struct cxl_dport *parent_dport)
{
@@ -695,12 +729,12 @@ static struct cxl_port *__devm_cxl_add_port(struct device *host,
struct device *dev;
int rc;
port = cxl_port_alloc(uport, component_reg_phys, parent_dport);
port = cxl_port_alloc(uport_dev, component_reg_phys, parent_dport);
if (IS_ERR(port))
return port;
dev = &port->dev;
if (is_cxl_memdev(uport))
if (is_cxl_memdev(uport_dev))
rc = dev_set_name(dev, "endpoint%d", port->id);
else if (parent_dport)
rc = dev_set_name(dev, "port%d", port->id);
@@ -709,6 +743,10 @@ static struct cxl_port *__devm_cxl_add_port(struct device *host,
if (rc)
goto err;
rc = cxl_port_setup_regs(port, component_reg_phys);
if (rc)
goto err;
rc = device_add(dev);
if (rc)
goto err;
@@ -735,28 +773,29 @@ err:
/**
* devm_cxl_add_port - register a cxl_port in CXL memory decode hierarchy
* @host: host device for devm operations
* @uport: "physical" device implementing this upstream port
* @uport_dev: "physical" device implementing this upstream port
* @component_reg_phys: (optional) for configurable cxl_port instances
* @parent_dport: next hop up in the CXL memory decode hierarchy
*/
struct cxl_port *devm_cxl_add_port(struct device *host, struct device *uport,
struct cxl_port *devm_cxl_add_port(struct device *host,
struct device *uport_dev,
resource_size_t component_reg_phys,
struct cxl_dport *parent_dport)
{
struct cxl_port *port, *parent_port;
port = __devm_cxl_add_port(host, uport, component_reg_phys,
port = __devm_cxl_add_port(host, uport_dev, component_reg_phys,
parent_dport);
parent_port = parent_dport ? parent_dport->port : NULL;
if (IS_ERR(port)) {
dev_dbg(uport, "Failed to add%s%s%s: %ld\n",
dev_dbg(uport_dev, "Failed to add%s%s%s: %ld\n",
parent_port ? " port to " : "",
parent_port ? dev_name(&parent_port->dev) : "",
parent_port ? "" : " root port",
PTR_ERR(port));
} else {
dev_dbg(uport, "%s added%s%s%s\n",
dev_dbg(uport_dev, "%s added%s%s%s\n",
dev_name(&port->dev),
parent_port ? " to " : "",
parent_port ? dev_name(&parent_port->dev) : "",
@@ -773,33 +812,34 @@ struct pci_bus *cxl_port_to_pci_bus(struct cxl_port *port)
if (is_cxl_root(port))
return NULL;
if (dev_is_pci(port->uport)) {
struct pci_dev *pdev = to_pci_dev(port->uport);
if (dev_is_pci(port->uport_dev)) {
struct pci_dev *pdev = to_pci_dev(port->uport_dev);
return pdev->subordinate;
}
return xa_load(&cxl_root_buses, (unsigned long)port->uport);
return xa_load(&cxl_root_buses, (unsigned long)port->uport_dev);
}
EXPORT_SYMBOL_NS_GPL(cxl_port_to_pci_bus, CXL);
static void unregister_pci_bus(void *uport)
static void unregister_pci_bus(void *uport_dev)
{
xa_erase(&cxl_root_buses, (unsigned long)uport);
xa_erase(&cxl_root_buses, (unsigned long)uport_dev);
}
int devm_cxl_register_pci_bus(struct device *host, struct device *uport,
int devm_cxl_register_pci_bus(struct device *host, struct device *uport_dev,
struct pci_bus *bus)
{
int rc;
if (dev_is_pci(uport))
if (dev_is_pci(uport_dev))
return -EINVAL;
rc = xa_insert(&cxl_root_buses, (unsigned long)uport, bus, GFP_KERNEL);
rc = xa_insert(&cxl_root_buses, (unsigned long)uport_dev, bus,
GFP_KERNEL);
if (rc)
return rc;
return devm_add_action_or_reset(host, unregister_pci_bus, uport);
return devm_add_action_or_reset(host, unregister_pci_bus, uport_dev);
}
EXPORT_SYMBOL_NS_GPL(devm_cxl_register_pci_bus, CXL);
@@ -847,22 +887,22 @@ static struct cxl_dport *find_dport(struct cxl_port *port, int id)
return NULL;
}
static int add_dport(struct cxl_port *port, struct cxl_dport *new)
static int add_dport(struct cxl_port *port, struct cxl_dport *dport)
{
struct cxl_dport *dup;
int rc;
device_lock_assert(&port->dev);
dup = find_dport(port, new->port_id);
dup = find_dport(port, dport->port_id);
if (dup) {
dev_err(&port->dev,
"unable to add dport%d-%s non-unique port id (%s)\n",
new->port_id, dev_name(new->dport),
dev_name(dup->dport));
dport->port_id, dev_name(dport->dport_dev),
dev_name(dup->dport_dev));
return -EBUSY;
}
rc = xa_insert(&port->dports, (unsigned long)new->dport, new,
rc = xa_insert(&port->dports, (unsigned long)dport->dport_dev, dport,
GFP_KERNEL);
if (rc)
return rc;
@@ -895,8 +935,8 @@ static void cxl_dport_remove(void *data)
struct cxl_dport *dport = data;
struct cxl_port *port = dport->port;
xa_erase(&port->dports, (unsigned long) dport->dport);
put_device(dport->dport);
xa_erase(&port->dports, (unsigned long) dport->dport_dev);
put_device(dport->dport_dev);
}
static void cxl_dport_unlink(void *data)
@@ -920,7 +960,7 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,
int rc;
if (is_cxl_root(port))
host = port->uport;
host = port->uport_dev;
else
host = &port->dev;
@@ -938,13 +978,29 @@ __devm_cxl_add_dport(struct cxl_port *port, struct device *dport_dev,
if (!dport)
return ERR_PTR(-ENOMEM);
dport->dport = dport_dev;
dport->port_id = port_id;
dport->component_reg_phys = component_reg_phys;
dport->port = port;
if (rcrb != CXL_RESOURCE_NONE)
if (rcrb != CXL_RESOURCE_NONE) {
dport->rcrb.base = rcrb;
component_reg_phys = __rcrb_to_component(dport_dev, &dport->rcrb,
CXL_RCRB_DOWNSTREAM);
if (component_reg_phys == CXL_RESOURCE_NONE) {
dev_warn(dport_dev, "Invalid Component Registers in RCRB");
return ERR_PTR(-ENXIO);
}
dport->rch = true;
dport->rcrb = rcrb;
}
if (component_reg_phys != CXL_RESOURCE_NONE)
dev_dbg(dport_dev, "Component Registers found for dport: %pa\n",
&component_reg_phys);
dport->dport_dev = dport_dev;
dport->port_id = port_id;
dport->port = port;
rc = cxl_dport_setup_regs(dport, component_reg_phys);
if (rc)
return ERR_PTR(rc);
cond_cxl_root_lock(port);
rc = add_dport(port, dport);
@@ -1004,14 +1060,12 @@ EXPORT_SYMBOL_NS_GPL(devm_cxl_add_dport, CXL);
* @port: the cxl_port that references this dport
* @dport_dev: firmware or PCI device representing the dport
* @port_id: identifier for this dport in a decoder's target list
* @component_reg_phys: optional location of CXL component registers
* @rcrb: mandatory location of a Root Complex Register Block
*
* See CXL 3.0 9.11.8 CXL Devices Attached to an RCH
*/
struct cxl_dport *devm_cxl_add_rch_dport(struct cxl_port *port,
struct device *dport_dev, int port_id,
resource_size_t component_reg_phys,
resource_size_t rcrb)
{
struct cxl_dport *dport;
@@ -1022,7 +1076,7 @@ struct cxl_dport *devm_cxl_add_rch_dport(struct cxl_port *port,
}
dport = __devm_cxl_add_dport(port, dport_dev, port_id,
component_reg_phys, rcrb);
CXL_RESOURCE_NONE, rcrb);
if (IS_ERR(dport)) {
dev_dbg(dport_dev, "failed to add RCH dport to %s: %ld\n",
dev_name(&port->dev), PTR_ERR(dport));
@@ -1161,7 +1215,7 @@ static struct device *grandparent(struct device *dev)
static void delete_endpoint(void *data)
{
struct cxl_memdev *cxlmd = data;
struct cxl_port *endpoint = dev_get_drvdata(&cxlmd->dev);
struct cxl_port *endpoint = cxlmd->endpoint;
struct cxl_port *parent_port;
struct device *parent;
@@ -1176,6 +1230,7 @@ static void delete_endpoint(void *data)
devm_release_action(parent, cxl_unlink_uport, endpoint);
devm_release_action(parent, unregister_port, endpoint);
}
cxlmd->endpoint = NULL;
device_unlock(parent);
put_device(parent);
out:
@@ -1187,7 +1242,7 @@ int cxl_endpoint_autoremove(struct cxl_memdev *cxlmd, struct cxl_port *endpoint)
struct device *dev = &cxlmd->dev;
get_device(&endpoint->dev);
dev_set_drvdata(dev, endpoint);
cxlmd->endpoint = endpoint;
cxlmd->depth = endpoint->depth;
return devm_add_action_or_reset(dev, delete_endpoint, cxlmd);
}
@@ -1363,7 +1418,7 @@ out:
rc = PTR_ERR(port);
else {
dev_dbg(&cxlmd->dev, "add to new port %s:%s\n",
dev_name(&port->dev), dev_name(port->uport));
dev_name(&port->dev), dev_name(port->uport_dev));
rc = cxl_add_ep(dport, &cxlmd->dev);
if (rc == -EBUSY) {
/*
@@ -1425,7 +1480,8 @@ retry:
if (port) {
dev_dbg(&cxlmd->dev,
"found already registered port %s:%s\n",
dev_name(&port->dev), dev_name(port->uport));
dev_name(&port->dev),
dev_name(port->uport_dev));
rc = cxl_add_ep(dport, &cxlmd->dev);
/*
@@ -1465,6 +1521,13 @@ retry:
}
EXPORT_SYMBOL_NS_GPL(devm_cxl_enumerate_ports, CXL);
struct cxl_port *cxl_pci_find_port(struct pci_dev *pdev,
struct cxl_dport **dport)
{
return find_cxl_port(pdev->dev.parent, dport);
}
EXPORT_SYMBOL_NS_GPL(cxl_pci_find_port, CXL);
struct cxl_port *cxl_mem_find_port(struct cxl_memdev *cxlmd,
struct cxl_dport **dport)
{
@@ -1550,7 +1613,7 @@ static int cxl_decoder_init(struct cxl_port *port, struct cxl_decoder *cxld)
/* Pre initialize an "empty" decoder */
cxld->interleave_ways = 1;
cxld->interleave_granularity = PAGE_SIZE;
cxld->target_type = CXL_DECODER_EXPANDER;
cxld->target_type = CXL_DECODER_HOSTONLYMEM;
cxld->hpa_range = (struct range) {
.start = 0,
.end = -1,

168
drivers/cxl/core/region.c
View File

@@ -125,10 +125,38 @@ static struct cxl_region_ref *cxl_rr_load(struct cxl_port *port,
return xa_load(&port->regions, (unsigned long)cxlr);
}
static int cxl_region_invalidate_memregion(struct cxl_region *cxlr)
{
if (!cpu_cache_has_invalidate_memregion()) {
if (IS_ENABLED(CONFIG_CXL_REGION_INVALIDATION_TEST)) {
dev_warn_once(
&cxlr->dev,
"Bypassing cpu_cache_invalidate_memregion() for testing!\n");
return 0;
} else {
dev_err(&cxlr->dev,
"Failed to synchronize CPU cache state\n");
return -ENXIO;
}
}
cpu_cache_invalidate_memregion(IORES_DESC_CXL);
return 0;
}
static int cxl_region_decode_reset(struct cxl_region *cxlr, int count)
{
struct cxl_region_params *p = &cxlr->params;
int i;
int i, rc = 0;
/*
* Before region teardown attempt to flush, and if the flush
* fails cancel the region teardown for data consistency
* concerns
*/
rc = cxl_region_invalidate_memregion(cxlr);
if (rc)
return rc;
for (i = count - 1; i >= 0; i--) {
struct cxl_endpoint_decoder *cxled = p->targets[i];
@@ -136,7 +164,6 @@ static int cxl_region_decode_reset(struct cxl_region *cxlr, int count)
struct cxl_port *iter = cxled_to_port(cxled);
struct cxl_dev_state *cxlds = cxlmd->cxlds;
struct cxl_ep *ep;
int rc = 0;
if (cxlds->rcd)
goto endpoint_reset;
@@ -155,14 +182,19 @@ static int cxl_region_decode_reset(struct cxl_region *cxlr, int count)
rc = cxld->reset(cxld);
if (rc)
return rc;
set_bit(CXL_REGION_F_NEEDS_RESET, &cxlr->flags);
}
endpoint_reset:
rc = cxled->cxld.reset(&cxled->cxld);
if (rc)
return rc;
set_bit(CXL_REGION_F_NEEDS_RESET, &cxlr->flags);
}
/* all decoders associated with this region have been torn down */
clear_bit(CXL_REGION_F_NEEDS_RESET, &cxlr->flags);
return 0;
}
@@ -256,9 +288,19 @@ static ssize_t commit_store(struct device *dev, struct device_attribute *attr,
goto out;
}
if (commit)
/*
* Invalidate caches before region setup to drop any speculative
* consumption of this address space
*/
rc = cxl_region_invalidate_memregion(cxlr);
if (rc)
return rc;
if (commit) {
rc = cxl_region_decode_commit(cxlr);
else {
if (rc == 0)
p->state = CXL_CONFIG_COMMIT;
} else {
p->state = CXL_CONFIG_RESET_PENDING;
up_write(&cxl_region_rwsem);
device_release_driver(&cxlr->dev);
@@ -268,18 +310,20 @@ static ssize_t commit_store(struct device *dev, struct device_attribute *attr,
* The lock was dropped, so need to revalidate that the reset is
* still pending.
*/
if (p->state == CXL_CONFIG_RESET_PENDING)
if (p->state == CXL_CONFIG_RESET_PENDING) {
rc = cxl_region_decode_reset(cxlr, p->interleave_ways);
/*
* Revert to committed since there may still be active
* decoders associated with this region, or move forward
* to active to mark the reset successful
*/
if (rc)
p->state = CXL_CONFIG_COMMIT;
else
p->state = CXL_CONFIG_ACTIVE;
}
}
if (rc)
goto out;
if (commit)
p->state = CXL_CONFIG_COMMIT;
else if (p->state == CXL_CONFIG_RESET_PENDING)
p->state = CXL_CONFIG_ACTIVE;
out:
up_write(&cxl_region_rwsem);
@@ -809,6 +853,18 @@ static int cxl_rr_alloc_decoder(struct cxl_port *port, struct cxl_region *cxlr,
return -EBUSY;
}
/*
* Endpoints should already match the region type, but backstop that
* assumption with an assertion. Switch-decoders change mapping-type
* based on what is mapped when they are assigned to a region.
*/
dev_WARN_ONCE(&cxlr->dev,
port == cxled_to_port(cxled) &&
cxld->target_type != cxlr->type,
"%s:%s mismatch decoder type %d -> %d\n",
dev_name(&cxled_to_memdev(cxled)->dev),
dev_name(&cxld->dev), cxld->target_type, cxlr->type);
cxld->target_type = cxlr->type;
cxl_rr->decoder = cxld;
return 0;
}
@@ -906,10 +962,10 @@ static int cxl_port_attach_region(struct cxl_port *port,
dev_dbg(&cxlr->dev,
"%s:%s %s add: %s:%s @ %d next: %s nr_eps: %d nr_targets: %d\n",
dev_name(port->uport), dev_name(&port->dev),
dev_name(port->uport_dev), dev_name(&port->dev),
dev_name(&cxld->dev), dev_name(&cxlmd->dev),
dev_name(&cxled->cxld.dev), pos,
ep ? ep->next ? dev_name(ep->next->uport) :
ep ? ep->next ? dev_name(ep->next->uport_dev) :
dev_name(&cxlmd->dev) :
"none",
cxl_rr->nr_eps, cxl_rr->nr_targets);
@@ -984,7 +1040,7 @@ static int check_last_peer(struct cxl_endpoint_decoder *cxled,
*/
if (pos < distance) {
dev_dbg(&cxlr->dev, "%s:%s: cannot host %s:%s at %d\n",
dev_name(port->uport), dev_name(&port->dev),
dev_name(port->uport_dev), dev_name(&port->dev),
dev_name(&cxlmd->dev), dev_name(&cxled->cxld.dev), pos);
return -ENXIO;
}
@@ -994,7 +1050,7 @@ static int check_last_peer(struct cxl_endpoint_decoder *cxled,
if (ep->dport != ep_peer->dport) {
dev_dbg(&cxlr->dev,
"%s:%s: %s:%s pos %d mismatched peer %s:%s\n",
dev_name(port->uport), dev_name(&port->dev),
dev_name(port->uport_dev), dev_name(&port->dev),
dev_name(&cxlmd->dev), dev_name(&cxled->cxld.dev), pos,
dev_name(&cxlmd_peer->dev),
dev_name(&cxled_peer->cxld.dev));
@@ -1026,7 +1082,7 @@ static int cxl_port_setup_targets(struct cxl_port *port,
*/
if (!is_power_of_2(cxl_rr->nr_targets)) {
dev_dbg(&cxlr->dev, "%s:%s: invalid target count %d\n",
dev_name(port->uport), dev_name(&port->dev),
dev_name(port->uport_dev), dev_name(&port->dev),
cxl_rr->nr_targets);
return -EINVAL;
}
@@ -1076,7 +1132,7 @@ static int cxl_port_setup_targets(struct cxl_port *port,
rc = granularity_to_eig(parent_ig, &peig);
if (rc) {
dev_dbg(&cxlr->dev, "%s:%s: invalid parent granularity: %d\n",
dev_name(parent_port->uport),
dev_name(parent_port->uport_dev),
dev_name(&parent_port->dev), parent_ig);
return rc;
}
@@ -1084,7 +1140,7 @@ static int cxl_port_setup_targets(struct cxl_port *port,
rc = ways_to_eiw(parent_iw, &peiw);
if (rc) {
dev_dbg(&cxlr->dev, "%s:%s: invalid parent interleave: %d\n",
dev_name(parent_port->uport),
dev_name(parent_port->uport_dev),
dev_name(&parent_port->dev), parent_iw);
return rc;
}
@@ -1093,7 +1149,7 @@ static int cxl_port_setup_targets(struct cxl_port *port,
rc = ways_to_eiw(iw, &eiw);
if (rc) {
dev_dbg(&cxlr->dev, "%s:%s: invalid port interleave: %d\n",
dev_name(port->uport), dev_name(&port->dev), iw);
dev_name(port->uport_dev), dev_name(&port->dev), iw);
return rc;
}
@@ -1113,7 +1169,7 @@ static int cxl_port_setup_targets(struct cxl_port *port,
rc = eig_to_granularity(eig, &ig);
if (rc) {
dev_dbg(&cxlr->dev, "%s:%s: invalid interleave: %d\n",
dev_name(port->uport), dev_name(&port->dev),
dev_name(port->uport_dev), dev_name(&port->dev),
256 << eig);
return rc;
}
@@ -1126,11 +1182,11 @@ static int cxl_port_setup_targets(struct cxl_port *port,
((cxld->flags & CXL_DECODER_F_ENABLE) == 0)) {
dev_err(&cxlr->dev,
"%s:%s %s expected iw: %d ig: %d %pr\n",
dev_name(port->uport), dev_name(&port->dev),
dev_name(port->uport_dev), dev_name(&port->dev),
__func__, iw, ig, p->res);
dev_err(&cxlr->dev,
"%s:%s %s got iw: %d ig: %d state: %s %#llx:%#llx\n",
dev_name(port->uport), dev_name(&port->dev),
dev_name(port->uport_dev), dev_name(&port->dev),
__func__, cxld->interleave_ways,
cxld->interleave_granularity,
(cxld->flags & CXL_DECODER_F_ENABLE) ?
@@ -1147,22 +1203,22 @@ static int cxl_port_setup_targets(struct cxl_port *port,
.end = p->res->end,
};
}
dev_dbg(&cxlr->dev, "%s:%s iw: %d ig: %d\n", dev_name(port->uport),
dev_dbg(&cxlr->dev, "%s:%s iw: %d ig: %d\n", dev_name(port->uport_dev),
dev_name(&port->dev), iw, ig);
add_target:
if (cxl_rr->nr_targets_set == cxl_rr->nr_targets) {
dev_dbg(&cxlr->dev,
"%s:%s: targets full trying to add %s:%s at %d\n",
dev_name(port->uport), dev_name(&port->dev),
dev_name(port->uport_dev), dev_name(&port->dev),
dev_name(&cxlmd->dev), dev_name(&cxled->cxld.dev), pos);
return -ENXIO;
}
if (test_bit(CXL_REGION_F_AUTO, &cxlr->flags)) {
if (cxlsd->target[cxl_rr->nr_targets_set] != ep->dport) {
dev_dbg(&cxlr->dev, "%s:%s: %s expected %s at %d\n",
dev_name(port->uport), dev_name(&port->dev),
dev_name(port->uport_dev), dev_name(&port->dev),
dev_name(&cxlsd->cxld.dev),
dev_name(ep->dport->dport),
dev_name(ep->dport->dport_dev),
cxl_rr->nr_targets_set);
return -ENXIO;
}
@@ -1172,8 +1228,8 @@ add_target:
out_target_set:
cxl_rr->nr_targets_set += inc;
dev_dbg(&cxlr->dev, "%s:%s target[%d] = %s for %s:%s @ %d\n",
dev_name(port->uport), dev_name(&port->dev),
cxl_rr->nr_targets_set - 1, dev_name(ep->dport->dport),
dev_name(port->uport_dev), dev_name(&port->dev),
cxl_rr->nr_targets_set - 1, dev_name(ep->dport->dport_dev),
dev_name(&cxlmd->dev), dev_name(&cxled->cxld.dev), pos);
return 0;
@@ -1492,7 +1548,7 @@ static int cmp_decode_pos(const void *a, const void *b)
if (!dev) {
struct range *range = &cxled_a->cxld.hpa_range;
dev_err(port->uport,
dev_err(port->uport_dev,
"failed to find decoder that maps %#llx-%#llx\n",
range->start, range->end);
goto err;
@@ -1507,14 +1563,15 @@ static int cmp_decode_pos(const void *a, const void *b)
put_device(dev);
if (a_pos < 0 || b_pos < 0) {
dev_err(port->uport,
dev_err(port->uport_dev,
"failed to find shared decoder for %s and %s\n",
dev_name(cxlmd_a->dev.parent),
dev_name(cxlmd_b->dev.parent));
goto err;
}
dev_dbg(port->uport, "%s comes %s %s\n", dev_name(cxlmd_a->dev.parent),
dev_dbg(port->uport_dev, "%s comes %s %s\n",
dev_name(cxlmd_a->dev.parent),
a_pos - b_pos < 0 ? "before" : "after",
dev_name(cxlmd_b->dev.parent));
@@ -1674,7 +1731,6 @@ static int cxl_region_attach(struct cxl_region *cxlr,
if (rc)
goto err_decrement;
p->state = CXL_CONFIG_ACTIVE;
set_bit(CXL_REGION_F_INCOHERENT, &cxlr->flags);
}
cxled->cxld.interleave_ways = p->interleave_ways;
@@ -2059,11 +2115,11 @@ static struct cxl_region *devm_cxl_add_region(struct cxl_root_decoder *cxlrd,
if (rc)
goto err;
rc = devm_add_action_or_reset(port->uport, unregister_region, cxlr);
rc = devm_add_action_or_reset(port->uport_dev, unregister_region, cxlr);
if (rc)
return ERR_PTR(rc);
dev_dbg(port->uport, "%s: created %s\n",
dev_dbg(port->uport_dev, "%s: created %s\n",
dev_name(&cxlrd->cxlsd.cxld.dev), dev_name(dev));
return cxlr;
@@ -2103,7 +2159,7 @@ static struct cxl_region *__create_region(struct cxl_root_decoder *cxlrd,
return ERR_PTR(-EBUSY);
}
return devm_cxl_add_region(cxlrd, id, mode, CXL_DECODER_EXPANDER);
return devm_cxl_add_region(cxlrd, id, mode, CXL_DECODER_HOSTONLYMEM);
}
static ssize_t create_pmem_region_store(struct device *dev,
@@ -2191,7 +2247,7 @@ static ssize_t delete_region_store(struct device *dev,
if (IS_ERR(cxlr))
return PTR_ERR(cxlr);
devm_release_action(port->uport, unregister_region, cxlr);
devm_release_action(port->uport_dev, unregister_region, cxlr);
put_device(&cxlr->dev);
return len;
@@ -2356,7 +2412,8 @@ int cxl_get_poison_by_endpoint(struct cxl_port *port)
rc = device_for_each_child(&port->dev, &ctx, poison_by_decoder);
if (rc == 1)
rc = cxl_get_poison_unmapped(to_cxl_memdev(port->uport), &ctx);
rc = cxl_get_poison_unmapped(to_cxl_memdev(port->uport_dev),
&ctx);
up_read(&cxl_region_rwsem);
return rc;
@@ -2732,7 +2789,7 @@ static struct cxl_region *construct_region(struct cxl_root_decoder *cxlrd,
err:
up_write(&cxl_region_rwsem);
devm_release_action(port->uport, unregister_region, cxlr);
devm_release_action(port->uport_dev, unregister_region, cxlr);
return ERR_PTR(rc);
}
@@ -2803,30 +2860,6 @@ out:
}
EXPORT_SYMBOL_NS_GPL(cxl_add_to_region, CXL);
static int cxl_region_invalidate_memregion(struct cxl_region *cxlr)
{
if (!test_bit(CXL_REGION_F_INCOHERENT, &cxlr->flags))
return 0;
if (!cpu_cache_has_invalidate_memregion()) {
if (IS_ENABLED(CONFIG_CXL_REGION_INVALIDATION_TEST)) {
dev_warn_once(
&cxlr->dev,
"Bypassing cpu_cache_invalidate_memregion() for testing!\n");
clear_bit(CXL_REGION_F_INCOHERENT, &cxlr->flags);
return 0;
} else {
dev_err(&cxlr->dev,
"Failed to synchronize CPU cache state\n");
return -ENXIO;
}
}
cpu_cache_invalidate_memregion(IORES_DESC_CXL);
clear_bit(CXL_REGION_F_INCOHERENT, &cxlr->flags);
return 0;
}
static int is_system_ram(struct resource *res, void *arg)
{
struct cxl_region *cxlr = arg;
@@ -2854,7 +2887,12 @@ static int cxl_region_probe(struct device *dev)
goto out;
}
rc = cxl_region_invalidate_memregion(cxlr);
if (test_bit(CXL_REGION_F_NEEDS_RESET, &cxlr->flags)) {
dev_err(&cxlr->dev,
"failed to activate, re-commit region and retry\n");
rc = -ENXIO;
goto out;
}
/*
* From this point on any path that changes the region's state away from

182
drivers/cxl/core/regs.c
View File

@@ -6,6 +6,7 @@
#include <linux/pci.h>
#include <cxlmem.h>
#include <cxlpci.h>
#include <pmu.h>
#include "core.h"
@@ -199,11 +200,13 @@ void __iomem *devm_cxl_iomap_block(struct device *dev, resource_size_t addr,
return ret_val;
}
int cxl_map_component_regs(struct device *dev, struct cxl_component_regs *regs,
struct cxl_register_map *map, unsigned long map_mask)
int cxl_map_component_regs(const struct cxl_register_map *map,
struct cxl_component_regs *regs,
unsigned long map_mask)
{
struct device *dev = map->dev;
struct mapinfo {
struct cxl_reg_map *rmap;
const struct cxl_reg_map *rmap;
void __iomem **addr;
} mapinfo[] = {
{ &map->component_map.hdm_decoder, &regs->hdm_decoder },
@@ -231,13 +234,13 @@ int cxl_map_component_regs(struct device *dev, struct cxl_component_regs *regs,
}
EXPORT_SYMBOL_NS_GPL(cxl_map_component_regs, CXL);
int cxl_map_device_regs(struct device *dev,
struct cxl_device_regs *regs,
struct cxl_register_map *map)
int cxl_map_device_regs(const struct cxl_register_map *map,
struct cxl_device_regs *regs)
{
struct device *dev = map->dev;
resource_size_t phys_addr = map->resource;
struct mapinfo {
struct cxl_reg_map *rmap;
const struct cxl_reg_map *rmap;
void __iomem **addr;
} mapinfo[] = {
{ &map->device_map.status, &regs->status, },
@@ -286,23 +289,30 @@ static bool cxl_decode_regblock(struct pci_dev *pdev, u32 reg_lo, u32 reg_hi,
}
/**
* cxl_find_regblock() - Locate register blocks by type
* cxl_find_regblock_instance() - Locate a register block by type / index
* @pdev: The CXL PCI device to enumerate.
* @type: Register Block Indicator id
* @map: Enumeration output, clobbered on error
* @index: Index into which particular instance of a regblock wanted in the
* order found in register locator DVSEC.
*
* Return: 0 if register block enumerated, negative error code otherwise
*
* A CXL DVSEC may point to one or more register blocks, search for them
* by @type.
* by @type and @index.
*/
int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,
struct cxl_register_map *map)
int cxl_find_regblock_instance(struct pci_dev *pdev, enum cxl_regloc_type type,
struct cxl_register_map *map, int index)
{
u32 regloc_size, regblocks;
int instance = 0;
int regloc, i;
map->resource = CXL_RESOURCE_NONE;
*map = (struct cxl_register_map) {
.dev = &pdev->dev,
.resource = CXL_RESOURCE_NONE,
};
regloc = pci_find_dvsec_capability(pdev, PCI_DVSEC_VENDOR_ID_CXL,
CXL_DVSEC_REG_LOCATOR);
if (!regloc)
@@ -323,20 +333,148 @@ int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,
if (!cxl_decode_regblock(pdev, reg_lo, reg_hi, map))
continue;
if (map->reg_type == type)
return 0;
if (map->reg_type == type) {
if (index == instance)
return 0;
instance++;
}
}
map->resource = CXL_RESOURCE_NONE;
return -ENODEV;
}
EXPORT_SYMBOL_NS_GPL(cxl_find_regblock_instance, CXL);
/**
* cxl_find_regblock() - Locate register blocks by type
* @pdev: The CXL PCI device to enumerate.
* @type: Register Block Indicator id
* @map: Enumeration output, clobbered on error
*
* Return: 0 if register block enumerated, negative error code otherwise
*
* A CXL DVSEC may point to one or more register blocks, search for them
* by @type.
*/
int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,
struct cxl_register_map *map)
{
return cxl_find_regblock_instance(pdev, type, map, 0);
}
EXPORT_SYMBOL_NS_GPL(cxl_find_regblock, CXL);
resource_size_t cxl_rcrb_to_component(struct device *dev,
resource_size_t rcrb,
enum cxl_rcrb which)
/**
* cxl_count_regblock() - Count instances of a given regblock type.
* @pdev: The CXL PCI device to enumerate.
* @type: Register Block Indicator id
*
* Some regblocks may be repeated. Count how many instances.
*
* Return: count of matching regblocks.
*/
int cxl_count_regblock(struct pci_dev *pdev, enum cxl_regloc_type type)
{
struct cxl_register_map map;
int rc, count = 0;
while (1) {
rc = cxl_find_regblock_instance(pdev, type, &map, count);
if (rc)
return count;
count++;
}
}
EXPORT_SYMBOL_NS_GPL(cxl_count_regblock, CXL);
int cxl_map_pmu_regs(struct pci_dev *pdev, struct cxl_pmu_regs *regs,
struct cxl_register_map *map)
{
struct device *dev = &pdev->dev;
resource_size_t phys_addr;
phys_addr = map->resource;
regs->pmu = devm_cxl_iomap_block(dev, phys_addr, CXL_PMU_REGMAP_SIZE);
if (!regs->pmu)
return -ENOMEM;
return 0;
}
EXPORT_SYMBOL_NS_GPL(cxl_map_pmu_regs, CXL);
static int cxl_map_regblock(struct cxl_register_map *map)
{
struct device *dev = map->dev;
map->base = ioremap(map->resource, map->max_size);
if (!map->base) {
dev_err(dev, "failed to map registers\n");
return -ENOMEM;
}
dev_dbg(dev, "Mapped CXL Memory Device resource %pa\n", &map->resource);
return 0;
}
static void cxl_unmap_regblock(struct cxl_register_map *map)
{
iounmap(map->base);
map->base = NULL;
}
static int cxl_probe_regs(struct cxl_register_map *map)
{
struct cxl_component_reg_map *comp_map;
struct cxl_device_reg_map *dev_map;
struct device *dev = map->dev;
void __iomem *base = map->base;
switch (map->reg_type) {
case CXL_REGLOC_RBI_COMPONENT:
comp_map = &map->component_map;
cxl_probe_component_regs(dev, base, comp_map);
dev_dbg(dev, "Set up component registers\n");
break;
case CXL_REGLOC_RBI_MEMDEV:
dev_map = &map->device_map;
cxl_probe_device_regs(dev, base, dev_map);
if (!dev_map->status.valid || !dev_map->mbox.valid ||
!dev_map->memdev.valid) {
dev_err(dev, "registers not found: %s%s%s\n",
!dev_map->status.valid ? "status " : "",
!dev_map->mbox.valid ? "mbox " : "",
!dev_map->memdev.valid ? "memdev " : "");
return -ENXIO;
}
dev_dbg(dev, "Probing device registers...\n");
break;
default:
break;
}
return 0;
}
int cxl_setup_regs(struct cxl_register_map *map)
{
int rc;
rc = cxl_map_regblock(map);
if (rc)
return rc;
rc = cxl_probe_regs(map);
cxl_unmap_regblock(map);
return rc;
}
EXPORT_SYMBOL_NS_GPL(cxl_setup_regs, CXL);
resource_size_t __rcrb_to_component(struct device *dev, struct cxl_rcrb_info *ri,
enum cxl_rcrb which)
{
resource_size_t component_reg_phys;
resource_size_t rcrb = ri->base;
void __iomem *addr;
u32 bar0, bar1;
u16 cmd;
@@ -395,4 +533,12 @@ resource_size_t cxl_rcrb_to_component(struct device *dev,
return component_reg_phys;
}
EXPORT_SYMBOL_NS_GPL(cxl_rcrb_to_component, CXL);
resource_size_t cxl_rcd_component_reg_phys(struct device *dev,
struct cxl_dport *dport)
{
if (!dport->rch)
return CXL_RESOURCE_NONE;
return __rcrb_to_component(dev, &dport->rcrb, CXL_RCRB_UPSTREAM);
}
EXPORT_SYMBOL_NS_GPL(cxl_rcd_component_reg_phys, CXL);

104
drivers/cxl/cxl.h
View File

@@ -56,7 +56,7 @@
#define CXL_HDM_DECODER0_CTRL_COMMIT BIT(9)
#define CXL_HDM_DECODER0_CTRL_COMMITTED BIT(10)
#define CXL_HDM_DECODER0_CTRL_COMMIT_ERROR BIT(11)
#define CXL_HDM_DECODER0_CTRL_TYPE BIT(12)
#define CXL_HDM_DECODER0_CTRL_HOSTONLY BIT(12)
#define CXL_HDM_DECODER0_TL_LOW(i) (0x20 * (i) + 0x24)
#define CXL_HDM_DECODER0_TL_HIGH(i) (0x20 * (i) + 0x28)
#define CXL_HDM_DECODER0_SKIP_LOW(i) CXL_HDM_DECODER0_TL_LOW(i)
@@ -176,14 +176,22 @@ static inline int ways_to_eiw(unsigned int ways, u8 *eiw)
/* CXL 2.0 8.2.8.4 Mailbox Registers */
#define CXLDEV_MBOX_CAPS_OFFSET 0x00
#define CXLDEV_MBOX_CAP_PAYLOAD_SIZE_MASK GENMASK(4, 0)
#define CXLDEV_MBOX_CAP_BG_CMD_IRQ BIT(6)
#define CXLDEV_MBOX_CAP_IRQ_MSGNUM_MASK GENMASK(10, 7)
#define CXLDEV_MBOX_CTRL_OFFSET 0x04
#define CXLDEV_MBOX_CTRL_DOORBELL BIT(0)
#define CXLDEV_MBOX_CTRL_BG_CMD_IRQ BIT(2)
#define CXLDEV_MBOX_CMD_OFFSET 0x08
#define CXLDEV_MBOX_CMD_COMMAND_OPCODE_MASK GENMASK_ULL(15, 0)
#define CXLDEV_MBOX_CMD_PAYLOAD_LENGTH_MASK GENMASK_ULL(36, 16)
#define CXLDEV_MBOX_STATUS_OFFSET 0x10
#define CXLDEV_MBOX_STATUS_BG_CMD BIT(0)
#define CXLDEV_MBOX_STATUS_RET_CODE_MASK GENMASK_ULL(47, 32)
#define CXLDEV_MBOX_BG_CMD_STATUS_OFFSET 0x18
#define CXLDEV_MBOX_BG_CMD_COMMAND_OPCODE_MASK GENMASK_ULL(15, 0)
#define CXLDEV_MBOX_BG_CMD_COMMAND_PCT_MASK GENMASK_ULL(22, 16)
#define CXLDEV_MBOX_BG_CMD_COMMAND_RC_MASK GENMASK_ULL(47, 32)
#define CXLDEV_MBOX_BG_CMD_COMMAND_VENDOR_MASK GENMASK_ULL(63, 48)
#define CXLDEV_MBOX_PAYLOAD_OFFSET 0x20
/*
@@ -209,6 +217,10 @@ struct cxl_regs {
struct_group_tagged(cxl_device_regs, device_regs,
void __iomem *status, *mbox, *memdev;
);
struct_group_tagged(cxl_pmu_regs, pmu_regs,
void __iomem *pmu;
);
};
struct cxl_reg_map {
@@ -229,16 +241,23 @@ struct cxl_device_reg_map {
struct cxl_reg_map memdev;
};
struct cxl_pmu_reg_map {
struct cxl_reg_map pmu;
};
/**
* struct cxl_register_map - DVSEC harvested register block mapping parameters
* @dev: device for devm operations and logging
* @base: virtual base of the register-block-BAR + @block_offset
* @resource: physical resource base of the register block
* @max_size: maximum mapping size to perform register search
* @reg_type: see enum cxl_regloc_type
* @component_map: cxl_reg_map for component registers
* @device_map: cxl_reg_maps for device registers
* @pmu_map: cxl_reg_maps for CXL Performance Monitoring Units
*/
struct cxl_register_map {
struct device *dev;
void __iomem *base;
resource_size_t resource;
resource_size_t max_size;
@@ -246,6 +265,7 @@ struct cxl_register_map {
union {
struct cxl_component_reg_map component_map;
struct cxl_device_reg_map device_map;
struct cxl_pmu_reg_map pmu_map;
};
};
@@ -253,23 +273,24 @@ void cxl_probe_component_regs(struct device *dev, void __iomem *base,
struct cxl_component_reg_map *map);
void cxl_probe_device_regs(struct device *dev, void __iomem *base,
struct cxl_device_reg_map *map);
int cxl_map_component_regs(struct device *dev, struct cxl_component_regs *regs,
struct cxl_register_map *map,
int cxl_map_component_regs(const struct cxl_register_map *map,
struct cxl_component_regs *regs,
unsigned long map_mask);
int cxl_map_device_regs(struct device *dev, struct cxl_device_regs *regs,
struct cxl_register_map *map);
int cxl_map_device_regs(const struct cxl_register_map *map,
struct cxl_device_regs *regs);
int cxl_map_pmu_regs(struct pci_dev *pdev, struct cxl_pmu_regs *regs,
struct cxl_register_map *map);
enum cxl_regloc_type;
int cxl_count_regblock(struct pci_dev *pdev, enum cxl_regloc_type type);
int cxl_find_regblock_instance(struct pci_dev *pdev, enum cxl_regloc_type type,
struct cxl_register_map *map, int index);
int cxl_find_regblock(struct pci_dev *pdev, enum cxl_regloc_type type,
struct cxl_register_map *map);
enum cxl_rcrb {
CXL_RCRB_DOWNSTREAM,
CXL_RCRB_UPSTREAM,
};
resource_size_t cxl_rcrb_to_component(struct device *dev,
resource_size_t rcrb,
enum cxl_rcrb which);
int cxl_setup_regs(struct cxl_register_map *map);
struct cxl_dport;
resource_size_t cxl_rcd_component_reg_phys(struct device *dev,
struct cxl_dport *dport);
#define CXL_RESOURCE_NONE ((resource_size_t) -1)
#define CXL_TARGET_STRLEN 20
@@ -290,8 +311,8 @@ resource_size_t cxl_rcrb_to_component(struct device *dev,
#define CXL_DECODER_F_MASK GENMASK(5, 0)
enum cxl_decoder_type {
CXL_DECODER_ACCELERATOR = 2,
CXL_DECODER_EXPANDER = 3,
CXL_DECODER_DEVMEM = 2,
CXL_DECODER_HOSTONLYMEM = 3,
};
/*
@@ -462,18 +483,20 @@ struct cxl_region_params {
int nr_targets;
};
/*
* Flag whether this region needs to have its HPA span synchronized with
* CPU cache state at region activation time.
*/
#define CXL_REGION_F_INCOHERENT 0
/*
* Indicate whether this region has been assembled by autodetection or
* userspace assembly. Prevent endpoint decoders outside of automatic
* detection from being added to the region.
*/
#define CXL_REGION_F_AUTO 1
#define CXL_REGION_F_AUTO 0
/*
* Require that a committed region successfully complete a teardown once
* any of its associated decoders have been torn down. This maintains
* the commit state for the region since there are committed decoders,
* but blocks cxl_region_probe().
*/
#define CXL_REGION_F_NEEDS_RESET 1
/**
* struct cxl_region - CXL region
@@ -541,7 +564,7 @@ struct cxl_dax_region {
* downstream port devices to construct a CXL memory
* decode hierarchy.
* @dev: this port's device
* @uport: PCI or platform device implementing the upstream port capability
* @uport_dev: PCI or platform device implementing the upstream port capability
* @host_bridge: Shortcut to the platform attach point for this port
* @id: id for port device-name
* @dports: cxl_dport instances referenced by decoders
@@ -549,6 +572,7 @@ struct cxl_dax_region {
* @regions: cxl_region_ref instances, regions mapped by this port
* @parent_dport: dport that points to this port in the parent
* @decoder_ida: allocator for decoder ids
* @comp_map: component register capability mappings
* @nr_dports: number of entries in @dports
* @hdm_end: track last allocated HDM decoder instance for allocation ordering
* @commit_end: cursor to track highest committed decoder for commit ordering
@@ -560,7 +584,7 @@ struct cxl_dax_region {
*/
struct cxl_port {
struct device dev;
struct device *uport;
struct device *uport_dev;
struct device *host_bridge;
int id;
struct xarray dports;
@@ -568,6 +592,7 @@ struct cxl_port {
struct xarray regions;
struct cxl_dport *parent_dport;
struct ida decoder_ida;
struct cxl_register_map comp_map;
int nr_dports;
int hdm_end;
int commit_end;
@@ -587,20 +612,25 @@ cxl_find_dport_by_dev(struct cxl_port *port, const struct device *dport_dev)
return xa_load(&port->dports, (unsigned long)dport_dev);
}
struct cxl_rcrb_info {
resource_size_t base;
u16 aer_cap;
};
/**
* struct cxl_dport - CXL downstream port
* @dport: PCI bridge or firmware device representing the downstream link
* @dport_dev: PCI bridge or firmware device representing the downstream link
* @comp_map: component register capability mappings
* @port_id: unique hardware identifier for dport in decoder target list
* @component_reg_phys: downstream port component registers
* @rcrb: base address for the Root Complex Register Block
* @rcrb: Data about the Root Complex Register Block layout
* @rch: Indicate whether this dport was enumerated in RCH or VH mode
* @port: reference to cxl_port that contains this downstream port
*/
struct cxl_dport {
struct device *dport;
struct device *dport_dev;
struct cxl_register_map comp_map;
int port_id;
resource_size_t component_reg_phys;
resource_size_t rcrb;
struct cxl_rcrb_info rcrb;
bool rch;
struct cxl_port *port;
};
@@ -641,27 +671,30 @@ struct cxl_region_ref {
/*
* The platform firmware device hosting the root is also the top of the
* CXL port topology. All other CXL ports have another CXL port as their
* parent and their ->uport / host device is out-of-line of the port
* parent and their ->uport_dev / host device is out-of-line of the port
* ancestry.
*/
static inline bool is_cxl_root(struct cxl_port *port)
{
return port->uport == port->dev.parent;
return port->uport_dev == port->dev.parent;
}
bool is_cxl_port(const struct device *dev);
struct cxl_port *to_cxl_port(const struct device *dev);
struct pci_bus;
int devm_cxl_register_pci_bus(struct device *host, struct device *uport,
int devm_cxl_register_pci_bus(struct device *host, struct device *uport_dev,
struct pci_bus *bus);
struct pci_bus *cxl_port_to_pci_bus(struct cxl_port *port);
struct cxl_port *devm_cxl_add_port(struct device *host, struct device *uport,
struct cxl_port *devm_cxl_add_port(struct device *host,
struct device *uport_dev,
resource_size_t component_reg_phys,
struct cxl_dport *parent_dport);
struct cxl_port *find_cxl_root(struct cxl_port *port);
int devm_cxl_enumerate_ports(struct cxl_memdev *cxlmd);
void cxl_bus_rescan(void);
void cxl_bus_drain(void);
struct cxl_port *cxl_pci_find_port(struct pci_dev *pdev,
struct cxl_dport **dport);
struct cxl_port *cxl_mem_find_port(struct cxl_memdev *cxlmd,
struct cxl_dport **dport);
bool schedule_cxl_memdev_detach(struct cxl_memdev *cxlmd);
@@ -671,7 +704,6 @@ struct cxl_dport *devm_cxl_add_dport(struct cxl_port *port,
resource_size_t component_reg_phys);
struct cxl_dport *devm_cxl_add_rch_dport(struct cxl_port *port,
struct device *dport_dev, int port_id,
resource_size_t component_reg_phys,
resource_size_t rcrb);
struct cxl_decoder *to_cxl_decoder(struct device *dev);
@@ -710,7 +742,6 @@ struct cxl_endpoint_dvsec_info {
struct cxl_hdm;
struct cxl_hdm *devm_cxl_setup_hdm(struct cxl_port *port,
struct cxl_endpoint_dvsec_info *info);
int devm_cxl_enable_hdm(struct cxl_port *port, struct cxl_hdm *cxlhdm);
int devm_cxl_enumerate_decoders(struct cxl_hdm *cxlhdm,
struct cxl_endpoint_dvsec_info *info);
int devm_cxl_add_passthrough_decoder(struct cxl_port *port);
@@ -750,6 +781,7 @@ void cxl_driver_unregister(struct cxl_driver *cxl_drv);
#define CXL_DEVICE_REGION 6
#define CXL_DEVICE_PMEM_REGION 7
#define CXL_DEVICE_DAX_REGION 8
#define CXL_DEVICE_PMU 9
#define MODULE_ALIAS_CXL(type) MODULE_ALIAS("cxl:t" __stringify(type) "*")
#define CXL_MODALIAS_FMT "cxl:t%d"

229
drivers/cxl/cxlmem.h
View File

@@ -5,6 +5,7 @@
#include <uapi/linux/cxl_mem.h>
#include <linux/cdev.h>
#include <linux/uuid.h>
#include <linux/rcuwait.h>
#include "cxl.h"
/* CXL 2.0 8.2.8.5.1.1 Memory Device Status Register */
@@ -38,6 +39,7 @@
* @detach_work: active memdev lost a port in its ancestry
* @cxl_nvb: coordinate removal of @cxl_nvd if present
* @cxl_nvd: optional bridge to an nvdimm if the device supports pmem
* @endpoint: connection to the CXL port topology for this memory device
* @id: id number of this memdev instance.
* @depth: endpoint port depth
*/
@@ -48,6 +50,7 @@ struct cxl_memdev {
struct work_struct detach_work;
struct cxl_nvdimm_bridge *cxl_nvb;
struct cxl_nvdimm *cxl_nvd;
struct cxl_port *endpoint;
int id;
int depth;
};
@@ -72,16 +75,18 @@ cxled_to_memdev(struct cxl_endpoint_decoder *cxled)
{
struct cxl_port *port = to_cxl_port(cxled->cxld.dev.parent);
return to_cxl_memdev(port->uport);
return to_cxl_memdev(port->uport_dev);
}
bool is_cxl_memdev(const struct device *dev);
static inline bool is_cxl_endpoint(struct cxl_port *port)
{
return is_cxl_memdev(port->uport);
return is_cxl_memdev(port->uport_dev);
}
struct cxl_memdev *devm_cxl_add_memdev(struct cxl_dev_state *cxlds);
struct cxl_memdev_state;
int cxl_memdev_setup_fw_upload(struct cxl_memdev_state *mds);
int devm_cxl_dpa_reserve(struct cxl_endpoint_decoder *cxled,
resource_size_t base, resource_size_t len,
resource_size_t skipped);
@@ -108,6 +113,9 @@ static inline struct cxl_ep *cxl_ep_load(struct cxl_port *port,
* variable sized output commands, it tells the exact number of bytes
* written.
* @min_out: (input) internal command output payload size validation
* @poll_count: (input) Number of timeouts to attempt.
* @poll_interval_ms: (input) Time between mailbox background command polling
* interval timeouts.
* @return_code: (output) Error code returned from hardware.
*
* This is the primary mechanism used to send commands to the hardware.
@@ -123,6 +131,8 @@ struct cxl_mbox_cmd {
size_t size_in;
size_t size_out;
size_t min_out;
int poll_count;
int poll_interval_ms;
u16 return_code;
};
@@ -195,7 +205,7 @@ static inline int cxl_mbox_cmd_rc2errno(struct cxl_mbox_cmd *mbox_cmd)
*/
#define CXL_CAPACITY_MULTIPLIER SZ_256M
/**
/*
* Event Interrupt Policy
*
* CXL rev 3.0 section 8.2.9.2.4; Table 8-52
@@ -215,8 +225,8 @@ struct cxl_event_interrupt_policy {
/**
* struct cxl_event_state - Event log driver state
*
* @event_buf: Buffer to receive event data
* @event_log_lock: Serialize event_buf and log use
* @buf: Buffer to receive event data
* @log_lock: Serialize event_buf and log use
*/
struct cxl_event_state {
struct cxl_get_event_payload *buf;
@@ -254,6 +264,115 @@ struct cxl_poison_state {
struct mutex lock; /* Protect reads of poison list */
};
/*
* Get FW Info
* CXL rev 3.0 section 8.2.9.3.1; Table 8-56
*/
struct cxl_mbox_get_fw_info {
u8 num_slots;
u8 slot_info;
u8 activation_cap;
u8 reserved[13];
char slot_1_revision[16];
char slot_2_revision[16];
char slot_3_revision[16];
char slot_4_revision[16];
} __packed;
#define CXL_FW_INFO_SLOT_INFO_CUR_MASK GENMASK(2, 0)
#define CXL_FW_INFO_SLOT_INFO_NEXT_MASK GENMASK(5, 3)
#define CXL_FW_INFO_SLOT_INFO_NEXT_SHIFT 3
#define CXL_FW_INFO_ACTIVATION_CAP_HAS_LIVE_ACTIVATE BIT(0)
/*
* Transfer FW Input Payload
* CXL rev 3.0 section 8.2.9.3.2; Table 8-57
*/
struct cxl_mbox_transfer_fw {
u8 action;
u8 slot;
u8 reserved[2];
__le32 offset;
u8 reserved2[0x78];
u8 data[];
} __packed;
#define CXL_FW_TRANSFER_ACTION_FULL 0x0
#define CXL_FW_TRANSFER_ACTION_INITIATE 0x1
#define CXL_FW_TRANSFER_ACTION_CONTINUE 0x2
#define CXL_FW_TRANSFER_ACTION_END 0x3
#define CXL_FW_TRANSFER_ACTION_ABORT 0x4
/*
* CXL rev 3.0 section 8.2.9.3.2 mandates 128-byte alignment for FW packages
* and for each part transferred in a Transfer FW command.
*/
#define CXL_FW_TRANSFER_ALIGNMENT 128
/*
* Activate FW Input Payload
* CXL rev 3.0 section 8.2.9.3.3; Table 8-58
*/
struct cxl_mbox_activate_fw {
u8 action;
u8 slot;
} __packed;
#define CXL_FW_ACTIVATE_ONLINE 0x0
#define CXL_FW_ACTIVATE_OFFLINE 0x1
/* FW state bits */
#define CXL_FW_STATE_BITS 32
#define CXL_FW_CANCEL BIT(0)
/**
* struct cxl_fw_state - Firmware upload / activation state
*
* @state: fw_uploader state bitmask
* @oneshot: whether the fw upload fits in a single transfer
* @num_slots: Number of FW slots available
* @cur_slot: Slot number currently active
* @next_slot: Slot number for the new firmware
*/
struct cxl_fw_state {
DECLARE_BITMAP(state, CXL_FW_STATE_BITS);
bool oneshot;
int num_slots;
int cur_slot;
int next_slot;
};
/**
* struct cxl_security_state - Device security state
*
* @state: state of last security operation
* @poll: polling for sanitization is enabled, device has no mbox irq support
* @poll_tmo_secs: polling timeout
* @poll_dwork: polling work item
* @sanitize_node: sanitation sysfs file to notify
*/
struct cxl_security_state {
unsigned long state;
bool poll;
int poll_tmo_secs;
struct delayed_work poll_dwork;
struct kernfs_node *sanitize_node;
};
/*
* enum cxl_devtype - delineate type-2 from a generic type-3 device
* @CXL_DEVTYPE_DEVMEM - Vendor specific CXL Type-2 device implementing HDM-D or
* HDM-DB, no requirement that this device implements a
* mailbox, or other memory-device-standard manageability
* flows.
* @CXL_DEVTYPE_CLASSMEM - Common class definition of a CXL Type-3 device with
* HDM-H and class-mandatory memory device registers
*/
enum cxl_devtype {
CXL_DEVTYPE_DEVMEM,
CXL_DEVTYPE_CLASSMEM,
};
/**
* struct cxl_dev_state - The driver device state
*
@@ -267,6 +386,36 @@ struct cxl_poison_state {
* @cxl_dvsec: Offset to the PCIe device DVSEC
* @rcd: operating in RCD mode (CXL 3.0 9.11.8 CXL Devices Attached to an RCH)
* @media_ready: Indicate whether the device media is usable
* @dpa_res: Overall DPA resource tree for the device
* @pmem_res: Active Persistent memory capacity configuration
* @ram_res: Active Volatile memory capacity configuration
* @component_reg_phys: register base of component registers
* @serial: PCIe Device Serial Number
* @type: Generic Memory Class device or Vendor Specific Memory device
*/
struct cxl_dev_state {
struct device *dev;
struct cxl_memdev *cxlmd;
struct cxl_regs regs;
int cxl_dvsec;
bool rcd;
bool media_ready;
struct resource dpa_res;
struct resource pmem_res;
struct resource ram_res;
resource_size_t component_reg_phys;
u64 serial;
enum cxl_devtype type;
};
/**
* struct cxl_memdev_state - Generic Type-3 Memory Device Class driver data
*
* CXL 8.1.12.1 PCI Header - Class Code Register Memory Device defines
* common memory device functionality like the presence of a mailbox and
* the functionality related to that like Identify Memory Device and Get
* Partition Info
* @cxlds: Core driver state common across Type-2 and Type-3 devices
* @payload_size: Size of space for payload
* (CXL 2.0 8.2.8.4.3 Mailbox Capabilities Register)
* @lsa_size: Size of Label Storage Area
@@ -275,9 +424,6 @@ struct cxl_poison_state {
* @firmware_version: Firmware version for the memory device.
* @enabled_cmds: Hardware commands found enabled in CEL.
* @exclusive_cmds: Commands that are kernel-internal only
* @dpa_res: Overall DPA resource tree for the device
* @pmem_res: Active Persistent memory capacity configuration
* @ram_res: Active Volatile memory capacity configuration
* @total_bytes: sum of all possible capacities
* @volatile_only_bytes: hard volatile capacity
* @persistent_only_bytes: hard persistent capacity
@@ -286,54 +432,48 @@ struct cxl_poison_state {
* @active_persistent_bytes: sum of hard + soft persistent
* @next_volatile_bytes: volatile capacity change pending device reset
* @next_persistent_bytes: persistent capacity change pending device reset
* @component_reg_phys: register base of component registers
* @info: Cached DVSEC information about the device.
* @serial: PCIe Device Serial Number
* @event: event log driver state
* @poison: poison driver state info
* @fw: firmware upload / activation state
* @mbox_send: @dev specific transport for transmitting mailbox commands
*
* See section 8.2.9.5.2 Capacity Configuration and Label Storage for
* See CXL 3.0 8.2.9.8.2 Capacity Configuration and Label Storage for
* details on capacity parameters.
*/
struct cxl_dev_state {
struct device *dev;
struct cxl_memdev *cxlmd;
struct cxl_regs regs;
int cxl_dvsec;
bool rcd;
bool media_ready;
struct cxl_memdev_state {
struct cxl_dev_state cxlds;
size_t payload_size;
size_t lsa_size;
struct mutex mbox_mutex; /* Protects device mailbox and firmware */
char firmware_version[0x10];
DECLARE_BITMAP(enabled_cmds, CXL_MEM_COMMAND_ID_MAX);
DECLARE_BITMAP(exclusive_cmds, CXL_MEM_COMMAND_ID_MAX);
struct resource dpa_res;
struct resource pmem_res;
struct resource ram_res;
u64 total_bytes;
u64 volatile_only_bytes;
u64 persistent_only_bytes;
u64 partition_align_bytes;
u64 active_volatile_bytes;
u64 active_persistent_bytes;
u64 next_volatile_bytes;
u64 next_persistent_bytes;
resource_size_t component_reg_phys;
u64 serial;
struct cxl_event_state event;
struct cxl_poison_state poison;
struct cxl_security_state security;
struct cxl_fw_state fw;
int (*mbox_send)(struct cxl_dev_state *cxlds, struct cxl_mbox_cmd *cmd);
struct rcuwait mbox_wait;
int (*mbox_send)(struct cxl_memdev_state *mds,
struct cxl_mbox_cmd *cmd);
};
static inline struct cxl_memdev_state *
to_cxl_memdev_state(struct cxl_dev_state *cxlds)
{
if (cxlds->type != CXL_DEVTYPE_CLASSMEM)
return NULL;
return container_of(cxlds, struct cxl_memdev_state, cxlds);
}
enum cxl_opcode {
CXL_MBOX_OP_INVALID = 0x0000,
CXL_MBOX_OP_RAW = CXL_MBOX_OP_INVALID,
@@ -342,6 +482,7 @@ enum cxl_opcode {
CXL_MBOX_OP_GET_EVT_INT_POLICY = 0x0102,
CXL_MBOX_OP_SET_EVT_INT_POLICY = 0x0103,
CXL_MBOX_OP_GET_FW_INFO = 0x0200,
CXL_MBOX_OP_TRANSFER_FW = 0x0201,
CXL_MBOX_OP_ACTIVATE_FW = 0x0202,
CXL_MBOX_OP_SET_TIMESTAMP = 0x0301,
CXL_MBOX_OP_GET_SUPPORTED_LOGS = 0x0400,
@@ -362,6 +503,8 @@ enum cxl_opcode {
CXL_MBOX_OP_GET_SCAN_MEDIA_CAPS = 0x4303,
CXL_MBOX_OP_SCAN_MEDIA = 0x4304,
CXL_MBOX_OP_GET_SCAN_MEDIA = 0x4305,
CXL_MBOX_OP_SANITIZE = 0x4400,
CXL_MBOX_OP_SECURE_ERASE = 0x4401,
CXL_MBOX_OP_GET_SECURITY_STATE = 0x4500,
CXL_MBOX_OP_SET_PASSPHRASE = 0x4501,
CXL_MBOX_OP_DISABLE_PASSPHRASE = 0x4502,
@@ -692,18 +835,20 @@ enum {
CXL_PMEM_SEC_PASS_USER,
};
int cxl_internal_send_cmd(struct cxl_dev_state *cxlds,
int cxl_internal_send_cmd(struct cxl_memdev_state *mds,
struct cxl_mbox_cmd *cmd);
int cxl_dev_state_identify(struct cxl_dev_state *cxlds);
int cxl_dev_state_identify(struct cxl_memdev_state *mds);
int cxl_await_media_ready(struct cxl_dev_state *cxlds);
int cxl_enumerate_cmds(struct cxl_dev_state *cxlds);
int cxl_mem_create_range_info(struct cxl_dev_state *cxlds);
struct cxl_dev_state *cxl_dev_state_create(struct device *dev);
void set_exclusive_cxl_commands(struct cxl_dev_state *cxlds, unsigned long *cmds);
void clear_exclusive_cxl_commands(struct cxl_dev_state *cxlds, unsigned long *cmds);
void cxl_mem_get_event_records(struct cxl_dev_state *cxlds, u32 status);
int cxl_set_timestamp(struct cxl_dev_state *cxlds);
int cxl_poison_state_init(struct cxl_dev_state *cxlds);
int cxl_enumerate_cmds(struct cxl_memdev_state *mds);
int cxl_mem_create_range_info(struct cxl_memdev_state *mds);
struct cxl_memdev_state *cxl_memdev_state_create(struct device *dev);
void set_exclusive_cxl_commands(struct cxl_memdev_state *mds,
unsigned long *cmds);
void clear_exclusive_cxl_commands(struct cxl_memdev_state *mds,
unsigned long *cmds);
void cxl_mem_get_event_records(struct cxl_memdev_state *mds, u32 status);
int cxl_set_timestamp(struct cxl_memdev_state *mds);
int cxl_poison_state_init(struct cxl_memdev_state *mds);
int cxl_mem_get_poison(struct cxl_memdev *cxlmd, u64 offset, u64 len,
struct cxl_region *cxlr);
int cxl_trigger_poison_list(struct cxl_memdev *cxlmd);
@@ -722,6 +867,8 @@ static inline void cxl_mem_active_dec(void)
}
#endif
int cxl_mem_sanitize(struct cxl_memdev_state *mds, u16 cmd);
struct cxl_hdm {
struct cxl_component_regs regs;
unsigned int decoder_count;

1
drivers/cxl/cxlpci.h
View File

@@ -67,6 +67,7 @@ enum cxl_regloc_type {
CXL_REGLOC_RBI_COMPONENT,
CXL_REGLOC_RBI_VIRT,
CXL_REGLOC_RBI_MEMDEV,
CXL_REGLOC_RBI_PMU,
CXL_REGLOC_RBI_TYPES
};

Some files were not shown because too many files have changed in this diff Show More