Documentation/driver-api/cxl/theory-of-operation.rst

Source file repositories/reference/linux-study-clean/Documentation/driver-api/cxl/theory-of-operation.rst

File Facts

System
Linux kernel
Corpus path
Documentation/driver-api/cxl/theory-of-operation.rst
Extension
.rst
Size
13048 bytes
Lines
416
Domain
Support Tooling And Documentation
Bucket
Documentation
Inferred role
Support Tooling And Documentation: documentation
Status
atlas-only

Why This File Exists

Repository support layer: documentation, build tooling, samples, user-space helper tools, generated initramfs support, licenses, and validation utilities.

Dependency Surface

Detected Declarations

Annotated Snippet

.. SPDX-License-Identifier: GPL-2.0
.. include:: <isonum.txt>

===============================================
Compute Express Link Driver Theory of Operation
===============================================

A Compute Express Link Memory Device is a CXL component that implements the
CXL.mem protocol. It contains some amount of volatile memory, persistent memory,
or both. It is enumerated as a PCI device for configuration and passing
messages over an MMIO mailbox. Its contribution to the System Physical
Address space is handled via HDM (Host Managed Device Memory) decoders
that optionally define a device's contribution to an interleaved address
range across multiple devices underneath a host-bridge or interleaved
across host-bridges.

The CXL Bus
===========
Similar to how a RAID driver takes disk objects and assembles them into a new
logical device, the CXL subsystem is tasked to take PCIe and ACPI objects and
assemble them into a CXL.mem decode topology. The need for runtime configuration
of the CXL.mem topology is also similar to RAID in that different environments
with the same hardware configuration may decide to assemble the topology in
contrasting ways. One may choose performance (RAID0) striping memory across
multiple Host Bridges and endpoints while another may opt for fault tolerance
and disable any striping in the CXL.mem topology.

Platform firmware enumerates a menu of interleave options at the "CXL root port"
(Linux term for the top of the CXL decode topology). From there, PCIe topology
dictates which endpoints can participate in which Host Bridge decode regimes.
Each PCIe Switch in the path between the root and an endpoint introduces a point
at which the interleave can be split. For example, platform firmware may say a
given range only decodes to one Host Bridge, but that Host Bridge may in turn
interleave cycles across multiple Root Ports. An intervening Switch between a
port and an endpoint may interleave cycles across multiple Downstream Switch
Ports, etc.

Here is a sample listing of a CXL topology defined by 'cxl_test'. The 'cxl_test'
module generates an emulated CXL topology of 2 Host Bridges each with 2 Root
Ports. Each of those Root Ports are connected to 2-way switches with endpoints
connected to those downstream ports for a total of 8 endpoints::

    # cxl list -BEMPu -b cxl_test
    {
      "bus":"root3",
      "provider":"cxl_test",
      "ports:root3":[
        {
          "port":"port5",
          "host":"cxl_host_bridge.1",
          "ports:port5":[
            {
              "port":"port8",
              "host":"cxl_switch_uport.1",
              "endpoints:port8":[
                {
                  "endpoint":"endpoint9",
                  "host":"mem2",
                  "memdev":{
                    "memdev":"mem2",
                    "pmem_size":"256.00 MiB (268.44 MB)",
                    "ram_size":"256.00 MiB (268.44 MB)",
                    "serial":"0x1",
                    "numa_node":1,
                    "host":"cxl_mem.1"
                  }
                },
                {
                  "endpoint":"endpoint15",
                  "host":"mem6",

Annotation

Implementation Notes