Documentation/admin-guide/cgroup-v1/rdma.rst

Source file repositories/reference/linux-study-clean/Documentation/admin-guide/cgroup-v1/rdma.rst

File Facts

System
Linux kernel
Corpus path
Documentation/admin-guide/cgroup-v1/rdma.rst
Extension
.rst
Size
4544 bytes
Lines
118
Domain
Support Tooling And Documentation
Bucket
Documentation
Inferred role
Support Tooling And Documentation: documentation
Status
atlas-only

Why This File Exists

Repository support layer: documentation, build tooling, samples, user-space helper tools, generated initramfs support, licenses, and validation utilities.

Dependency Surface

Detected Declarations

Annotated Snippet

===============
RDMA Controller
===============

.. Contents

   1. Overview
     1-1. What is RDMA controller?
     1-2. Why RDMA controller needed?
     1-3. How is RDMA controller implemented?
   2. Usage Examples

1. Overview
===========

1-1. What is RDMA controller?
-----------------------------

RDMA controller allows user to limit RDMA/IB specific resources that a given
set of processes can use. These processes are grouped using RDMA controller.

RDMA controller defines two resources which can be limited for processes of a
cgroup.

1-2. Why RDMA controller needed?
--------------------------------

Currently user space applications can easily take away all the rdma verb
specific resources such as AH, CQ, QP, MR etc. Due to which other applications
in other cgroup or kernel space ULPs may not even get chance to allocate any
rdma resources. This can lead to service unavailability.

Therefore RDMA controller is needed through which resource consumption
of processes can be limited. Through this controller different rdma
resources can be accounted.

1-3. How is RDMA controller implemented?
----------------------------------------

RDMA cgroup allows limit configuration of resources. Rdma cgroup maintains
resource accounting per cgroup, per device using resource pool structure.
Each such resource pool is limited up to 64 resources in given resource pool
by rdma cgroup, which can be extended later if required.

This resource pool object is linked to the cgroup css. Typically there
are 0 to 4 resource pool instances per cgroup, per device in most use cases.
But nothing limits to have it more. At present hundreds of RDMA devices per
single cgroup may not be handled optimally, however there is no
known use case or requirement for such configuration either.

Since RDMA resources can be allocated from any process and can be freed by any
of the child processes which shares the address space, rdma resources are
always owned by the creator cgroup css. This allows process migration from one
to other cgroup without major complexity of transferring resource ownership;
because such ownership is not really present due to shared nature of
rdma resources. Linking resources around css also ensures that cgroups can be
deleted after processes migrated. This allow progress migration as well with
active resources, even though that is not a primary use case.

Whenever RDMA resource charging occurs, owner rdma cgroup is returned to
the caller. Same rdma cgroup should be passed while uncharging the resource.
This also allows process migrated with active RDMA resource to charge
to new owner cgroup for new resource. It also allows to uncharge resource of
a process from previously charged cgroup which is migrated to new cgroup,
even though that is not a primary use case.

Resource pool object is created in following situations.
(a) User sets the limit and no previous resource pool exist for the device
of interest for the cgroup.
(b) No resource limits were configured, but IB/RDMA stack tries to

Annotation

Implementation Notes