drivers/infiniband/ulp/rtrs/README

Source file repositories/reference/linux-study-clean/drivers/infiniband/ulp/rtrs/README

File Facts

System
Linux kernel
Corpus path
drivers/infiniband/ulp/rtrs/README
Extension
[no extension]
Size
10457 bytes
Lines
214
Domain
Driver Families
Bucket
drivers/infiniband
Inferred role
Driver Families: drivers/infiniband
Status
atlas-only

Why This File Exists

Repeatable hardware-adapter layer. Deep compatibility for every driver is out of scope; this atlas records patterns, probe lifecycles, bus glue, IRQ/DMA usage, and links back to core abstractions.

Dependency Surface

Detected Declarations

Annotated Snippet

RDMA Transport (RTRS)
****************************

RTRS (RDMA Transport) is a reliable high speed transport library
which provides support to establish optimal number of connections
between client and server machines using RDMA (InfiniBand, RoCE, iWarp)
transport. It is optimized to transfer (read/write) IO blocks.

In its core interface it follows the BIO semantics of providing the
possibility to either write data from an sg list to the remote side
or to request ("read") data transfer from the remote side into a given
sg list.

RTRS provides I/O fail-over and load-balancing capabilities by using
multipath I/O (see "add_path" and "mp_policy" configuration entries in
Documentation/ABI/testing/sysfs-class-rtrs-client).

RTRS is used by the RNBD (RDMA Network Block Device) modules.

==================
Transport protocol
==================

Overview
--------
An established connection between a client and a server is called rtrs
session. A session is associated with a set of memory chunks reserved on the
server side for a given client for rdma transfer. A session
consists of multiple paths, each representing a separate physical link
between client and server. Those are used for load balancing and failover.
Each path consists of as many connections (QPs) as there are cpus on
the client.

When processing an incoming write or read request, rtrs client uses memory
chunks reserved for him on the server side. Their number, size and addresses
need to be exchanged between client and server during the connection
establishment phase. Apart from the memory related information client needs to
inform the server about the session name and identify each path and connection
individually.

On an established session client sends to server write or read messages.
Server uses immediate field to tell the client which request is being
acknowledged and for errno. Client uses immediate field to tell the server
which of the memory chunks has been accessed and at which offset the message
can be found.

Module parameter always_invalidate is introduced for the security problem
discussed in LPC RDMA MC 2019. When always_invalidate=Y, on the server side we
invalidate each rdma buffer before we hand it over to RNBD server and
then pass it to the block layer. A new rkey is generated and registered for the
buffer after it returns back from the block layer and RNBD server.
The new rkey is sent back to the client along with the IO result.
The procedure is the default behaviour of the driver. This invalidation and
registration on each IO causes performance drop of up to 20%. A user of the
driver may choose to load the modules with this mechanism switched off
(always_invalidate=N), if he understands and can take the risk of a malicious
client being able to corrupt memory of a server it is connected to. This might
be a reasonable option in a scenario where all the clients and all the servers
are located within a secure datacenter.


Connection establishment
------------------------

1. Client starts establishing connections belonging to a path of a session one
by one via attaching RTRS_MSG_CON_REQ messages to the rdma_connect requests.
Those include uuid of the session and uuid of the path to be
established. They are used by the server to find a persisting session/path or
to create a new one when necessary. The message also contains the protocol
version and magic for compatibility, total number of connections per session

Annotation

Implementation Notes