Documentation/driver-api/md/md-cluster.rst

Source file repositories/reference/linux-study-clean/Documentation/driver-api/md/md-cluster.rst

File Facts

System
Linux kernel
Corpus path
Documentation/driver-api/md/md-cluster.rst
Extension
.rst
Size
13621 bytes
Lines
386
Domain
Support Tooling And Documentation
Bucket
Documentation
Inferred role
Support Tooling And Documentation: documentation
Status
atlas-only

Why This File Exists

Repository support layer: documentation, build tooling, samples, user-space helper tools, generated initramfs support, licenses, and validation utilities.

Dependency Surface

Detected Declarations

Annotated Snippet

==========
MD Cluster
==========

The cluster MD is a shared-device RAID for a cluster, it supports
two levels: raid1 and raid10 (limited support).


1. On-disk format
=================

Separate write-intent-bitmaps are used for each cluster node.
The bitmaps record all writes that may have been started on that node,
and may not yet have finished. The on-disk layout is::

  0                    4k                     8k                    12k
  -------------------------------------------------------------------
  | idle                | md super            | bm super [0] + bits |
  | bm bits[0, contd]   | bm super[1] + bits  | bm bits[1, contd]   |
  | bm super[2] + bits  | bm bits [2, contd]  | bm super[3] + bits  |
  | bm bits [3, contd]  |                     |                     |

During "normal" functioning we assume the filesystem ensures that only
one node writes to any given block at a time, so a write request will

 - set the appropriate bit (if not already set)
 - commit the write to all mirrors
 - schedule the bit to be cleared after a timeout.

Reads are just handled normally. It is up to the filesystem to ensure
one node doesn't read from a location where another node (or the same
node) is writing.


2. DLM Locks for management
===========================

There are three groups of locks for managing the device:

2.1 Bitmap lock resource (bm_lockres)
-------------------------------------

 The bm_lockres protects individual node bitmaps. They are named in
 the form bitmap000 for node 1, bitmap001 for node 2 and so on. When a
 node joins the cluster, it acquires the lock in PW mode and it stays
 so during the lifetime the node is part of the cluster. The lock
 resource number is based on the slot number returned by the DLM
 subsystem. Since DLM starts node count from one and bitmap slots
 start from zero, one is subtracted from the DLM slot number to arrive
 at the bitmap slot number.

 The LVB of the bitmap lock for a particular node records the range
 of sectors that are being re-synced by that node.  No other
 node may write to those sectors.  This is used when a new nodes
 joins the cluster.

2.2 Message passing locks
-------------------------

 Each node has to communicate with other nodes when starting or ending
 resync, and for metadata superblock updates.  This communication is
 managed through three locks: "token", "message", and "ack", together
 with the Lock Value Block (LVB) of one of the "message" lock.

2.3 new-device management
-------------------------

 A single lock: "no-new-dev" is used to coordinate the addition of
 new devices - this must be synchronized across the array.
 Normally all nodes hold a concurrent-read lock on this device.

Annotation

Implementation Notes