driver-api/md/raid5-cache.rst

5 Raid 4/5/6 could include an extra disk for data cache besides normal RAID
7 caches data to the RAID disks. The cache can be in write-through (supported
8 since 4.4) or write-back mode (supported since 4.10). mdadm (supported since
9 3.4) has a new option '--write-journal' to create array with cache. Please
11 in write-through mode. A user can switch it to write-back mode by::
13 	echo "write-back" > /sys/block/md0/md/journal_mode
15 And switch it back to write-through mode by::
17 	echo "write-through" > /sys/block/md0/md/journal_mode
22 write-through mode
25 This mode mainly fixes the 'write hole' issue. For RAID 4/5/6 array, an unclean
26 shutdown can cause data in some stripes to not be in consistent state, eg, data
27 and parity don't match. The reason is that a stripe write involves several RAID
29 unclean shutdown. We call an array degraded if it has inconsistent data. MD
31 resync completes, any system crash will expose the chance of real data
32 corruption in the RAID array. This problem is called 'write hole'.
34 The write-through cache will cache all data on cache disk first. After the data
35 is safe on the cache disk, the data will be flushed onto RAID disks. The
36 two-step write will guarantee MD can recover correct data after unclean
37 shutdown even the array is degraded. Thus the cache can close the 'write hole'.
39 In write-through mode, MD reports IO completion to upper layer (usually
40 filesystems) after the data is safe on RAID disks, so cache disk failure
41 doesn't cause data loss. Of course cache disk failure means the array is
42 exposed to 'write hole' again.
44 In write-through mode, the cache disk isn't required to be big. Several
47 write-back mode
50 write-back mode fixes the 'write hole' issue too, since all write data is
51 cached on cache disk. But the main goal of 'write-back' cache is to speed up
52 write. If a write crosses all RAID disks of a stripe, we call it full-stripe
53 write. For non-full-stripe writes, MD must read old data before the new parity
54 can be calculated. These synchronous reads hurt write throughput. Some writes
56 overhead too. Write-back cache will aggregate the data and flush the data to
57 RAID disks only after the data becomes a full stripe write. This will
59 typical workload which does sequential write followed by fsync is an example.
61 In write-back mode, MD reports IO completion to upper layer (usually
62 filesystems) right after the data hits cache disk. The data is flushed to raid
64 data loss.
66 In write-back mode, MD also caches data in memory. The memory cache includes
67 the same data stored on cache disk, so a power loss doesn't cause data loss.
73 Too small cache disk will make the write aggregation less efficient in this
75 least several gigabytes size in write-back mode.
80 The write-through and write-back cache use the same disk format. The cache disk
81 is organized as a simple write log. The log consists of 'meta data' and 'data'
82 pairs. The meta data describes the data. It also includes checksum and sequence
83 ID for recovery identification. Data can be IO data and parity data. Data is
84 checksummed too. The checksum is stored in the meta data ahead of the data. The
85 checksum is an optimization because MD can write meta and data freely without
86 worry about the order. MD superblock has a field pointed to the valid meta data
90 order in which MD writes data to cache disk and RAID disks. Specifically, in
91 write-through mode, MD calculates parity for IO data, writes both IO data and
92 parity to the log, writes the data and parity to RAID disks after the data and
96 In write-back mode, MD writes IO data to the log and reports IO completion. The
97 data is also fully cached in memory at that time, which means read must query
98 memory cache. If some conditions are met, MD will flush the data to RAID disks.
99 MD will calculate parity for the data and write parity into the log. After this
100 is finished, MD will write both data and parity into RAID disks, then MD can
102 stripe write, free cache disk space is low or free in-kernel memory cache space
105 After an unclean shutdown, MD does recovery. MD reads all meta data and data
107 data and data. If MD finds a stripe with data and valid parities (1 parity for
108 raid4/5 and 2 for raid6), MD will write the data and parities to RAID disks. If
109 parities are incompleted, they are discarded. If part of data is corrupted,
110 they are discarded too. MD then loads valid data and writes them to RAID disks