Documentation/PCI/pci-error-recovery.rst

1 .. SPDX-License-Identifier: GPL-2.0
8 :Authors: - Linas Vepstas <linasvepstas@gmail.com>
9           - Richard Lary <rlary@us.ibm.com>
10           - Mike Mason <mmlnx@us.ibm.com>
14 PCI errors on the bus, such as parity errors on the data and address
16 chipsets are able to deal with these errors; these include PCI-E chipsets,
17 and the PCI-host bridges found on IBM Power4, Power5 and Power6-based
22 offered, so that the affected PCI device(s) are reset and put back
23 into working condition. The reset phase requires coordination
32 including multiple instances of a device driver on multi-function
34 waiting for some i/o-space register to change, when it never will.
39 is forced by the need to handle multi-function devices, that is,
42 of reset it desires, the choices being a simple re-enabling of I/O
43 or requesting a slot reset.
45 If any driver requests a slot reset, that is what will be done.
47 After a reset and/or a re-enabling of I/O, all drivers are
50 "resume normal operations" event is sent out.
52 The biggest reason for choosing a kernel-based implementation rather
53 than a user-space implementation was the need to deal with bus
56 file system is disconnected, a user-space mechanism would have to go
62 for example, the SCSI-generic layer already provides significant
69 Design and implementation details below, based on a chain of
74 pci_driver. A driver that fails to provide the structure is "non-aware",
85 		void (*resume)(struct pci_dev *dev);
101 		PCI_ERS_RESULT_CAN_RECOVER, /* Device driver can recover without slot reset */
102 		PCI_ERS_RESULT_NEED_RESET,  /* Device driver wants slot to be reset. */
110 For example, if mmio_enabled() and resume() aren't there, then it
112 a slot reset.  Typically a driver will want to know about
116 event will be platform-dependent, but will follow the general
120 -------------------
121 A PCI bus error is detected by the PCI hardware.  On powerpc, the slot
127 --------------------
128 Platform calls the error_detected() callback on every instance of
131 At this point, the device might not be accessible anymore, depending on
132 the platform (the slot will be isolated on powerpc). The driver may
144   - PCI_ERS_RESULT_CAN_RECOVER
149   - PCI_ERS_RESULT_NEED_RESET
151       slot reset.
152   - PCI_ERS_RESULT_DISCONNECT
155 The next step taken will depend on the result codes returned by the
158 If all drivers on the segment/slot return PCI_ERS_RESULT_CAN_RECOVER,
159 then the platform should re-enable IOs on the slot (or do nothing in
163 If any driver requested a slot reset (by returning PCI_ERS_RESULT_NEED_RESET),
164 then recovery proceeds to STEP 4 (Slot Reset).
175    Doing better requires complex multi-threaded logic in the error
182    a value of 0xff on read, and writes will be dropped. If more than
189 --------------------
190 The platform re-enables MMIO to the device (but typically not the
191 DMA), and then calls the mmio_enabled() callback on all affected
198 reset or some such, but not restart operations. This callback is made if
199 all drivers on a segment agree that they can try to recover and if no automatic
200 link reset was performed by the HW. If the platform can't just re-enable IOs
201 without a slot reset or a link reset, it will not call this callback, and
202 instead will have gone directly to STEP 3 (Link Reset) or STEP 4 (Slot Reset)
211    such an error might cause IOs to be re-blocked for the whole
213    on the same segment might have done, forcing the whole segment
214    into one of the next states, that is, link reset or slot reset.
217   - PCI_ERS_RESULT_RECOVERED
222       allowed to proceed, as another driver on the
224       slot reset on platforms that support it.
226   - PCI_ERS_RESULT_NEED_RESET
228       recoverable in its current state and it needs a slot
229       reset to proceed.
231   - PCI_ERS_RESULT_DISCONNECT
233       reset driver dead. (To be defined more precisely)
235 The next step taken depends on the results returned by the drivers.
237 proceeds to either STEP3 (Link Reset) or to STEP 5 (Resume Operations).
240 proceeds to STEP 4 (Slot Reset)
242 STEP 3: Link Reset
243 ------------------
244 The platform resets the link.  This is a PCI-Express specific step
248 STEP 4: Slot Reset
249 ------------------
252 platform will perform a slot reset on the requesting PCI device(s).
253 The actual steps taken by a platform to perform a slot reset
254 will be platform-dependent. Upon completion of slot reset, the
257 Powerpc platforms implement two levels of slot reset:
258 soft reset(default) and fundamental(optional) reset.
260 Powerpc soft reset consists of asserting the adapter #RST line and then
263 power-on followed by power-on BIOS/system firmware initialization.
264 Soft reset is also known as hot-reset.
266 Powerpc fundamental reset is supported by PCI Express cards only
270 For most PCI devices, a soft reset will be sufficient for recovery.
271 Optional fundamental reset is provided to support a limited number
272 of PCI Express devices for which a soft reset is not sufficient
275 If the platform supports PCI hotplug, then the reset might be
276 performed by toggling the slot electrical power off/on.
280 a slot reset, the device driver will almost always use its standard
284 This call gives drivers the chance to re-initialize the hardware
285 (re-download firmware, etc.).  At this point, the driver may assume
288 memory mapped I/O space and DMA. Interrupts (Legacy, MSI, or MSI-X)
292 at this point.  If all device drivers report success on this
293 callback, the platform will call resume() to complete the sequence,
297 it can't get the device operational after reset.  If the platform
298 previously tried a soft reset, it might now try a hard reset (power
304 Drivers for multi-function cards will need to coordinate among
305 themselves as to which driver instance will perform any "one-shot"
309 	+       if (PCI_FUNC(pdev->devfn) == 0)
313 	- PCI_ERS_RESULT_DISCONNECT
316 Drivers for PCI Express cards that require a fundamental reset must
321 	+	/* Set EEH reset type to fundamental if required by hba  */
323 	+		pdev->needs_freset = 1;
326 Platform proceeds either to STEP 5 (Resume Operations) or STEP 6 (Permanent
331    The current powerpc implementation does not try a power-cycle
332    reset if the driver returned PCI_ERS_RESULT_DISCONNECT.
336 STEP 5: Resume Operations
337 -------------------------
338 The platform will call the resume() callback on all affected device
339 drivers if all drivers on the segment have returned
349 -------------------------
355 cancel all pending I/O, refuse all new I/O, returning -EIO to
361 permanent failure in some way.  If the device is hotplug-capable,
364 caused by over-heating, some by a poorly seated card. Many
367 errors. See the discussion in Documentation/arch/powerpc/eeh-pci-error-recovery.rst
368 for additional detail on real-life experience of the causes of
373 ---------------------------
375 no slot reset capability may want to just "ignore" drivers that can't
376 recover (disconnect them) and try to let other cards on the same segment
385  - There is no guarantee that interrupt delivery can proceed from any
386    device on the segment starting from the error detection and until the
390  - There is no guarantee that interrupt delivery is stopped, that is,
397    interrupts are routed to error-management capable slots and can deal
407    the file Documentation/arch/powerpc/eeh-pci-error-recovery.rst
413    - drivers/scsi/ipr
414    - drivers/scsi/sym53c8xx_2
415    - drivers/scsi/qla2xxx
416    - drivers/scsi/lpfc
417    - drivers/next/bnx2.c
418    - drivers/next/e100.c
419    - drivers/net/e1000
420    - drivers/net/e1000e
421    - drivers/net/ixgbe
422    - drivers/net/cxgb3
423    - drivers/net/s2io.c
429    - drivers/cxl/pci.c
432 -------