arch/s390/vfio-ccw.rst

2 vfio-ccw: the basic infrastructure
6 ------------
9 Linux/s390. Motivation for vfio-ccw is to passthrough subchannels to a
13 I/O access method, which is so called Channel I/O. It has its own access
16 - Channel programs run asynchronously on a separate (co)processor.
17 - The channel subsystem will access any memory designated by the caller
18   in the channel program directly, i.e. there is no iommu involved.
24 regions to pass the channel programs from the mdev to its parent device
31 - A good start to know Channel I/O in general:
33 - s390 architecture:
34   s390 Principles of Operation manual (IBM Form. No. SA22-7832)
35 - The existing QEMU code which implements a simple emulated channel
41 - Documentation/driver-api/vfio-mediated-device.rst
43 Motivation of vfio-ccw
44 ----------------------
47 paravirtualized virtio devices via the "Virtio Over Channel I/O
48 (virtio-ccw)" transport. This makes virtio devices discoverable via
49 standard operating system algorithms for handling channel devices.
52 use the standard Channel I/O based mechanism, we also need to provide
58 For passing a device to a guest, we want to use the same interface as
59 everybody else, namely vfio. We implement this vfio support for channel
64 ------------------------------
66 s390 architecture has implemented a so called channel subsystem, that
74 All I/O requires the use of channel command words (CCWs). A CCW is an
75 instruction to a specialized I/O channel processor. A channel program is
76 a sequence of CCWs which are executed by the I/O channel subsystem.  To
77 issue a channel program to the channel subsystem, it is required to
80 operating system signals the I/O channel subsystem to begin executing
81 the channel program with a SSCH (start sub-channel) instruction. The
82 central processor is then free to proceed with non-I/O instructions
86 Back to vfio-ccw, in short:
88 - ORBs and channel programs are built in guest kernel (with guest
90 - ORBs and channel programs are passed to the host kernel.
91 - Host kernel translates the guest physical addresses to real addresses
92   and starts the I/O with issuing a privileged Channel I/O instruction
94 - channel programs run asynchronously on a separate processor.
95 - I/O completion will be signaled to the host with I/O interruptions.
100 -------------------------------------------
102 As mentioned above, we realize vfio-ccw with a mdev implementation.
104 Channel I/O does not have IOMMU hardware support, so the physical
105 vfio-ccw device does not have an IOMMU level translation or isolation.
108 handling the I/O instruction interception, vfio-ccw has the software
109 policing and translation how the channel program is programmed before
115 - The vfio_ccw driver for the physical subchannel device.
122   vfio_ccw also provides an I/O region to accept channel program
127 - The vfio_mdev driver for the mediated vfio ccw device.
135   it simply stores the translations for use by later requests. This
142   of the iova<->vaddr mappings in this operation. And they export a
148  +-------------+
150  | +---------+ | mdev_register_driver() +--------------+
151  | |  Mdev   | +<-----------------------+              |
153  | | driver  | +----------------------->+              |<-> VFIO user
154  | +---------+ |    probe()/remove()    +--------------+    APIs
159  | +---------+ | mdev_register_parent() +--------------+
160  | |Physical | +<-----------------------+              |
161  | | device  | |                        |  vfio_ccw.ko |<-> subchannel
162  | |interface| +----------------------->+              |     device
163  | +---------+ |       callback         +--------------+
164  +-------------+
176    Use the 'mdev_create' sysfs file, we need to manually create one (and
184 VFIO-CCW Regions
185 ----------------
187 The vfio-ccw driver exposes MMIO regions to accept requests from and return
190 vfio-ccw I/O region
191 -------------------
193 An I/O region is used to accept channel program request from user
221 ``-EOPNOTSUPP``
225 ``-EIO``
229 ``-EBUSY``
232 ``-EAGAIN``
235 ``-EACCES``
236   The channel path(s) used for the I/O were found to be not operational.
238 ``-ENODEV``
241 ``-EINVAL``
246 vfio-ccw cmd region
247 -------------------
249 The vfio-ccw cmd region is used to accept asynchronous instructions
261 Currently, CLEAR SUBCHANNEL and HALT SUBCHANNEL use this region.
269 ``-ENODEV``
272 ``-EINVAL``
275 ``-EIO``
279 ``-EAGAIN``
282 ``-EBUSY``
285 vfio-ccw schib region
286 ---------------------
288 The vfio-ccw schib region is used to return Subchannel-Information
301 vfio-ccw crw region
302 ---------------------
304 The vfio-ccw crw region is used to return Channel Report Word (CRW)
315 subchannel (e.g. one reporting changes in channel path state) is
319 similar to how STORE CHANNEL REPORT WORD works.
321 vfio-ccw operation details
322 --------------------------
324 vfio-ccw follows what vfio-pci did on the s390 platform and uses
325 vfio-iommu-type1 as the vfio iommu backend.
331   space, and assemble a runnable kernel channel program by updating the
333   Note that we have to use IDALs even for direct-access CCWs, as the
349   channel program to the kernel, to do further CCW translation before
355 The use of vfio-ccw is not limited to QEMU, while QEMU is definitely a
362 - Q1-Q7: QEMU side process.
363 - K1-K5: Kernel side process.
376     Write the guest channel program and ORB to the I/O region.
381 	Translate the guest channel program to a host kernel space
382 	channel program, which becomes runnable for a real device.
406 -----------
408 The current vfio-ccw implementation focuses on supporting basic commands
415 https://en.wikipedia.org/wiki/Direct-access_storage_device
419 through DASD/ECKD device online in a guest now and use it as a block
422 The current code allows the guest to start channel programs via
426 Currently all channel programs are prefetched, regardless of the
427 p-bit setting in the ORB.  As a result, self modifying channel
430 in QEMU's s390-ccw bios as of QEMU 4.1.
432 vfio-ccw supports classic (command mode) channel I/O only. Transport
439 ---------
440 1. ESA/s390 Principles of Operation manual (IBM Form. No. SA22-7832)
441 2. ESA/390 Common I/O Device Commands manual (IBM Form. No. SA22-7204)
444 5. Documentation/driver-api/vfio.rst
445 6. Documentation/driver-api/vfio-mediated-device.rst