kvm/x86/running-nested-guests.rst

1 .. SPDX-License-Identifier: GPL-2.0
4 Running nested guests with KVM
8 can be KVM-based or a different hypervisor).  The straightforward
12               .----------------.  .----------------.
17               |----------------'--'----------------|
22       .------------------------------------------------------.
25       |------------------------------------------------------|
27       '------------------------------------------------------'
31 - L0 – level-0; the bare metal host, running KVM
33 - L1 – level-1 guest; a VM running on L0; also called the "guest
34   hypervisor", as it itself is capable of running KVM.
36 - L2 – level-2 guest; a VM running on L1, this is the "nested guest"
42           For example, s390x always has an LPAR (LogicalPARtition)
43           hypervisor running on bare metal, adding another layer and
45           metal, running the LPAR hypervisor), L1 (host hypervisor), L2
48           This document will stick with the three-level terminology (L0,
54 ---------
59 - As a developer, you want to test your software on different operating
62   hypervisor" (level-1 guest).  This in turn allows you to create
63   multiple nested guests (level-2 guests), running different OSes, on
66 - Live migration of "guest hypervisors" and their nested guests, for
69 - VM image creation tools (e.g. ``virt-install``,  etc) often run
72 - Some OSes use virtualization internally for security (e.g. to let
77 -----------------------
83 In case you are running a Linux kernel older than v4.19, to enable
91     $ lsmod | grep -i kvm
97     $ modinfo kvm_intel | grep -i nested
105     options kvm-intel nested=y
107 4. Unload and re-load the KVM Intel module::
109     $ sudo rmmod kvm-intel
110     $ sudo modprobe kvm-intel
118 name is ``kvm-amd``.
121 Additional nested-related kernel parameters (x86)
122 -------------------------------------------------
139 .. note:: If you suspect your L2 (i.e. nested guest) is running slower,
145 -----------------------------
150     $ qemu-kvm -cpu host [...]
152 The above will pass through the host CPU's capabilities as-is to the
156     $ qemu-kvm -cpu Haswell-noTSX-IBRS,vmx=on
158 then the guest hypervisor will subsequently be capable of running a
163 -------------------------
177    (via the command-line ``-cpu host``).
185 ------------------------------
194 or save-and-load an L1 guest while an L2 guest is running will result in
199 actually running L2 guests, is expected to function normally even on AMD
202 Migrating an L2 guest is always expected to succeed, so all the following
205 - Migrating a nested guest (L2) to another L1 guest on the *same* bare
208 - Migrating a nested guest (L2) to another L1 guest on a *different*
211 - Migrating a nested guest (L2) to a bare metal host.
214 -----------------------------------
217 L0, L1 and L2; this can result in tedious back-n-forth between the bug
220 - Mention that you are in a "nested" setup.  If you are running any kind
225 - Ensure you are actually running KVM on KVM.  Sometimes people do not
227   them running with pure emulation or what QEMU calls it as "TCG", but
228   they think they're running nested KVM.  Thus confusing "nested Virt"
236   - Kernel, libvirt, and QEMU version from L0
238   - Kernel, libvirt and QEMU version from L1
240   - QEMU command-line of L1 -- when using libvirt, you'll find it here:
243   - QEMU command-line of L2 -- as above, when using libvirt, get the
244     complete libvirt-generated QEMU command-line
246   - ``cat /sys/cpuinfo`` from L0
248   - ``cat /sys/cpuinfo`` from L1
250   - ``lscpu`` from L0
252   - ``lscpu`` from L1
254   - Full ``dmesg`` output from L0
256   - Full ``dmesg`` output from L1
258 x86-specific info to collect
264   - Output of: ``x86info -a`` from L0
266   - Output of: ``x86info -a`` from L1
268   - Output of: ``dmidecode`` from L0
270   - Output of: ``dmidecode`` from L1
272 s390x-specific info to collect
278   - ``/proc/sysinfo`` from L1; this will also include the info from L0