Hypervisors in the cloud

• Type 1 (Xen, KVM…):
  • Optimized for maximum resource virtualization
    • Bare metal
  • Low performance overhead
    • Only one (big) task: run guest OSes
  • More secure
    • Isolation of guest OSes at lower level
• Type 2 (VirtualBox, QEMU/KVM…):
  • Easier to install and use
• For these reasons, the cloud relies on type 1 rather than type 2
  • But also operating system-level virtualization (next chapter)

Virtual machine: the stack

Modes of virtualization of a guest OS

CPU protection rings

• General protection mechanism
• Userspace in ring 3
  • Use hardware by asking the kernel through system calls (syscalls)
• Kernel in ring 0
  • Full, exclusive control over the hardware
• Other rings generally unused

CPU protection rings: full virtualization (1/2)

• Guest userspace in ring 3
  • Use hardware by asking the kernel through system calls (syscalls)
  • Implementation of syscalls uses interrupts, which control is privileged: the hypervisor redirects syscalls to the kernel in ring 1
• Kernel in ring 1, unmodified
  • Privileged operations are caught by the hypervisor
• Hypervisor in ring 0
  • Full, exclusive control over the hardware

CPU protection rings: full virtualization (2/2)

• The hypervisor implements trap and emulate workflow
  1. Privileged operations from the guest OS in ring 1 trigger General Protection Faults
  2. Hardware calls into ring 0 (i.e., hypervisor) to handle them
  3. Hypervisor emulates guest OS operations (shadowing)
• Upside: unmodified guest OS
• Downside: huge performance impact
  • This is visible when running a VM with QEMU, but without KVM!

CPU protection rings: paravirtualization (1/2)

• Guest userspace in ring 3
  • Use hardware by asking the kernel through system calls (syscalls)
  • Implementation of syscalls uses interrupts, which control is privileged: the hypervisor redirects syscalls to the kernel in ring 1
• Kernel in ring 1, modified for paravirtualization
  • Unmodified privileged operations are caught by the hypervisor
  • Modified privilege operations are implemented by requesting the hypervisor via hypercalls
• Hypervisor in ring 0
  • Full, exclusive control over the hardware

CPU protection rings: paravirtualization (2/2)

• The hypervisor offers an API (hypercalls) for the guest OS to ask for privileged operations without trap and emulation
• Upside: very good performance
• Downside: work to paravirtualize the guest OS
• Extends to paravirtualized devices:
  • Implementations tailored for virtual environments
  • Two sides: a front-end driver in the guest OS, and a back-end driver in the hypervisor
  • In QEMU/KVM: virtio drivers

CPU protection rings: hardware-assisted virtualization (1/2)

• Guest userspace in ring 3
  • Use hardware by asking the kernel through system calls (syscalls)
• Kernel in ring 0, unmodified
  • Most privileged operations are actually executed by the hardware, in a safe way
  • Some may be selected for trapping by the hypervisor
    • They trigger a VMEXIT to pass control
• Hypervisor in ring “-1”
  • Full, exclusive control over the hardware
  • Not an actual ring, but conceptually similar

CPU protection rings: hardware-assisted virtualization (1/2)

• Support from the hardware allows selected traps to be taken by the hypervisor
• Upside: very good performance with unmodified guest
• Downside: none (hardware upgrades, but it’s now widely available)
• Extends to memory: Second Level Address Translation (SLAT)
  • Intel: Extended Page Table (EPT)
  • AMD: Nested Page Table
• Extends to devices: IOMMU, virtualization of interrupts…

Hardware-assisted virtualization with KVM

kvmfd = open("/dev/kvm");
vmfd = ioctl(kvmfd, KVM_CREATE_VM);

struct kvm_userspace_memory_region region = {
    .guest_phys_addr = 0x1000, .memory_size = 0x4000,
    .userspace_addr = &guest_memory };
ioctl(vmfd, KVM_SET_USER_MEMORY_REGION, &region);

vcpufd = ioctl(vmfd, KVM_CREATE_VCPU);
// [...] Other architecture-specific vCPU initialization.

while (1) {
  switch (ioctl(vcpufd, KVM_RUN)) {
  case KVM_EXIT_HLT:      // [...] VM halted
  case KVM_EXIT_IO:       // [...] VM issued I/O operation
  case KVM_EXIT_SHUTDOWN: // [...] Virtual hardware fault
  // [...] Other error cases
  } }

• KVM relies on structures managed by the CPU: Virtual Machine Control Structure (VMCS, Intel)
  • Stores vCPU context: registers, flags, etc.)
  • Includes reason for switching to hypervisor context…
• KVM ioctls use special CPU instructions (examples are from Intel’s set)

Virtualized memory translation

• The physical MMU is already used for the hypervisor page table
• Add a level of memory address translation:
  • Software solution: shadow page table
  • Hardware-assisted solution: Second Level Address Translation (SLAT)

Virtualized memory translation: shadow page table

• Maintain a shadow page table (GVA to HPA) in the hypervisor
  • This is the one installed in the MMU
  • The guest OS’s page table (GVA to GPA) is unused
• Trap changes to the page table made by the guest OS
  • Every write is recalculated and stored in the shadow page table
• Pros: 1-dimension page walk (see SLAT next)
• Cons:
  • Very inefficient because of traps (full virtualization) on the critical path of memory management operations
  • Complex implementation

Virtualized memory translation: Second Level Address Translation (SLAT) (1/2)

1. Guest OS writes to its page table as natively, installs it in the MMU (GVA to GPA)
2. Hypervisor manages a second level page table, also installs it in the MMU (HVA to HPA)
3. The SLAT-capable MMU understands GPA as Host Virtual Address (HVA)

Virtualized memory translation: Second Level Address Translation (SLAT) (2/2)

• Pros:
  • Efficient memory management operations for an unmodified guest OS
  • Almost no implementation work
• Cons: 2-dimension page walk, 6 times slower address translation in the worst case
• Still the better solution:
  • Avoids complex implementation of shadow paging
  • Shadow paging is very slow anyway because of traps on memory management operations
  • Most performance overhead is compensated by:
    • The Translation Look aside Buffer (TLB) caches most translations
    • Huge pages avoid one level of translation

Virtualized memory translation: Second Level Address Translation (SLAT): Problem of 2-D page walk

• Given the native case: 1-D, 4 levels of page table → 4 memory accesses
• Virtualized case with SLAT: 4 levels of GVA to GPA, times 4 levels of GPA to HPA → 24 memory accesses
  • 4 levels of guest page table, addressed as GPA
  • → 4 memory accesses to translate the GPA of 1 level to HPA, plus 1 access to actually read the level = (4 + 1) × 4 = 20 accesses to walk the page table
  • The walk gives a GPA -> 4 more memory accesses to translate to HPA

Resource management: memory

• Hard to manage: spatial sharing
  • You can’t get more memory!
  • Different from CPU: time sharing, you can simply wait
• Overcommitment: resources are virtual, so give more memory than physically available
  • Rarely used: too harmful when it collapses because the system thrashes, swapping pages
• Ballooning: reclaim memory from guests
  1. Inflate: ask for memory pages
  2. Give the pages back to the HV
  • Paravirtualized mechanism
  • Rarely used: too hard to estimate balloon size
    • Too hard to estimate working set size
    • Too big makes the VM swap, destroys performance