Xen – Memory Management
(Part II)

Virtual Execution Environments – Group 009 Department of Computer Science and Engineering at IST

Daniel Andrade

Mudit Verma 2011/2012

Andr´ Amado e

Abstract
This paper presents an overview of Xen’s memory management system. Keywords: Xen, memory management, virtual machine, ballooning, migration

1

Introduction

Xen is an x86 virtual machine monitor that allows several operating systems to run in a managed manner on commodity hardware, with a minor impact on performance [1]. It provides a virtual machine abstraction, similar to the underlying hardware, to which operating systems can be ported. In this paper we detail Xen’s memory management model focusing on the ubiquitous x86. We first make a small introduction to some memory management fundamentals, proceed to talk about Xen’s memory management and conclude with an explanation of how it deals with memory during live migration.

ory and after the instruction is executed, results may be stored back in memory. Memory management is a central piece of an operating system. To understand Xen’s memory management model we must first be aware of how it is done in a regular operating system. Several approaches exist from a bare-machine approach to paging or segmentation [4]. Paging consists in dividing physical memory into fixed-sized blocks called frames and dividing logical memory into blocks of the same size called pages. When a process is near execution its pages are loaded into available memory frames. A page table is used to map logical memory addresses into physical memory addresses. This technique allows for the physical address space of a process to be noncontiguous solving the fragmentation problem present in other memory allocation strategies. In the segmentation approach memory is viewed as variable-sized segments instead of a linear array of bytes. The logical address space is composed by a collection of segments and external memory is subject to fragmentation. A segment table is used to translate between logical addresses and physical addresses.

2

Memory management overview 3 Xen’s memory management
Xen is responsible for managing memory allocation to guests and paging and segmentation hardware. It ensures that an unprivileged guest cannot access 11

The CPU can only access main memory and its own registers. It fetches instructions from memory and decodes those instructions. Once decoded more operands might have to be fetched from mem-

another domain’s share of memory, thus guaranteeing isolation among them. Guests running in HVM mode believe they are running in a regular system and as such can perform memory management as if they were not virtualized. These guests might detect that they are running in a Xen HVM domain and switch to using paravirtualized memory management for performance reasons. While in full virtualization guest OSes believe they are running on top of a real machine, in paravirtualization the guest OS understands that it is running inside a virtual machine allowing some optimizations to be made. The latter approach requires an explicit port of the operating system. Xen has two types of domains: domainO and domainU. DomainO start automatically on boot and is an OS with high privileges and direct access to physical hardware. It can be used to manage domainU OSes. DomainU is usually a modified version of a commodity operating systems though Xen also supports unmodified OSes. Ported OSes do, however, have access to Xen’s enhanced features and will be more efficient than their unmodified counterparts.

Xen takes advantage of this and places itself in ring 0 giving the guest OS kernel ring 1 access and its applications keep the ring 3 access. This way way the kernel and the applications do not have to be in the same protection ring. When two guest OS processes want to switch context they use page table and segmentation hypercalls and invoke Xen to switch the kernel stack pointer using the instruction stack switch(unsigned long ss, unsigned long esp) where ss is the new stack segment and esp is the new stack pointer.

3.3

Memory allocation

3.1

Page tables

Having read-only access to their page tables, guest OSes must invoke Xen through hypercalls when performing updates. To update a domain’s page table the instruction mmu update(mmu update t *req, int count, int *success count) can be used. This instruction allows hypercalls to be batched with count being the number of updates submitted for processing and successful_count the number of updates done successfully. Each element of req contains a value and an address with the lest significat two bits of the latter being used to identify the type of update.

3.2

Context switching

x86 is composed by four rings with privileges from 0 to 3 but usually only 0, the ring with the highest privilege, and 3 are used by the operating system. 12

The amount of memory allocated to each domain is decided on their creation. A maximum amount of allocatable memory, past that limit, can also be defined. Guest OSes can claim it from Xen when required and release it once it is not needed in order to save resources. To do this Xen uses a balloon driver module. The ballooning technique [5] seen in Fig. 1 induces the guest OS to cooperate in memory management. A small module is loaded into the guest OS as a device driver or kernel service. Inflating the balloon increases memory pressure causing the guest OS to invoke its own memory management algorithms. If memory is sufficiently low it will release memory in order to satisfy the fake module request. Deflating the balloon frees up memory to be used by the guest OS. The maximum memory value of the domain cannot be changed but the current memory amount can be grown or reduced using dom mem op(unsigned int op, unsigned long *extent list, unsigned long nr extents, unsigned int extent order) where op is the amount of memory to increase or decrease. Most OSes assume memory to be a continuous segment but Xen cannot guarantee the allocation of contigous regions of memory to guests. Typically guests themselves create the illusion of contiguous physical memory on top of dispersed hardware memory. It is their responsability to map from physical addresses to hardware addresses. Xen provides a read-only shared translation array to all

Figure 1: Ballooning: inflating the balloon increases memory pressure, forcing the guest OS to invoke its memory management algorithms; deflating the balloon decreases memory pressure freeing memory. domains to support an efficient hardware to physical addresses translation and needs only to validate updates to make sure guests do not modify others’ hardware page frames.

Figure 2: Shadow-mode is used by Xen for HVM domains. Since full virtualization of their page tables is needed, shadow page tables must be used but they are quite expensive to maintain.

3.4

Virtual address translation

Two possible approaches can be used for virtual address translation: shadow-mode shown in Fig. 2 and direct-mode shown in Fig. 3. While the first approach has a higher overhead it is still used for cases where the guest OS’s memory operations cannot be paravirtualized such as in Microsoft Windows [3]. Full virtualization uses shadow page tables. Each guest OS is given its own virtual page table with each access causing a trap. The hypervisor catches this trap, validates the update and propagates it to the MMU-visible shadow page table. In Xen this can done by intercepting only the updates to the virtual page table to reduce the overall cost of translation. Guest OS’ page tables are directly registered with the MMU. Guests are given 13

Figure 3: In direct-mode the guest OS has a copy of the page table in a set of read-only pages. When the guest tries to update these a fault occurs and control is transferred to the hypervisor, which then validates and translates them from pseudo-physical to machine.

read-only permissions to the page table and Xen receives update requests via a hypercall and validates them to ensure safity. Validation is aided by a type and a reference count associated with each machine page frame. A frame can be in one of the following states: page directory (PD), page table (PT), local descriptor table (LDT), global descriptor table (GDT), or writable (RW). A frame’s state can only be changed once its reference count reaches zero and it is unpinned, otherwise this could compromise safety. A guest OS indicates when a frame is allocated for page-table use requiring Xen to validate every entry in the frame. This frame’s type is then pinned to PD or PT until a request to unpin is made by the guest OS. This way there is no need to validate the page table when the page table base pointer changes. The number of hypercalls can be minimized by the Guest OSes’ by batching requests though this must be done in time to maintain the correct state of the system. A TLB flush is usually done by guests before using a mapping – ensuring cach translations are invalidated – so committing updates before flushing is sufficient to guarantee correctness. Some guests wait until no stale entries exist in the TLB, before flushing, which may cause a fault. The guest OS fault handler then as to check for outstanding updates flushing them if found and retries the instruction. A portion of RAM is always allocated by Xen on the top of the address space for its own private use. This is done to avoid a TLB flush when switching between hypervisor mode since it is a region inaccessible to guests. 3.4.1 Alternative mode

see the changes until Xen validates and reconnects the page.

4

Live migration

Xen virtual machines can be migrated in real-time beetween physical machines in a local-area network without loss of availability. This allows for a clear separation between software and hardware and brings several other advantages such as load balancing in a cluster, maintenance of the host machine without service interruption and avoidance of residual dependencies. The migration of a virtual machine from one physical host to another requires not only that its memory be transferred but also its local resources. These resources include disks and network interfaces but how live migration is achieved for devices is out of the scope of this paper. When moving a virtual machine’s memory from one host to another both downtime and total migration time must be minimized. Three approaches exist to transfer memory: push, stop-and-copy, and pull. Although the three of them could be mixed it does not usually happen in practice. Push Pages are pushed accross the network to the destination while the machine continues running. Modified pages are re-sent to ensure consistency. Stop-and-copy The source VM is stopped, pages are copied across the network to the destination and the new VM is started. This simple approach has the disadvantage that both downtime and total migration time are proportional to the amount of physical memory allocated to the VM. Pull The new VM executes once essential pages are copied to the new machine and pulls the remaining pages across the network when a page fault occurs. The downtime is smaller than in stop-and-copy but the total migration time increases as well as page faults until the transfer is complete. 14

There is an alternative mode – in between directand shadow-modes – of operation where guests have the illusion of having directly writable page tables. For security reasons this is not the case. What happens is that a page table update will cause a trap which in turn causes the page to be disconnected from its page table. At this point the guest can write to the page but the MMU will not

Xen’s approach to live migration – pre-copy migration – consists of a push phase, where modified pages are transferred in rounds, and a typically very short final stop-and-copy phase, with downtimes as low as 60ms [2].

5

Conclusion

As seen in this study, Xen employs interesting and clever memory management mechanisms. It provides a stable virtualization platform for several different uses while keeping the system security and virtual machine isolation.

References
[1] Barham, P., Dragovic, B., Fraser, K., Hand, S., Harris, T., Ho, A., Neugebauer, R., Pratt, I., and Warfield, A. Xen and the art of virtualization. SIGOPS Oper. Syst. Rev. 37, 5 (Oct. 2003), 164–177. [2] Clark, C., Fraser, K., Hand, S., Hansen, J. G., Jul, E., Limpach, C., Pratt, I., and Warfield, A. Live migration of virtual machines. In Proceedings of the 2nd conference on Symposium on Networked Systems Design & Implementation - Volume 2 (Berkeley, CA, USA, 2005), NSDI’05, USENIX Association, pp. 273–286. [3] Dittner, R., and Rule, D. The Best Damn Server Virtualization Book Period: Including Vmware, Xen, and Microsoft Virtual Server. Syngress Publishing, 2007. [4] Silberschatz, A., Galvin, P. B., and Gagne, G. Operating System Concepts, 8th ed. Wiley Publishing, 2008. [5] Waldspurger, C. A. Memory resource management in vmware esx server. SIGOPS Oper. Syst. Rev. 36, SI (Dec. 2002), 181–194.

15