Computer Organization and Design
The Hardware/Software Interface
F I F T H E D I T I O N

David A. Patterson University of California, Berkeley John L. Hennessy Stanford University With contributions by Perry Alexander The University of Kansas Peter J. Ashenden Ashenden Designs Pty Ltd Jason D. Bakos University of South Carolina Javier Bruguera Universidade de Santiago de Compostela Jichuan Chang Hewlett-Packard Matthew Farrens University of California, Davis David Kaeli
Northeastern University Nicole Kaiyan University of Adelaide David Kirk NVIDIA

James R. Larus School of Computer and Communications Science at EPFL Jacob Leverich Hewlett-Packard Kevin Lim
Hewlett-Packard

John Nickolls NVIDIA John Oliver
Cal Poly, San Luis Obispo Milos Prvulovic Georgia Tech

Partha Ranganathan Hewlett-Packard

Table of Contents
Cover image Title page In Praise of Computer Organization and Design: The Hardware/Software Interface, Fifth Edition Front-matter Copyright Dedication Acknowledgments Preface
About This Book About the Other Book Changes for the Fifth Edition Changes for the Fifth Edition Concluding Remarks Acknowledgments for the Fifth Edition

1. Computer Abstractions and Technology
1.1 Introduction 1.2 Eight Great Ideas in Computer Architecture 1.3 Below Your Program 1.4 Under the Covers 1.5 Technologies for Building Processors and Memory 1.6 Performance 1.7 The Power Wall 1.8 The Sea Change: The Switch from Uniprocessors to Multiprocessors 1.9 Real Stuff: Benchmarking the Intel Core i7 1.10 Fallacies and Pitfalls 1.11 Concluding Remarks 1.12 Historical Perspective and Further Reading 1.13 Exercises

2. Instructions: Language of the Computer
2.1 Introduction 2.2 Operations of the Computer Hardware 2.3 Operands of the Computer Hardware 2.4 Signed and Unsigned Numbers 2.5 Representing Instructions in the Computer 2.6 Logical Operations 2.7 Instructions for Making Decisions 2.8 Supporting Procedures in Computer Hardware 2.9 Communicating with People 2.10 MIPS Addressing for 32-bit Immediates and Addresses 2.11 Parallelism and Instructions: Synchronization

2.12 Translating and Starting a Program 2.13 A C Sort Example to Put It All Together 2.14 Arrays versus Pointers 2.15 Advanced Material: Compiling C and Interpreting Java 2.16 Real Stuff: ARMv7 (32-bit) Instructions 2.17 Real Stuff: x86 Instructions 2.18 Real Stuff: ARMv8 (64-bit) Instructions 2.19 Fallacies and Pitfalls 2.20 Concluding Remarks 2.21 Historical Perspective and Further Reading 2.22 Exercises

3. Arithmetic for Computers
3.1 Introduction 3.2 Addition and Subtraction 3.3 Multiplication 3.4 Division 3.5 Floating Point 3.6 Parallelism and Computer Arithmetic: Subword Parallelism 3.7 Real Stuff: Streaming SIMD Extensions and Advanced Vector Extensions in x86 3.8 Going Faster: Subword Parallelism and Matrix Multiply 3.9 Fallacies and Pitfalls 3.10 Concluding Remarks 3.11 Historical Perspective and Further Reading 3.12 Exercises

4. The Processor
4.1 Introduction 4.2 Logic Design Conventions 4.3 Building a Datapath 4.4 A Simple Implementation Scheme 4.5 An Overview of Pipelining 4.6 Pipelined Datapath and Control 4.7 Data Hazards: Forwarding versus Stalling 4.8 Control Hazards 4.9 Exceptions 4.10 Parallelism via Instructions 4.11 Real Stuff: The ARM Cortex-A8 and Intel Core i7 Pipelines 4.12 Going Faster: Instruction-Level Parallelism and Matrix Multiply 4.13 Advanced Topic: an Introduction to Digital Design Using a Hardware Design Language to Describe and Model a Pipeline and More Pipelining Illustrations 4.14 Fallacies and Pitfalls 4.15 Concluding Remarks 4.16 Historical Perspective and Further Reading 4.17 Exercises

5. Large and Fast: Exploiting Memory Hierarchy
5.1 Introduction 5.2 Memory Technologies 5.3 The Basics of Caches 5.4 Measuring and Improving Cache Performance 5.5 Dependable Memory Hierarchy 5.6 Virtual Machines

5.7 Virtual Memory 5.8 A Common Framework for Memory Hierarchy 5.9 Using a Finite-State Machine to Control a Simple Cache 5.10 Parallelism and Memory Hierarchy: Cache Coherence 5.11 Parallelism and Memory Hierarchy: Redundant Arrays of Inexpensive Disks 5.12 Advanced Material: Implementing Cache Controllers 5.13 Real Stuff: The ARM Cortex-A8 and Intel Core i7 Memory Hierarchies 5.14 Going Faster: Cache Blocking and Matrix Multiply 5.15 Fallacies and Pitfalls 5.16 Concluding Remarks 5.17 Historical Perspective and Further Reading 5.18 Exercises

6. Parallel Processors from Client to Cloud
6.1 Introduction 6.2 The Difficulty of Creating Parallel Processing Programs 6.3 SISD, MIMD, SIMD, SPMD, and Vector 6.4 Hardware Multithreading 6.5 Multicore and Other Shared Memory Multiprocessors 6.6 Introduction to Graphics Processing Units 6.7 Clusters, Warehouse Scale Computers, and Other Message-Passing Multiprocessors 6.8 Introduction to Multiprocessor Network Topologies 6.9 Communicating to the Outside World: Cluster Networking 6.10 Multiprocessor Benchmarks and Performance Models 6.11 Real Stuff: Benchmarking and Rooflines of the Intel Core i7 960 and the NVIDIA Tesla GPU 6.12 Going Faster: Multiple Processors and Matrix Multiply 6.13 Fallacies and Pitfalls 6.14 Concluding Remarks 6.15 Historical Perspective and Further Reading 6.16 Exercises

Appendix A. Assemblers, Linkers, and the SPIM Simulator
Abstract A.1 Introduction A.2 Assemblers A.3 Linkers A.4 Loading A.5 Memory Usage A.6 Procedure Call Convention A.7 Exceptions and Interrupts A.8 Input and Output A.9 SPIM A.10 MIPS R2000 Assembly Language A.11 Concluding Remarks A.12 Exercises Further Reading

Appendix B. The Basics of Logic Design
Abstract B.1 Introduction B.2 Gates, Truth Tables, and Logic Equations B.3 Combinational Logic

B.4 Using a Hardware Description Language B.5 Constructing a Basic Arithmetic Logic Unit B.6 Faster Addition: Carry Lookahead B.7 Clocks B.8 Memory Elements: Flip-Flops, Latches, and Registers B.9 Memory Elements: SRAMs and DRAMs B.10 Finite-State Machines B.11 Timing Methodologies B.12 Field Programmable Devices B.13 Concluding Remarks B.14 Exercises Further Reading

Appendix C. Graphics and Computing GPUs
Abstract C.1 Introduction C.2 GPU System Architectures C.3 Programming GPUs C.4 Multithreaded Multiprocessor Architecture C.5 Parallel Memory System C.6 Floating-point Arithmetic C.7 Real Stuff: The NVIDIA GeForce 8800 C.8 Real Stuff: Mapping Applications to GPUs C.9 Fallacies and Pitfalls C.10 Concluding Remarks C.11 Historical Perspective and Further Reading Further Reading

Appendix D. Mapping Control to Hardware
Abstract D.1 Introduction D.2 Implementing Combinational Control Units Mapping the ALU Control Function to Gates Mapping the Main Control Function to Gates D.3 Implementing Finite-State Machine Control A ROM Implementation A PLA Implementation D.4 Implementing the Next-State Function with a Sequencer Optimizing the Control Implementation D.5 Translating a Microprogram to Hardware Organizing the Control to Reduce the Logic D.6 Concluding Remarks D.7 Exercises

Appendix E. A Survey of RISC Architectures for Desktop, Server, and Embedded Computers
Abstract E.1 Introduction E.2 Addressing Modes and Instruction Formats E.3 Instructions: the MIPS Core Subset MIPS Core Instructions Compare and Conditional Branch E.4 Instructions: Multimedia Extensions of the Desktop/Server RISCs

E.5 Instructions: Digital Signal-Processing Extensions of the Embedded RISCs E.6 Instructions: Common Extensions to MIPS Core E.7 Instructions Unique to MIPS-64 Nonaligned Data Transfers Remaining Instructions E.8 Instructions Unique to Alpha Remaining Instructions E.9 Instructions Unique to SPARC v.9 Register Windows Fast Traps Support for LISP and Smalltalk Overlapped Integer and Floating-Point Operations Remaining Instructions E.10 Instructions Unique to PowerPC Branch Registers: Link and Counter Remaining Instructions E.11 Instructions Unique to PA-RISC 2.0 Nullification A Cornucopia of Conditional Branches Synthesized Multiply and Divide Decimal Operations Remaining Instructions E.12 Instructions Unique to ARM Remaining Instructions E.13 Instructions Unique to Thumb E.14 Instructions Unique to SuperH E.15 Instructions Unique to M32R E.16 Instructions Unique to MIPS-16 E.17 Concluding Remarks Further Reading

Glossary Further Reading
Chapter 1 Chapter 2 Chapter 3 Chapter 4 Chapter 5 Chapter 6 Appendix A Appendix B Appendix C Appendix E

Index

In Praise of Computer Organization and Design: The Hardware/Software Interface, Fifth Edition
“Textbook selection is often a frustrating act of compromise—pedagogy, content coverage, quality of exposition, level of rigor, cost. Computer Organization and Design is the rare book that hits all the right notes across the board, without compromise. It is not only the premier computer organization textbook, it is a shining example of what all computer science textbooks could and should be.”
—Michael Goldweber, Xavier University

“I have been using Computer Organization and Design for years, from the very first edition. The new Fifth Edition is yet another outstanding improvement on an already classic text. The evolution from desktop computing to mobile computing to Big Data brings new coverage of embedded processors such as the ARM, new material on how software and hardware interact to increase performance, and cloud computing. All this without sacrificing the fundamentals.”
—Ed Harcourt, St. Lawrence University

“To Millennials: Computer Organization and Design is the computer architecture book you should keep on your (virtual) bookshelf. The book is both old and new, because it develops venerable principles—Moore’s Law, abstraction, common case fast, redundancy, memory hierarchies, parallelism, and pipelining—but illustrates them with contemporary designs, e.g., ARM Cortex A8 and Intel Core i7.”
—Mark D. Hill, University of Wisconsin-Madison

“The new edition of Computer Organization and Design keeps pace with advances in emerging embedded and many-core (GPU) systems, where tablets and smartphones will are quickly becoming our new desktops. This text acknowledges these changes, but continues to provide a rich foundation of the fundamentals in computer organization and design which will be needed for the designers of hardware and software that power this new class of devices and systems.”
—Dave Kaeli, Northeastern University

“The Fifth Edition of Computer Organization and Design provides more than an introduction to computer architecture. It prepares the reader for the changes necessary to meet the ever-increasing performance needs of mobile systems and big data processing at a time that difficulties in semiconductor scaling are making all systems power constrained. In this new era for computing, hardware and software must be co-designed and system-level architecture is as critical as component-level optimizations.”
—Christos Kozyrakis, Stanford University

“Patterson and Hennessy brilliantly address the issues in ever-changing computer hardware architectures, emphasizing on interactions among hardware and software components at various abstraction levels. By interspersing I/O and parallelism concepts with a variety of mechanisms in hardware and software throughout the book, the new edition achieves an excellent holistic presentation of computer architecture for the PostPC era. This book is an essential guide to hardware and software professionals facing energy efficiency and parallelization challenges in Tablet PC to cloud computing.”
—Jae C. Oh, Syracuse University

Front-matter

David A. Pa erson has been teaching computer architecture at the U niversity of California, Berkeley, since joining the faculty in 1977, where he holds the Pardee Chair of Computer S cience. His teaching has been honored by the Distinguished Teaching Award from the U niversity of California, the Karlstrom Award from ACM, and the Mulligan Education Medal and U ndergraduate Teaching Award from IEEE. Pa erso received the IEEE Technical Achievement Award and the ACM Eckert-Mauchly Award for contributions to RIS C, and he shared the IEEE Johns Information S torage Award for contributions to RAID. He also shared the IEEE John von Neumann Medal and the C & C P rize with John Hennes Like his co-author, Pa erson is a Fellow of the American Academy of Arts and S ciences, the Computer History Museum, ACM, and IEEE, and h was elected to the National Academy of Engineering, the National Academy of Sciences, and the Silicon Valley Engineering Hall of Fame. He served on the Information Technology Advisory Commi ee to the U .S . P resident, as chair of the CS division in the Berkeley EECS department, as chair o the Computing Research Association, and as President of ACM. This record led to Distinguished Service Awards from ACM and CRA. At Berkeley, Pa erson led the design and implementation of RIS C I, likely the first VLS I reduced instruction set computer, and the foundation of the commercial S PARC architecture. He was a leader of the Redundant Arrays of Inexpensive Disks (RAID) project, which led to dependable storag systems from many companies. He was also involved in the Network of Workstations (NO W) project, which led to cluster technology used by Internet companies and later to cloud computing. These projects earned three dissertation awards from ACM. His current research projects are Algorithm-Machine-People and Algorithms and S pecializers for P rovably O ptimal Implementations with Resilience and Efficiency. The AMP Lab developing scalable machine learning algorithms, warehouse-scale-computer-friendly programming models, and crowd-sourcing tools to gain valuable insights quickly from big data in the cloud. The AS P IRE Lab uses deep hardware and software co-tuning to achieve the highest possible performance and energy efficiency for mobile and rack computing systems. John L. Hennessyis the tenth president of S tanford U niversity, where he has been a member of the faculty since 1977 in the departments of electrical engineering and computer science. Hennessy is a Fellow of the IEEE and ACM; a member of the National Academy of Engineering, the National Academy of S cience, and the American P hilosophical S ociety; and a Fellow of the American Academy of Arts and S ciences. Among his many awards are the 2001 Eckert-Mauchly Award for his contributions to RIS C technology, the 2001 S eymour Cray Computer Engineering Award and the 2000 John von Neumann Award, which he shared with David Patterson. He has also received seven honorary doctorates. In 1981, he started the MIP S project at S tanford with a handful of graduate students. After completing the project in 1984, he took a leave from the university to cofound MIPS Computer Systems (now MIPS Technologies), which developed one of the first commercial RISC microprocessors. As of 2006, over 2 billion MIP S microprocessors have been shipped in devices ranging from video games and palmtop computers to laser printers and network switches. Hennessy subsequently led the DAS H (Director Architecture for S hared Memory) project, which prototyped the first scalable cache coherent multiprocessor; many of the key ideas have been adopted in modern multiprocessors. In addition to his technical activities and university responsibilities, he has continued to work with numerous start-ups both as an early-stage advisor and an investor.

Copyright
Acquiring Editor: Todd Green Development Editor: Nate McFadden Project Manager: Lisa Jones Designer: Russell Purdy Morgan Kaufmann is an imprint of Elsevier The Boulevard, Langford Lane, Kidlington, Oxford, OX5 1GB 225 Wyman Street, Waltham, MA 02451, USA Copyright © 2014 Elsevier Inc. All rights reserved No part of this publication may be reproduced or transmitted in any form or by any means, electronic or mechanical, including photocopying, recording, or any information storage and retrieval system, without permission in writing from the publisher. Details on how to seek permission, further information about the Publisher’s permissions policies and our arrangements with organizations such as the Copyright Clearance Center and the Copyright Licensing Agency, can be found at our website: www.elsevier.com/permissions This book and the individual contributions contained in it are protected under copyright by the Publisher (other than as may be noted herein).

Notices Knowledge and best practice in this field are constantly changing. As new research and experience broaden our understanding, changes in research methods or professional practices, may become necessary. Practitioners and researchers must always rely on their own experience and knowledge in evaluating and using any information or methods described herein. In using such information or methods they should be mindful of their own safety and the safety of others, including parties for whom they have a professional responsibility. To the fullest extent of the law, neither the publisher nor the authors, contributors, or editors, assume any liability for any injury and/or damage to persons or property as a matter of products liability, negligence or otherwise, or from any use or operation of any methods, products, instructions, or ideas contained in the material herein.

Library of Congress Cataloging-in-Publication Data Patterson, David A. Computer organization and design: the hardware/software interface/David A. Patterson, John L. Hennessy. — 5th ed. p. cm. — (The Morgan Kaufmann series in computer architecture and design) Rev. ed. of: Computer organization and design/John L. Hennessy, David A. Patterson. 1998. Summary: “Presents the fundamentals of hardware technologies, assembly language, computer arithmetic, pipelining, memory hierarchies and I/O”— Provided by publisher. ISBN 978-0-12-407726-3 (pbk.) 1. Computer organization. 2. Computer engineering. 3. Computer interfaces. I. Hennessy, John L. II. Hennessy, John L. Computer organization and design. III. Title. British Library Cataloguing-in-Publication Data A catalogue record for this book is available from the British Library ISBN: 978-0-12-407726-3

For information on all MK publications visit our website at www.mkp.com

Printed and bound in the United States of America 13 14 15 16 10 9 8 7 6 5 4 3 2 1

Dedication
To Linda, who has been, is, and always will be the love of my life

Acknowledgments
Figures 1.7, 1.8 Courtesy of iFixit (www.ifixit.com). Figure 1.9 Courtesy of Chipworks (www.chipworks.com). Figure 1.13 Courtesy of Intel. Figures 1.10.1, 1.10.2, 4.15.2 Courtesy of the Charles Babbage Institute, University of Minnesota Libraries, Minneapolis. Figures 1.10.3, 4.15.1, 4.15.3, 5.12.3, 6.14.2 Courtesy of IBM. Figure 1.10.4 Courtesy of Cray Inc. Figure 1.10.5 Courtesy of Apple Computer, Inc. Figure 1.10.6 Courtesy of the Computer History Museum. Figures 5.17.1, 5.17.2 Courtesy of Museum of Science, Boston. Figure 5.17.4 Courtesy of MIPS Technologies, Inc. Figure 6.15.1 Courtesy of NASA Ames Research Center.

Preface
David A. Patterson

The most beautiful thing we can experience is the mysterious. It is the source of all true art and science.
Albert Einstein, What I Believe, 1930

About This Book
We believe that learning in computer science and engineering should reflect the current state of the field, as well as introduce the principles that are shaping computing. We also feel that readers in every specialty of computing need to appreciate the organizational paradigms that determine the capabilities, performance, energy, and, ultimately, the success of computer systems. Modern computer technology requires professionals of every computing specialty to understand both hardware and software. The interaction between hardware and software at a variety of levels also offers a framework for understanding the fundamentals of computing. Whether your primary interest is hardware or software, computer science or electrical engineering, the central ideas in computer organization and design are the same. Thus, our emphasis in this book is to show the relationship between hardware and software and to focus on the concepts that are the basis for current computers. The recent switch from uniprocessor to multicore microprocessors confirmed the soundness of this perspective, given since the first edition. While programmers could ignore the advice and rely on computer architects, compiler writers, and silicon engineers to make their programs run faster or be more energy-efficient without change, that era is over. For programs to run faster, they must become parallel. While the goal of many researchers is to make it possible for programmers to be unaware of the underlying parallel nature of the hardware they are programming, it will take many years to realize this vision. O ur view is that for at least the next decade, most programmers are going to have to understand the hardware/software interface if they want programs to run efficiently on parallel computers. The audience for this book includes those with li le experience in assembly language or logic design who need to understand basic computer organization as well as readers with backgrounds in assembly language and/or logic design who want to learn how to design a computer or understand how a system works and why it performs as it does.

About the Other Book
S ome readers may be familiar with Computer Architecture: A Quantitative Approach popularly known as Hennessy and Pa erson. (This book in , turn is often called Pa erson and Hennessy.) O ur motivation in writing the earlier book was to describe the principles of computer architecture using solid engineering fundamentals and quantitative cost/performance tradeoffs. We used an approach that combined examples and measurements, based on commercial systems, to create realistic design experiences. O ur goal was to demonstrate that computer architecture could be learned using quantitative methodologies instead of a descriptive approach. It was intended for the serious computing professional who wanted a detailed understanding of computers. A majority of the readers for this book do not plan to become computer architects. The performance and energy efficiency of future software systems will be dramatically affected, however, by how well software designers understand the basic hardware techniques at work in a system. Thus, compiler writers, operating system designers, database programmers, and most other software engineers need a firm grounding in the principles presented in this book. Similarly, hardware designers must understand clearly the effects of their work on software applications. Thus, we knew that this book had to be much more than a subset of the material in Computer Architecture, and the material was extensively revised to match the different audience. We were so happy with the result that the subsequent editions of Computer Architecture were revised to remove most of the introductory material; hence, there is much less overlap today than with the first editions of both books.

Changes for the Fifth Edition
We had six major goals for the fifth edition of Computer Organization and Design:demonstrate the importance of understanding hardware with a running example; highlight major themes across the topics using margin icons that are introduced early; update examples to reflect changeover from PC era to PostPC era; spread the material on I/O throughout the book rather than isolating it into a single chapter; update the technical content to reflect changes in the industry since the publication of the fourth edition in 2009; and put appendices and optional sections online instead of including a CD to lower costs and to make this edition viable as an electronic book. Before discussing the goals in detail, let’s look at the table on the next page. It shows the hardware and software paths through the material. Chapters 1, 4, 5, and 6 are found on both paths, no matter what the experience or the focus. Chapter 1 discusses the importance of energy and how it motivates the switch from single core to multicore microprocessors and introduces the eight great ideas in computer architecture. Chapter 2 is likely to be review material for the hardware-oriented, but it is essential reading for the software-oriented, especially for those readers interested in learning more about compilers and object-oriented programming languages. Chapter 3 is for readers interested in constructing a datapath or in learning more about floating-point arithmetic. S ome will skip parts of Chapter 3, either because they don’t need them or because they offer a review. However, we introduce the running example of matrix multiply in this chapter, showing how subword parallels offers a fourfold improvement, so don’t skip sections 3.6 to 3.8. Chapter 4 explains pipelined processors. S ections 4.1, 4.5, and 4.10 give overviews and S ection 4.12 gives the next performance boost for matrix multiply for those with a software focus. Those with a hardware focus, however, will find that this chapter presents core material; they may also, depending on their background, want to read Appendix C on logic design first. The last chapter on multicores, multiprocessors, and clusters, is mostly new content and should be read by everyone. It was significantly reorganized in this edition to make the flow of ideas more natural and to include much more depth on GP U s, warehouse scale computers, and the hardware-software interface of network interface cards that are key to clusters.

The first of the six goals for this firth edition was to demonstrate the importance of understanding modern hardware to get good performance and energy efficiency with a concrete example. As mentioned above, we start with subword parallelism in Chapter 3 to improve matrix multiply by a factor of 4. We double performance in Chapter 4 by unrolling the loop to demonstrate the value of instruction level parallelism. Chapter 5 doubles performance again by optimizing for caches using blocking. Finally, Chapter 6 demonstrates a speedup of 14 from 16 processors by using threadlevel parallelism. All four optimizations in total add just 24 lines of C code to our initial matrix multiply example. The second goal was to help readers separate the forest from the trees by identifying eight great ideas of computer architecture early and then pointing out all the places they occur throughout the rest of the book. We use (hopefully) easy to remember margin icons and highlight the corresponding word in the text to remind readers of these eight themes. There are nearly 100 citations in the book. No chapter has less than seven examples of great ideas, and no idea is cited less than five times. Performance via parallelism, pipelining, and prediction are the three most popular great ideas, followed closely by Moore’s Law. The processor chapter (4) is the one with the most examples, which is not a surprise since it probably received the most a ention from computer architects. The one great idea found in every chapter is performance via parallelism, which is a pleasant

received the most a ention from computer architects. The one great idea found in every chapter is performance via parallelism, which is a pleasant observation given the recent emphasis in parallelism in the field and in editions of this book. The third goal was to recognize the generation change in computing from the P C era to the P ostP C era by this edition with our examples and material. Thus, Chapter 1 dives into the guts of a tablet computer rather than a P C, andChapter 6 describes the computing infrastructure of the cloud. We also feature the ARM, which is the instruction set of choice in the personal mobile devices of the P ostP C era, as well as the x86 instruction set that dominated the PC Era and (so far) dominates cloud computing. The fourth goal was to spread the I/O material throughout the book rather than have it in its own chapter, much as we spread parallelism throughout all the chapters in the fourth edition. Hence, I/O material in this edition can be found inS ections 1.4, 4.9, 5.2, 5.5, 5.11, and 6.9. The thought is that readers (and instructors) are more likely to cover I/O if it’s not segregated to its own chapter. This is a fast-moving field, and, as is always the case for our new editions, an important goal is to update the technical content. The running example is the ARM Cortex A8 and the Intel Core i7, reflecting our P ostP C Era. O ther highlights include an overview the new 64-bit instruction se of ARMv8, a tutorial on GP U s that explains their unique terminology, more depth on the warehouse scale computers that make up the cloud, and a deep dive into 10 Gigabyte Ethernet cards. To keep the main book short and compatible with electronic books, we placed the optional material as online appendices instead of on a companion CD as in prior editions. Finally, we updated all the exercises in the book. While some elements changed, we have preserved useful book elements from prior editions. To make the book work be er as a reference, we still place definitions of new terms in the margins at their first occurrence. The book element called “U nderstanding P rogram Performance” sections helps readers understand the performance of their programs and how to improve it, just as the “Hardware/S oftware Interface” book element helped readers understand the tradeoffs at this interface. “The Big P icture” section remains so that the reader sees the forest despite all the trees. “Check Yourself” sections help readers to confirm their comprehension of the material on the first time through with answers provided at the end of each chapter. This edition still includes the green MIP S reference card, which was inspired by the “Green Card” of the IBM S ystem/360. This card ha been updated and should be a handy reference when writing MIPS assembly language programs.

Changes for the Fifth Edition
We have collected a great deal of material to help instructors teach courses using this book. S olutions to exercises, figures from the book, lecture slides, and other materials are available to adopters from the publisher. Check the publisher’s Web site for more information:

slides, and other materials are available to adopters from the publisher. Check the publisher’s Web site for more information: textbooks.elsevier.com/9780124077263

Concluding Remarks
If you read the following acknowledgments section, you will see that we went to great lengths to correct mistakes. S ince a book goes through many printings, we have the opportunity to make even more corrections. If you uncover any remaining, resilient bugs, please contact the publisher by electronic mail at cod5bugs@mkp.com or by low-tech mail using the address found on the copyright page. This edition is the second break in the long-standing collaboration between Hennessy and Pa erson, which started in 1989. The demands of running one of the world’s great universities meant that P resident Hennessy could no longer make the substantial commitment to create a new edition. The remaining author felt once again like a tightrope walker without a safety net. Hence, the people in the acknowledgments and Berkeley colleagues played an even larger role in shaping the contents of this book. Nevertheless, this time around there is only one author to blame for the new material in what you are about to read.

Acknowledgments for the Fifth Edition

With every edition of this book, we are very fortunate to receive help from many readers, reviewers, and contributors. Each of these people has helped to make this book better. Chapter 6 was so extensively revised that we did a separate review for ideas and contents, and I made changes based on the feedback from every reviewer. I’d like to thank C hristos Kozyrakis of S tanford U niversity for suggesting using the network interface for clusters to demonstrate the hardware-software interface of I/O and for suggestions on organizing the rest of the chapter;Mario Flagsilk of S tanford U niversity for providing details, diagrams, and performance measurements of the NetFP GA NIC; and the following for suggestions on how to improve the chapter: David Kaeli of Northeastern U niversity,Partha Ranganathan of HP Labs,David Wood of the U niversity of Wisconsin, and my Berkeley colleagues Siamak Faridani, Shoaib Kamil, Yunsup Lee, Zhangxi Tan, and Andrew Waterman. S pecial thanks goes to Rimas Avizenis of U C Berkeley, who developed the various versions of matrix multiply and supplied the performance numbers as well. As I worked with his father while I was a graduate student at UCLA, it was a nice symmetry to work with Rimas at UCB. I also wish to thank my longtime collaborator Randy Ka of U C Berkeley, who helped develop the concept of great ideas in computer architecture as part of the extensive revision of an undergraduate class that we did together. I’d like to thank David Kirk, John Nickolls, and their colleagues at NVIDIA (Michael Garland, John Montrym, Doug Voorhies, Lars Nyland Erik Lindholm, Paulius Micikevicius, Massimiliano Fatica, S tuart O berman, and Vasily Volkov) for writing the first in-depth appendix on GP U s. I’d like to express again my appreciation to Jim Larus, recently named Dean of the S chool of Computer and Communications S cience at EP FL, for his willingness in contributing his expertise on assembly language programming, as well as for welcoming readers of this book with regard to using the simulator he developed and maintains. I am also very grateful to Jason Bakos of the U niversity of S outh Carolina, who updated and created new exercises for this edition, working from originals prepared for the fourth edition by Perry Alexander (The U niversity of Kansas);Javier B ruguera (U niversidade de S antiago de Compostela); Matthew Farrens (U niversity of California, Davis);David Kaeli (Northeastern U niversity);Nicole Kaiyan (U niversity of Adelaide); John Oliver (Cal P oly, S an Luis O bispo); Milos Prvulovic (Georgia Tech); and Jichuan C hang, Jacob Leverich, Kevin Lim, and Partha Ranganathan (all from Hewlett-Packard). Additional thanks goes to Jason Bakos for developing the new lecture slides. I am grateful to the many instructors who have answered the publisher’s surveys, reviewed our proposals, and a ended focus groups to analyze and respond to our plans for this edition. They include the following individuals: Focus Groups in 2012: Bruce Barton (S uffolk County Community College), Jeff Braun (Montana Tech), Ed Gehringer (North Carolina S tate), Michael Goldweber (Xavier U niversity), Ed Harcourt (S t. Lawren U niversity), Mark Hill (U niversity of Wisconsin, Madison), Patrick Homer (U niversity of Arizona), Norm Jouppi (HP Labs), Dave Kae (Northeastern U niversity), Christos Kozyrakis (S tanford U niversity), Z achary Kurmas (Grand Valley S tate U niversity), Jae C. O h (S yracu U niversity), Lu Peng (LS U ), Milos P rvulovic (Georgia Tech), Partha Ranganathan (HP Labs), David Wood (U niversity of Wisconsin), Craig Z il (U niversity of Illinois at U rbana-Champaign). S urveys and Reviews: Mahmoud Abou-Nasr (Wayne S tate U niversity), Perry Alexander (Th U niversity of Kansas), Hakan Aydin (George Mason U niversity), Hussein Badr (S tate U niversity of New York at S tony Brook), Mac Baker (Virgin Military Institute), Ron Barnes (George Mason U niversity), Douglas Blough (Georgia Institute of Technology), Kevin Bolding (S ea le Pacifi U niversity), Miodrag Bolic (U niversity of O awa), John Bonomo (Westminster College), Jeff Braun (Montana Tech), Tom Briggs (S hippensbur U niversity), S co Burgess (Humboldt S tate U niversity), Fazli Can (Bilkent U niversity), Warren R. Carithers (Rochester Institute of Technology Bruce Carlton (Mesa Community College), Nicholas Carter (U niversity of Illinois at U rbana-Champaign), Anthony Cocchi (The City U niversity New York), Don Cooley (U tah S tate U niversity), Robert D. Cupper (Allegheny College), Edward W. Davis (North Carolina S tate U niversit Nathaniel J. Davis (Air Force Institute of Technology), Molisa Derk (O klahoma City U niversity), Derek Eager (U niversity of S askatchewan), Erne Ferguson (Northwest Missouri S tate U niversity), Rhonda Kay Gaede (The U niversity of Alabama), Etienne M. Gagnon (U Q AM), Costa Gerou (Christopher Newport U niversity), Paul Gillard (Memorial U niversity of Newfoundland), Michael Goldweber (Xavier U niversity), Georgia Gran (College of S an Mateo), Merrill Hall (The Master’s College), Tyson Hall (S outhern Adventist U niversity), Ed Harcourt (S t. Lawrence U niversit Justin E. Harlow (U niversity of S outh Florida), Paul F. Hemler (Hampden-S ydney College), Martin Herbordt (Boston U niversity), S teve J. Hodg (Cabrillo College), Kenneth Hopkinson (Cornell U niversity), Dalton Hunkins (S t. Bonaventure U niversity), Baback Izadi (S tate U niversity of Ne York—New Pal ), Reza Jafari, Robert W. Johnson (Colorado Technical U niversity), Bharat Joshi (U niversity of North Carolina, Charlo e Nagarajan Kandasamy (Drexel U niversity), Rajiv Kapadia, Ryan Kastner (U niversity of California, S anta Barbara), E.J. Kim (Texas A&M U niversity), Jihong Kim (S eoul National U niversity), Jim Kirk (U nion U niversity), Geoffrey S . Knauth (Lycoming College), Manish M. Kochh (Wayne S tate), S uzan Koknar-Tezel (S aint Joseph’s U niversity), Angkul Kongmunva ana (Columbus S tate U niversity), April Kontostathis (U rsinu College), Christos Kozyrakis (S tanford U niversity), Danny Krizanc (Wesleyan U niversity), Ashok Kumar, S . Kumar (The U niversity of Texas Z achary Kurmas (Grand Valley S tate U niversity), Robert N. Lea (U niversity of Houston), Baoxin (Arizona S tate U niversity), Li Liao (U niversity Li of Delaware), Gary Livingston (U niversity of Massachuse s), Michael Lyle, Douglas W. Lynn (O regon Institute of Technology), Yashwant K Malaiya (Colorado S tate U niversity), Bill Mark (U niversity of Texas at Austin), Ananda Mondal (Claflin U niversity), Alvin Moser (S ea l U niversity), Walid Najjar (U niversity of California, Riverside), Danial J. Neebel (Loras College), John Nestor (Lafaye e College), Jae C. O h (S yracu U niversity), Joe O ldham (Centre College), Timour Paltashev, James Parkerson (U niversity of Arkansas), S haunak Pawagi (S U NY at S tony Brook S teve Pearce, Ted Pedersen (U niversity of Minnesota), Lu Peng (Louisiana S tate U niversity), Gregory D Peterson (The U niversity of Tennessee Milos P rvulovic (Georgia Tech), Partha Ranganathan (HP Labs), Dejan Raskovic (U niversity of Alaska, Fairbanks) Brad Richards (U niversity P uget S ound), Roman Rozanov, Louis Rubinfield (Villanova U niversity), Md Abdus S alam (S outhern U niversity), Augustine S amba (Kent S ta U niversity), Robert S chaefer (Daniel Webster College), Carolyn J. C. S chauble (Colorado S tate U niversity), Keith S chubert (CS U S an Bernardin William L. S chul , Kelly S haw (U niversity of Richmond), S hahram S hirani (McMaster U niversity), S co S igman (Drury U niversity), Bruce S mit David S mith, Jeff W. S mith (U niversity of Georgia, Athens), Mark S motherman (Clemson U niversity), P hilip S nyder (Johns Hopkins U niversity Alex S printson (Texas A&M), Timothy D. S tanley (Brigham Young U niversity), Dean S tevens (Morningside College), Nozar Tabrizi (Ke erin U niversity), Y uval Tamir (U CLA), Alexander Taubin (Boston U niversity), Will Thacker (Winthrop U niversity), Mithuna Tho ethodi (P urdu U niversity), Manghui Tu (S outhern U tah U niversity), Dean Tullsen (U C S an Diego), Rama Viswanathan (Beloit College), Ken Vollmar (Missou S tate U niversity), Guoping Wang (Indiana-P urdue U niversity), Patricia Wenner (Bucknell U niversity), Kent Wilken (U niversity of California Davis), David Wolfe (Gustavus Adolphus College), David Wood (U niversity of Wisconsin, Madison), Ki Hwan Y um (U niversity of Texas, S a Antonio), Mohamed Z ahran (City College of New York), Gerald D. Z arne (Ryerson U niversity), Nian Z hang (S outh Dakota S chool of Mines Technology), Jiling Zhong (Troy University), Huiyang Zhou (The University of Central Florida), Weiyu Zhu (Illinois Wesleyan University). A special thanks also goes to Mark Smotherman for making multiple passes to find technical and writing glitches that significantly improved the quality of this edition. We wish to thank the extended Morgan Kaufmann family for agreeing to publish this book again under the able leadership ofT odd Green and Nate McFadden: I certainly couldn’t have completed the book without them. We also want to extend thanks toLisa Jones, who managed the book production process, and Russell Purdy, who did the cover design. The new cover cleverly connects the P ostP C Era content of this edition to the cover of the first edition. The contributions of the nearly 150 people we mentioned here have helped make this fifth edition what I hope will be our best book yet. Enjoy!

Computer Abstractions and Technology
This chapter explains that although it is difficult to predict exactly what level of cost/performance computers will have in the future, it’s very probable that they will be much be er than they are today. To participate in these advances, computer designers and programmers must understand a wider variety of issues, including factors such as power, reliability, cost of ownership, and scalability. While this chapter focuses on cost, performance, and power, it emphasizes that the best designs will strike the appropriate balance for a given market among all the factors. This chapter also discusses the Post-PC era, with personal mobile devices (PMDs) and tablets largely replacing desktop computers, and Cloud Computing and Warehouse Scale Computers (WSCs) taking over from the traditional server.

Keywords

power wall; uniprocessor; multiprocessor; Intel Core i7; desktop computer; server; supercomputer; datacenter; embedded computer; multicore microprocessor; systems software; operating system; compiler; instruction; assembler; assembly language; machine language; high-level programming language; input device; output device; liquid crystal display; active matrix display; motherboard; integrated circuit; chip; memory; dynamic random access memory; DRAM; dual inline memory module; DIMM; central processor unit; CPU; datapath; control; cache memory; static random access memory; S RAM; abstraction; instruction set architecture; architecture; application binary interface; ABI; implementation; volatile memory; nonvolatile memory; main memory; primary memory; secondary memory; magnetic disk; hard disk; flash memory; local area network; LAN; wide area network; WAN; vacuum tube; transistor; very large scale integrated circuit; VLS I; response time; execution time; throughput; bandwidth; CP U execution time; CP U time; user CP U time; system CP U time; clock cycle; clock period; clock cycles per instruction; CP instruction count; instruction mix; silicon; semiconductor; silicon crystal ingot; wafer; defect; die; yield; workload; benchmark; Amdahl’s law; million instructions per second; MIP S ; cloud computing; mobile computing; cloud; mobile; personal mobile device; P MD; warehouse scale computer; WS C; Software as a Service; SaaS

Civilization advances by extending the number of important operations which we can perform without thinking about them.
Alfred North Whitehead, An Introduction to Mathematics, 1911 1.1 Introduction 1.2 Eight Great Ideas in Computer Architecture 1.3 Below Your Program 1.4 Under the Covers 1.5 Technologies for Building Processors and Memory 1.6 Performance 1.7 The Power Wall 1.8 The Sea Change: The Switch from Uniprocessors to Multiprocessors 1.9 Real Stuff: Benchmarking the Intel Core i7 1.10 Fallacies and Pitfalls 1.11 Concluding Remarks 1.12 Historical Perspective and Further Reading 1.13 Exercises

1.1 Introduction
Welcome to this book! We’re delighted to have this opportunity to convey the excitement of the world of computer systems. This is not a dry and dreary field, where progress is glacial and where new ideas atrophy from neglect. No! Computers are the product of the incredibly vibrant information technology industry, all aspects of which are responsible for almost 10% of the gross national product of the U nited S tates, and whose economy has become dependent in part on the rapid improvements in information technology promised by Moore’s Law. This unusual industry embraces innovation at a breath-taking rate. In the last 30 years, there have been a number of new computers whose introduction appeared to revolutionize the computing industry; these revolutions were cut short only because someone else built an even better computer. This race to innovate has led to unprecedented progress since the inception of electronic computing in the late 1940s. Had the transportation industry kept pace with the computer industry, for example, today we could travel from New York to London in a second for a penny. Take just a moment to contemplate how such an improvement would change society—living in Tahiti while working in S an Francisco, going to Moscow for an evening at the Bolshoi Ballet—and you can appreciate the implications of such a change. Computers have led to a third revolution for civilization, with the information revolution taking its place alongside the agricultural and the industrial revolutions. The resulting multiplication of humankind’s intellectual strength and reach naturally has affected our everyday lives profoundly and changed the ways in which the search for new knowledge is carried out. There is now a new vein of scientific investigation, with computational scientists joining theoretical and experimental scientists in the exploration of new frontiers in astronomy, biology, chemistry, and physics, among others. The computer revolution continues. Each time the cost of computing improves by another factor of 10, the opportunities for computers multiply. Applications that were economically infeasible suddenly become practical. In the recent past, the following applications were “computer science fiction.” ■ Computers in automobiles: Until microprocessors improved dramatically in price and performance in the early 1980s, computer control of cars was ludicrous. Today, computers reduce pollution, improve fuel efficiency via engine controls, and increase safety through blind spot warnings, lane departure warnings, moving object detection, and air bag inflation to protect occupants in a crash. ■ Cell phones: Who would have dreamed that advances in computer systems would lead to more than half of the planet having mobile phones, allowing person-to-person communication to almost anyone anywhere in the world? ■ Human genome project: The cost of computer equipment to map and analyze human DNA sequences was hundreds of millions of dollars. It’s unlikely that anyone would have considered this project had the computer costs been 10 to 100 times higher, as they would have been 15 to 25 years earlier. Moreover, costs continue to drop; you will soon be able to acquire your own genome, allowing medical care to be tailored to you. ■ World Wide Web: Not in existence at the time of the first edition of this book, the web has transformed our society. For many, the web has replaced libraries and newspapers. ■ Search engines: As the content of the web grew in size and in value, finding relevant information became increasingly important. Today, many people rely on search engines for such a large part of their lives that it would be a hardship to go without them. Clearly, advances in this technology now affect almost every aspect of our society. Hardware advances have allowed programmers to create wonderfully useful software, which explains why computers are omnipresent. Today’s science fiction suggests tomorrow’s killer applications: already on their way are glasses that augment reality, the cashless society, and cars that can drive themselves.

Classes of Computing Applications and Their Characteristics
Although a common set of hardware technologies (see Sections 1.4 and 1.5) is used in computers ranging from smart home appliances to cell phones to the largest supercomputers, these different applications have different design requirements and employ the core hardware technologies in different ways. Broadly speaking, computers are used in three different classes of applications. Personal computers (PCs) are possibly the best known form of computing, which readers of this book have likely used extensively. Personal computers emphasize delivery of good performance to single users at low cost and usually execute third-party software. This class of computing drove the evolution of many computing technologies, which is only about 35 years old!

p e r s o n a l

c o m p u t e r

( P C )

A computer designed for use by an individual, usually incorporating a graphics display, a keyboard, and a mouse.

Servers are the modern form of what were once much larger computers, and are usually accessed only via a network. S ervers are oriented to carrying large workloads, which may consist of either single complex applications—usually a scientific or engineering application—or handling many small jobs, such as would occur in building a large web server. These applications are usually based on software from another source (such as a database or simulation system), but are often modified or customized for a particular function. S ervers are built from the same basic technology as desktop computers, but provide for greater computing, storage, and input/output capacity. In general, servers also place a greater emphasis on dependability, since a crash is usually more costly than it would be on a single-user PC.

s e r v e r
A computer used for running larger programs for multiple users, often simultaneously, and typically accessed only via a network.

S ervers span the widest range in cost and capability. At the low end, a server may be li le more than a desktop computer without a screen or keyboard and cost a thousand dollars. These low-end servers are typically used for file storage, small business applications, or simple web serving (see S ection 6.10). At the other extreme are supercomputers, which at the present consist of thousands of processors and many terabytes of memory, and cost tens to hundreds of millions of dollars. S upercomputers are usually used for high-end scientific and engineering calculations, such as weather forecasting, oil exploration, protein structure determination, and other large-scale problems. Although such supercomputers represent the peak of computing capability, they represent a relatively small fraction of the servers and a relatively small fraction of the overall computer market

peak of computing capability, they represent a relatively small fraction of the servers and a relatively small fraction of the overall computer market in terms of total revenue.

s u p e r c o m p u t e r
A class of computers with the highest performance and cost; they are configured as servers and typically cost tens to hundreds of millions of dollars.

t e r a b y t e

( T B )

O riginally 1,099,511,627,776 (240) bytes, although communications and secondary storage systems developers started using the term to mean 1,000,000,000,000 (1012) bytes. To reduce confusion, we now use the term tebibyte (TiB) for 240 bytes, defining terabyte (TB) to mean 1012 bytes. Figure 1.1 shows the full range of decimal and binary values and names.

FIGURE 1.1 The 2X vs. 10Y bytes ambiguity was resolved by adding a binary notation for all the common size terms. In the last column we note how much larger the binary term is than its corresponding decimal term, which is compounded as we head down the chart. These prefixes work for bits as well as bytes, so gigabit (Gb) is 109 bits while gibibits (Gib) is 230 bits.

Embedded computers are the largest class of computers and span the widest range of applications and performance. Embedded computers include the microprocessors found in your car, the computers in a television set, and the networks of processors that control a modern airplane or cargo ship. Embedded computing systems are designed to run one application or one set of related applications that are normally integrated with the hardware and delivered as a single system; thus, despite the large number of embedded computers, most users never really see that they are using a computer!

e m b e d d e d

c o m p u t e r

A computer inside another device used for running one predetermined application or collection of software.

Embedded applications often have unique application requirements that combine a minimum performance with stringent limitations on cost or power. For example, consider a music player: the processor need only be as fast as necessary to handle its limited function, and beyond that, minimizing cost and power are the most important objectives. Despite their low cost, embedded computers often have lower tolerance for failure, since the results can vary from upse ing (when your new television crashes) to devastating (such as might occur when the computer in a plane or cargo ship crashes). In consumer-oriented embedded applications, such as a digital home appliance, dependability is achieved primarily through simplicity—the emphasis is on doing one function as perfectly as possible. In large embedded systems, techniques of redundancy from the server world are often employed. Although this book focuses on general-purpose computers, most concepts apply directly, or with slight modifications, to embedded computers.

E l a b o r a t i o n
Elaborations are short sections used throughout the text to provide more detail on a particular subject that may be of interest. Disinterested readers may skip over an elaboration, since the subsequent material will never depend on the contents of the elaboration. Many embedded processors are designed using processor cores, a version of a processor wri en in a hardware description language, such as Verilog or VHDL (see Chapter 4). The core allows a designer to integrate other application-specific hardware with the processor core for fabrication on a single chip.

Welcome to the PostPC Era
The continuing march of technology brings about generational changes in computer hardware that shake up the entire information technology industry. S ince the last edition of the book we have undergone such a change, as significant in the past as the switch starting 30 years ago to personal computers. Replacing the P C is thepersonal mobile device (PMD). P MDs are ba ery operated with wireless connectivity to the Internet and typically cost hundreds of dollars, and, like P Cs, users can download software (“apps”) to run on them. U nlike P Cs, they no longer have a keyboard and mouse, and are more likely to rely on a touch-sensitive screen or even speech input. Today’s P MD is a smart phone or a tablet

keyboard and mouse, and are more likely to rely on a touch-sensitive screen or even speech input. Today’s P MD is a smart phone or a tablet computer, but tomorrow it may include electronic glasses. Figure 1.2 shows the rapid growth time of tablets and smart phones versus that of P Cs and traditional cell phones.

P e r s o n a l

m o b i l e

d e v i c e s

( P M D

are small wireless devices to connect to the Internet; they rely on ba eries for power, and software is installed by downloading apps. Conventional examples are smart phones and tablets.

FIGURE 1.2 The number manufactured per year of tablets and smart phones, which reflect the PostPC era, versus personal computers and traditional cell phones. Smart phones represent the recent growth in the cell phone industry, and they passed PCs in 2011. Tablets are the fastest growing category, nearly doubling between 2011 and 2012. Recent PCs and traditional cell phone categories are relatively flat or declining.

Taking over from the traditional server is Cloud Computing, which relies upon giant datacenters that are now known as Warehouse Scale Computers (WS Cs). Companies like Amazon and Google build these WS Cs containing 100,000 servers and then let companies rent portions of them so that they can provide software services to P MDs without having to build WS Cs of their own. Indeed, Software as a Service (SaaS) deployed via the cloud is revolutionizing the software industry just as P MDs and WS Cs are revolutionizing the hardware industry. Today’s software developers will often have a portion of their application that runs on the PMD and a portion that runs in the Cloud.

C l o u d

C o m p u t i n g

refers to large collections of servers that provide services over the Internet; some providers rent dynamically varying numbers of servers as a utility.

S o f t w a r e

a s

a

S e r v i c e

( S a a S )

delivers software and data as a service over the Internet, usually via a thin program such as a browser that runs on local client devices, instead of binary code that must be installed, and runs wholly on that device. Examples include web search and social networking.

What You Can Learn in This Book
S uccessful programmers have always been concerned about the performance of their programs, because ge ing results to the user quickly is critical in creating successful software. In the 1960s and 1970s, a primary constraint on computer performance was the size of the computer’s memory. Thus, programmers often followed a simple credo: minimize memory space to make programs fast. In thelast decade, advances in computer design and memory technology have greatly reduced the importance of small memory size in most applications other than those in embedded computing systems. P rogrammers interested in performance now need to understand the issues that have replaced the simple memory model of the 1960s: the parallel nature of processors and the hierarchical nature of memories. Moreover, as we explain in S ection 1.7, today’s programmers need to worry about energy efficiency of their programs running either on the P MD or in the Cloud, which also requires understanding what is below your code. Programmers who seek to build competitive versions of software will therefore need to increase their knowledge of computer organization. We are honored to have the opportunity to explain what’s inside this revolutionary machine, unraveling the software below your program and the hardware under the covers of your computer. By the time you complete this book, we believe you will be able to answer the following questions: ■ How are programs written in a high-level language, such as C or Java, translated into the language of the hardware, and how does the hardware execute the resulting program? Comprehending these concepts forms the basis of understanding the aspects of both the hardware and software that affect program performance. ■ What is the interface between the software and the hardware, and how does software instruct the hardware to perform needed functions? These concepts are vital to understanding how to write many kinds of software. ■ What determines the performance of a program, and how can a programmer improve the performance? As we will see, this depends on the original program, the software translation of that program into the computer’s language, and the effectiveness of the hardware in executing the program. ■ What techniques can be used by hardware designers to improve performance? This book will introduce the basic concepts of modern computer design. The interested reader will find much more material on this topic in our advanced book, Computer Architecture: A Quantitative Approach. ■ What techniques can be used by hardware designers to improve energy efficiency? What can the programmer do to help or hinder energy

■ What techniques can be used by hardware designers to improve energy efficiency? What can the programmer do to help or hinder energy

efficiency?
■ What are the reasons for and the consequences of the recent switch from sequential processing to parallel processing? This book gives the

motivation, describes the current hardware mechanisms to support parallelism, and surveys the new generation of “multicore” microprocessors (see Chapter 6). ■ Since the first commercial computer in 1951, what great ideas did computer architects come up with that lay the foundation of modern computing?

m u l t i c o r e

m i c r o p r o c e s s o r

A microprocessor containing multiple processors (“cores”) in a single integrated circuit.

Without understanding the answers to these questions, improving the performance of your program on a modern computer or evaluating what features might make one computer be er than another for a particular application will be a complex process of trial and error, rather than a scientific procedure driven by insight and analysis. This first chapter lays the foundation for the rest of the book. It introduces the basic ideas and definitions, places the major components of software and hardware in perspective, shows how to evaluate performance and energy, introduces integrated circuits (the technology that fuels the computer revolution), and explains the shift to multicores. In this chapter and later ones, you will likely see many new words, or words that you may have heard but are not sure what they mean. Don’t panic! Yes, there is a lot of special terminology used in describing modern computers, but the terminology actually helps, since it enables us to describe precisely a function or capability. In addition, computer designers (including your authors) love using acronyms, which are easy to understand once you know what the le ers stand for! To help you remember and locate terms, we have included a highlighted definition of every term in the margins the first time it appears in the text. After a short time of working with the terminology, you will be fluent, and your friends will be impressed as you correctly use acronyms such as BIOS, CPU, DIMM, DRAM, PCIe, SATA, and many others.

a c r o n y m
A word constructed by taking the initial le ers of a string of words. For example:RAM is an acronym for Random Access Memory, and CPU is an acronym for Central Processing Unit.

To reinforce how the software and hardware systems used to run a program will affect performance, we use a special section, Understanding Program Performance, throughout the book to summarize important insights into program performance. The first one appears below.

U n d e r s t a n d i n g

P r o g r a m

P e r f o r

The performance of a program depends on a combination of the effectiveness of the algorithms used in the program, the software systems used to create and translate the program into machine instructions, and the effectiveness of the computer in executing those instructions, which may include input/output (I/O ) operations. This table summarizes how the hardware and software affect performance.
Hardware or software component How this component affects performance Where is this topic covered?

Algorithm Programming language, compiler, and architecture Processor and memory system I/O system (hardware and operating system)

Determines both the number of source-level statements and the number of I/O operations executed Determines the number of computer instructions for each sourcelevel statement Determines how fast instructions can be executed Determines how fast I/O operations may be executed

Other books! Chapters 2 and 3 Chapters 4, 5, and 6 Chapters 4, 5, and 6

To demonstrate the impact of the ideas in this book, we improve the performance of a C program that multiplies a matrix times a vector in a sequence of chapters. Each step leverages understanding how the underlying hardware really works in a modern microprocessor to improve performance by a factor of 200! ■ In the category of data level parallelism, in Chapter 3 we use subword parallelism via C intrinsics to increase performance by a factor of 3.8. ■ In the category of instruction level parallelism, in Chapter 4 we use loop unrolling to exploit multiple instruction issue and out-of-order execution hardware to increase performance by another factor of 2.3. ■ In the category of memory hierarchy optimization, in Chapter 5 we use cache blocking to increase performance on large matrices by another factor of 2.5. ■ In the category of thread level parallelism, in Chapter 6 we use parallel for loops in OpenMP to exploit multicore hardware to increase performance by another factor of 14.

C h e c k

Y o u r s e l f

Check Yourself sections are designed to help readers assess whether they comprehend the major concepts introduced in a chapter and understand the implications of those concepts. S ome Check Yourself questions have simple answers; others are for discussion among a

understand the implications of those concepts. S ome Check Yourself questions have simple answers; others are for discussion among a group. Answers to the specific questions can be found at the end of the chapter. Check Yourself questions appear only at the end of a section, making it easy to skip them if you are sure you understand the material. 1. The number of embedded processors sold every year greatly outnumbers the number of PC and even PostPC processors. Can you confirm or deny this insight based on your own experience? Try to count the number of embedded processors in your home. How does it compare with the number of conventional computers in your home? 2. As mentioned earlier, both the software and hardware affect the performance of a program. Can you think of examples where each of the following is the right place to look for a performance bottleneck? ■ The algorithm chosen ■ The programming language or compiler ■ The operating system ■ The processor ■ The I/O system and devices

1.2 Eight Great Ideas in Computer Architecture
We now introduce eight great ideas that computer architects have been invented in the last 60 years of computer design. These ideas are so powerful they have lasted long after the first computer that used them, with newer architects demonstrating their admiration by imitating their predecessors. These great ideas are themes that we will weave through this and subsequent chapters as examples arise. To point out their influence, in this section we introduce icons and highlighted terms that represent the great ideas and we use them to identify the nearly 100 sections of the book that feature use of the great ideas.

Design for Moore’s Law
The one constant for computer designers is rapid change, which is driven largely by Moore’s Law. It states that integrated circuit resources double every 18–24 months. Moore’s Law resulted from a 1965 prediction of such growth in IC capacity made by Gordon Moore, one of the founders of Intel. As computer designs can take years, the resources available per chip can easily double or quadruple between the start and finish of the project. Like a skeet shooter, computer architects must anticipate where the technology will be when the design finishes rather than design for where it starts. We use an “up and to the right” Moore’s Law graph to represent designing for rapid change.

Use Abstraction to Simplify Design
Both computer architects and programmers had to invent techniques to make themselves more productive, for otherwise design time would lengthen as dramatically as resources grew by Moore’s Law. A major productivity technique for hardware and software is to useabstractions to represent the design at different levels of representation; lower-level details are hidden to offer a simpler model at higher levels. We’ll use the abstract painting icon to represent this second great idea.

Make the Common Case Fast
Making the common case fast will tend to enhance performance be er than optimizing the rare case. Ironically, the common case is often simpler than the rare case and hence is often easier to enhance. This common sense advice implies that you know what the common case is, which is only possible with careful experimentation and measurement (see S ection 1.6). We use a sports car as the icon for making the common case fast, as the most common trip has one or two passengers, and it’s surely easier to make a fast sports car than a fast minivan!

Performance via Parallelism
S ince the dawn of computing, computer architects have offered designs that get more performance by performing operations in parallel. We’ll see many examples of parallelism in this book. We use multiple jet engines of a plane as our icon for parallel performance.

Performance via Pipelining

A particular pa ern of parallelism is so prevalent in computer architecture that it merits its own name: pipelining. For example, before fire engines, a “bucket brigade” would respond to a fire, which many cowboy movies show in response to a dastardly act by the villain. The townsfolk form a human chain to carry a water source to fire, as they could much more quickly move buckets up the chain instead of individuals running back and forth. Our pipeline icon is a sequence of pipes, with each section representing one stage of the pipeline.

Performance via Prediction
Following the saying that it can be be er to ask for forgiveness than to ask for permission, the final great idea is prediction. In some cases it can be faster on average to guess and start working rather than wait until you know for sure, assuming that the mechanism to recover from a misprediction is not too expensive and your prediction is relatively accurate. We use the fortune-teller’s crystal ball as our prediction icon.

Hierarchy of Memories
P rogrammers want memory to be fast, large, and cheap, as memory speed often shapes performance, capacity limits the size of problems that can be solved, and the cost of memory today is often the majority of computer cost. Architects have found that they can address these conflicting demands with a hierarchy of memories, with the fastest, smallest, and most expensive memory per bit at the top of the hierarchy and the slowest, largest, and cheapest per bit at the bo om. As we shall see in Chapter 5, caches give the programmer the illusion that main memory is nearly as fast as the top of the hierarchy and nearly as big and cheap as the bo om of the hierarchy. We use a layered triangle icon to represent the memory hierarchy. The shape indicates speed, cost, and size: the closer to the top, the faster and more expensive per bit the memory; the wider the base of the layer, the bigger the memory.

Dependability via Redundancy
Computers not only need to be fast; they need to be dependable. S ince any physical device can fail, we make systemsdependable by including redundant components that can take over when a failure occurs and to help detect failures. We use the tractor-trailer as our icon, since the dual tires on each side of its rear axels allow the truck to continue driving even when one tire fails. (P resumably, the truck driver heads immediately to a repair facility so the flat tire can be fixed, thereby restoring redundancy!)

1.3 Below Your Program
In Paris they simply stared when I spoke to them in French; I never did succeed in making those idiots understand their own language.
Mark Twain, The Innocents Abroad, 1869

A typical application, such as a word processor or a large database system, may consist of millions of lines of code and rely on sophisticated software libraries that implement complex functions in support of the application. As we will see, the hardware in a computer can only execute extremely simple low-level instructions. To go from a complex application to the simple instructions involves several layers of software that interpret or translate high-level operations into simple computer instructions, an example of the great idea of abstraction.

Figure 1.3 shows that these layers of software are organized primarily in a hierarchical fashion, with applications being the outermost ring and a variety of systems software sitting between the hardware and applications software.

s y s t e m s

s o f t w a r e

Software that provides services that are commonly useful, including operating systems, compilers, loaders, and assemblers.

FIGURE 1.3 A simplified view of hardware and software as hierarchical layers, shown as concentric circles with hardware in the center and applications software outermost. In complex applications, there are often multiple layers of application software as well. For example, a database system may run on top of the systems software hosting an application, which in turn runs on top of the database.

There are many types of systems software, but two types of systems software are central to every computer system today: an operating system and a compiler. An operating system interfaces between a user’s program and the hardware and provides a variety of services and supervisory functions. Among the most important functions are: ■ Handling basic input and output operations ■ Allocating storage and memory ■ Providing for protected sharing of the computer among multiple applications using it simultaneously.

o p e r a t i n g

s y s t e m

Supervising program that manages the resources of a computer for the benefit of the programs that run on that computer.

Examples of operating systems in use today are Linux, iOS, and Windows. Compilers perform another vital function: the translation of a program wri en in a high-level language, such as C, C++, Java, or Visual Basic into instructions that the hardware can execute. Given the sophistication of modern programming languages and the simplicity of the instructions executed by the hardware, the translation from a high-level language program to hardware instructions is complex. We give a brief overview of the process here and then go into more depth in Chapter 2 and in Appendix A.

c o m p i l e r
A program that translates high-level language statements into assembly language statements.

From a High-Level Language to the Language of Hardware
To actually speak to electronic hardware, you need to send electrical signals. The easiest signals for computers to understand areon and off, and so the computer alphabet is just two le ers. Just as the 26 le ers of the English alphabet do not limit how much can be wri en, the two le ers of the computer alphabet do not limit what computers can do. The two symbols for these two le ers are the numbers 0 and 1, and we commonly think of the computer language as numbers in base 2, or binary numbers. We refer to each “le er” as a binary digit or bit. Computers are slaves to our commands, which are called instructions. Instructions, which are just collections of bits that the computer understands and obeys, can be thought of as numbers. For example, the bits
1000110010100000

tell one computer to add two numbers. Chapter 2 explains why we use numbers for instructions and data; we don’t want to steal that chapter’s thunder, but using numbers for both instructions and data is a foundation of computing.

b i n a r y

d i g i t

Also called a bit. One of the two numbers in base 2 (0 or 1) that are the components of information.

i n s t r u c t i o n
A command that computer hardware understands and obeys.

The first programmers communicated to computers in binary numbers, but this was so tedious that they quickly invented new notations that were closer to the way humans think. At first, these notations were translated to binary by hand, but this process was still tiresome. U sing the computer to help program the computer, the pioneers invented programs to translate from symbolic notation to binary. The first of these programs was named an assembler. This program translates a symbolic version of an instruction into the binary version. For example, the programmer would write add A,B

and the assembler would translate this notation into
1000110010100000

a s s e m b l e r
A program that translates a symbolic version of instructions into the binary version.

This instruction tells the computer to add the two numbers A and B. The name coined for this symbolic language, still used today, is assembly language. In contrast, the binary language that the machine understands is the machine language.

a s s e m b l y

l a n g u a g e

A symbolic representation of machine instructions.

m a c h i n e

l a n g u a g e

A binary representation of machine instructions.

Although a tremendous improvement, assembly language is still far from the notations a scientist might like to use to simulate fluid flow or that an accountant might use to balance the books. Assembly language requires the programmer to write one line for every instruction that the computer will follow, forcing the programmer to think like the computer. The recognition that a program could be wri en to translate a more powerful language into computer instructions was one of the great breakthroughs in the early days of computing. P rogrammers today owe their productivity—and their sanity—to the creation of high-level programming languages and compilers that translate programs in such languages into instructions. Figure 1.4 shows the relationships among these programs and languages, which are more examples of the power of abstraction.

h i g h - l e v e l

p r o g r a m m i n g

l a n g u

A portable language such as C, C++, Java, or Visual Basic that is composed of words and algebraic notation that can be translated by a compiler into assembly language.

FIGURE 1.4 C program compiled into assembly language and then assembled into binary machine language. Although the translation from high-level language to binary machine language is shown in two steps, some compilers cut out the middleman and produce binary machine language directly. These languages and this program are examined in more detail in Chapter 2.

A compiler enables a programmer to write this high-level language expression:
A+B

The compiler would compile it into this assembly language statement: add A,B

As shown above, the assembler would translate this statement into the binary instructions that tell the computer to add the two numbers A and B. High-level programming languages offer several important benefits. First, they allow the programmer to think in a more natural language, using English words and algebraic notation, resulting in programs that look much more like text than like tables of cryptic symbols (see Figure 1.4). Moreover, they allow languages to be designed according to their intended use. Hence, Fortran was designed for scientific computation, Cobol for business data processing, Lisp for symbol manipulation, and so on. There are also domain-specific languages for even narrower groups of users, such as those interested in simulation of fluids, for example. The second advantage of programming languages is improved programmer productivity. O ne of the few areas of widespread agreement in software development is that it takes less time to develop programs when they are wri en in languages that require fewer lines to express an idea. Conciseness is a clear advantage of high-level languages over assembly language. The final advantage is that programming languages allow programs to be independent of the computer on which they were developed, since compilers and assemblers can translate high-level language programs to the binary instructions of any computer. These three advantages are so strong that today little programming is done in assembly language.

1.4 Under the Covers
Now that we have looked below your program to uncover the underlying software, let’s open the covers of your computer to learn about the underlying hardware. The underlying hardware in any computer performs the same basic functions: inpu ing data, outpu ing data, processing data, and storing data. How these functions are performed is the primary topic of this book, and subsequent chapters deal with different parts of these four tasks. When we come to an important point in this book, a point so important that we hope you will remember it forever, we emphasize it by identifying it as a Big Picture item. We have about a dozen Big P ictures in this book, the first being the five components of a computer that perform the tasks of inputting, outputting, processing, and storing data. Two key components of computers are input devices, such as the microphone, and output devices, such as the speaker. As the names suggest, input feeds the computer, and output is the result of computation sent to the user. S ome devices, such as wireless networks, provide both input and output to the computer.

i n p u t

d e v i c e

A mechanism through which the computer is fed information, such as a microphone.

o u t p u t

d e v i c e

A mechanism that conveys the result of a computation to a user, such as a display, or to another computer.

Chapters 5 and 6 describe input/output (I/O ) devices in more detail, but let’s take an introductory tour through the computer hardware, starting with the external I/O devices.

T h e

B I G

P i c t u r e

The five classic components of a computer are input, output, memory, datapath, and control, with the last two sometimes combined and called the processor. Figure 1.5 shows the standard organization of a computer. This organization is independent of hardware technology: you can place every piece of every computer, past and present, into one of these five categories. To help you keep all this in perspective, the five components of a computer are shown on the front page of each of the following chapters, with the portion of interest to that chapter highlighted.

FIGURE 1.5 The organization of a computer, showing the five classic components. The processor gets instructions and data from memory. Input writes data to memory, and output reads data from memory. Control sends the signals that determine the operations of the datapath, memory, input, and output.

Through the Looking Glass

Through computer displays I have landed an airplane on the deck of a moving carrier, observed a nuclear particle hit a potential well, flown in a rocket at nearly the speed of light and watched a computer reveal its innermost workings.
Ivan Sutherland, the “father” of computer graphics, Scientific American, 1984

The most fascinating I/O device is probably the graphics display. Most personal mobile devices use liquid crystal displays (LCDs) to get a thin, low-power display. The LCD is not the source of light; instead, it controls the transmission of light. A typical LCD includes rod-shaped molecules in a liquid that form a twisting helix that bends light entering the display, from either a light source behind the display or less often from reflected light. The rods straighten out when a current is applied and no longer bend the light. S ince the liquid crystal material is between two screens polarized at 90 degrees, the light cannot pass through unless it is bent. Today, most LCD displays use an active matrix that has a tiny transistor switch at each pixel to precisely control current and make sharper images. A red-green-blue mask associated with each dot on the display determines the intensity of the three-color components in the final image; in a color active matrix LCD, there are three transistor switches at each point.

l i q u i d

c r y s t a l

d i s p l a y

A display technology using a thin layer of liquid polymers that can be used to transmit or block light according to whether a charge is applied.

a c t i v e

m a t r i x

d i s p l a y

A liquid crystal display using a transistor to control the transmission of light at each individual pixel.

The image is composed of a matrix of picture elements, or pixels, which can be represented as a matrix of bits, called a bit map. Depending on the size of the screen and the resolution, the display matrix in a typical tablet ranges in size from 1024×768 to 2048×1536. A color display might use 8 bits for each of the three colors (red, blue, and green), for 24 bits per pixel, permitting millions of different colors to be displayed.

p i x e l
The smallest individual picture element. Screens are composed of hundreds of thousands to millions of pixels, organized in a matrix.

The computer hardware support for graphics consists mainly of a raster refresh buffer, or frame buffer, to store the bit map. The image to be represented onscreen is stored in the frame buffer, and the bit pattern per pixel is read out to the graphics display at the refresh rate. Figure 1.6 shows a frame buffer with a simplified design of just 4 bits per pixel.

FIGURE 1.6 Each coordinate in the frame buffer on the left determines the shade of the corresponding coordinate for the raster scan CRT display on the right. Pixel (X0, Y0) contains the bit pattern 0011, which is a lighter shade on the screen than the bit pattern 1101 in pixel (X1, Y1).

The goal of the bit map is to faithfully represent what is on the screen. The challenges in graphics systems arise because the human eye is very good at detecting even subtle changes on the screen.

Touchscreen

While P Cs also use LCD displays, the tablets and smartphones of the P ostP C era have replaced the keyboard and mouse with touch sensitive displays, which has the wonderful user interface advantage of users pointing directly what they are interested in rather than indirectly with a mouse. While there are a variety of ways to implement a touch screen, many tablets today use capacitive sensing. S ince people are electrical conductors, if an insulator like glass is covered with a transparent conductor, touching distorts the electrostatic field of the screen, which results in a change in capacitance. This technology can allow multiple touches simultaneously, which allows gestures that can lead to attractive user interfaces.

Opening the Box
Figure 1.7 shows the contents of the Apple iPad 2 tablet computer. U nsurprisingly, of the five classic components of the computer, I/O dominates this reading device. The list of I/O devices includes a capacitive multitouch LCD display, front facing camera, rear facing camera, microphone, headphone jack, speakers, accelerometer, gyroscope, Wi-Fi network, and Bluetooth network. The datapath, control, and memory are a tiny portion of the components.

FIGURE 1.7 Components of the Apple iPad 2 A1395. The metal back of the iPad (with the reversed Apple logo in the middle) is in the center. At the top is the capacitive multitouch screen and LCD display. To the far right is the 3.8 V, 25 watt-hour, polymer battery, which consists of three Li-ion cell cases and offers 10 hours of battery life. To the far left is the metal frame that attaches the LCD to the back of the iPad. The small components surrounding the metal back in the center are what we think of as the computer; they are often Lshaped to fit compactly inside the case next to the battery. Figure 1.8 shows a close-up of the L-shaped board to the lower left of the metal case, which is the logic printed circuit board that contains the processor and the memory. The tiny rectangle below the logic board contains a chip that provides wireless communication: Wi-Fi, Bluetooth, and FM tuner: it fits into a small slot in the lower left corner of the logic board. Near the upper left corner of the case is another L-shaped component, which is a front-facing camera assembly that includes the camera, headphone jack, and microphone. Near the right upper corner of the case is the board containing the volume control and silent/screen rotation lock button along with a gyroscope and accelerometer. These last two chips combine to allow the iPad to recognize 6-axis motion. The tiny rectangle next to it is the rear-facing camera. Near the bottom right of the case is the L-shaped speaker assembly. The cable at the bottom is the connector between the logic board and the camera/volume control board. The board between the cable and the speaker assembly is the controller for the capacitive touchscreen. (Courtesy iFixit, www.ifixit.com)

The small rectangles in Figure 1.8 contain the devices that drive our advancing technology, called integrated circuits and nicknamed chips. The A5 package seen in the middle of in Figure 1.8 contains two ARM processors that operate with a clock rate of 1 GHz. The processor is the active part of the computer, following the instructions of a program to the le er. It adds numbers, tests numbers, signals I/O devices to activate, and so on. Occasionally, people call the processor the CPU, for the more bureaucratic-sounding central processor unit.

i n t e g r a t e d

c i r c u i t

Also called a chip. A device combining dozens to millions of transistors.

c e n t r a l

p r o c e s s o r

u n i t

( C P U )

Also called processor. The active part of the computer, which contains the datapath and control and which adds numbers, tests numbers, signals I/O devices to activate, and so on.

FIGURE 1.8 The logic board of Apple iPad 2 in Figure 1.7. The photo highlights five integrated circuits. The large integrated circuit in the middle is the Apple A5 chip, which contains a dual ARM processor cores that run at 1 GHz as well as 512 MB of main memory inside the package. Figure 1.9 shows a photograph of the processor chip inside the A5 package. The similar sized chip to the left is the 32 GB flash memory chip for nonvolatile storage. There is an empty space between the two chips where a second flash chip can be installed to double storage capacity of the iPad. The chips to the right of the A5 include power controller and I/O controller chips. (Courtesy iFixit, www.ifixit.com)

Descending even lower into the hardware, Figure 1.9 reveals details of a microprocessor. The processor logically comprises two main components: datapath and control, the respective brawn and brain of the processor. The datapath performs the arithmetic operations, and control tells the datapath, memory, and I/O devices what to do according to the wishes of the instructions of the program.Chapter 4 explains the datapath and control for a higher-performance design.

d a t a p a t h
The component of the processor that performs arithmetic operations

c o n t r o l
The component of the processor that commands the datapath, memory, and I/O devices according to the instructions of the program.

FIGURE 1.9 The processor integrated circuit inside the A5 package. The size of chip is 12.1 by 10.1 mm, and it was manufactured originally in a 45-nm process (see Section 1.5). It has two identical ARM processors or cores in the middle left of the chip and a PowerVR graphical processor unit (GPU) with four datapaths in the upper left quadrant. To the left and bottom side of the ARM cores are interfaces to main memory (DRAM). (Courtesy Chipworks, www.chipworks.com)

The A5 package in Figure 1.8 also includes two memory chips, each with 2 gibibits of capacity, thereby supplying 512 MiB. The memory is where the programs are kept when they are running; it also contains the data needed by the running programs. The memory is built from DRAM chips. DRAM stands for dynamic random access memory. Multiple DRAMs are used together to contain the instructions and data of a program. In contrast to sequential access memories, such as magnetic tapes, the RAM portion of the term DRAM means that memory accesses take basically the same amount of time no matter what portion of the memory is read.

m e m o r y
The storage area in which programs are kept when they are running and that contains the data needed by the running programs.

d y n a m i c

r a n d o m

a c c e s s

m e m o r y

Memory built as an integrated circuit; it provides random access to any location. Access times are 50 nanoseconds and cost per gigabyte in 2012 was $5 to $10.

Descending into the depths of any component of the hardware reveals insights into the computer. Inside the processor is another type of memory —cache memory. Cache memory consists of a small, fast memory that acts as a buffer for the DRAM memory. (The nontechnical definition of cache is a safe place for hiding things.) Cache is built using a different memory technology, static random access memory (SRAM). S RAM is faster but less dense, and hence more expensive, than DRAM (see Chapter 5). SRAM and DRAM are two layers of the memory hierarchy.

c a c h e

m e m o r y

A small, fast memory that acts as a buffer for a slower, larger memory.

s t a t i c

r a n d o m

a c c e s s

m e m o r y

(

Also memory built as an integrated circuit, but faster and less dense than DRAM.

As mentioned above, one of the great ideas to improve design is abstraction. O ne of the most importantabstractions is the interface between the hardware and the lowest-level software. Because of its importance, it is given a special name: the instruction set architecture, or simply architecture, of a computer. The instruction set architecture includes anything programmers need to know to make a binary machine language program work correctly, including instructions, I/O devices, and so on. Typically, the operating system will encapsulate the details of doing I/O , allocating memory, and other low-level system functions so that application programmers do not need to worry about such details. The combination of the basic instruction set and the operating system interface provided for application programmers is called the application binary interface (ABI).

i n s t r u c t i o n

s e t

a r c h i t e c t u r e

Also called architecture. An abstract interface between the hardware and the lowest-level software that encompasses all the information necessary to write a machine language program that will run correctly, including instructions, registers, memory access, I/O, and so on.

a p p l i c a t i o n

b i n a r y

i n t e r f a c e

(

The user portion of the instruction set plus the operating system interfaces used by application programmers. It defines a standard for binary portability across computers.

An instruction set architecture allows computer designers to talk about functions independently from the hardware that performs them. For example, we can talk about the functions of a digital clock (keeping time, displaying the time, se ing the alarm) independently from the clock hardware (quar crystal, LED displays, plastic bu ons). Computer designers distinguish architecture from animplementation of an architecture along the same lines: an implementation is hardware that obeys the architecture abstraction. These ideas bring us to another Big Picture.

i m p l e m e n t a t i o n
Hardware that obeys the architecture abstraction.

T h e

B I G

P i c t u r e

Both hardware and software consist of hierarchical layers using abstraction, with each lower layer hiding details from the level above. O ne key interface between the levels of abstraction is the instruction set architecture—the interface between the hardware and lowlevel software. This abstract interface enables many implementations of varying cost and performance to run identical software.

A Safe Place for Data
Thus far, we have seen how to input data, compute using the data, and display data. If we were to lose power to the computer, however, everything would be lost because the memory inside the computer is volatile—that is, when it loses power, it forgets. In contrast, a DVD disk doesn’t forget the movie when you turn off the power to the DVD player, and is thus a nonvolatile memory technology.

v o l a t i l e

m e m o r y

Storage, such as DRAM, that retains data only if it is receiving power.

n o n v o l a t i l e

m e m o r y

A form of memory that retains data even in the absence of a power source and that is used to store programs between runs. A DVD disk is nonvolatile.

To distinguish between the volatile memory used to hold data and programs while they are running and this nonvolatile memory used to store data and programs between runs, the term main memory or primary memory is used for the former, and secondary memory for the la er. S econdary memory forms the next lower layer of the memory hierarchy. DRAMs have dominated main memory since 1975, butmagnetic disks dominated secondary memory starting even earlier. Because of their size and form factor, personal Mobile Devices useflash memory, a nonvolatile semiconductor memory, instead of disks. Figure 1.8 shows the chip containing the flash memory of the iPad 2. While slower than DRAM, it is much cheaper than DRAM in addition to being nonvolatile. Although costing more per bit than disks, it is smaller, it comes in much smaller capacities, it is more rugged, and it is more power efficient than disks. Hence, flash memory is the standard secondary memory for P MDs. Alas, unlike disks and DRAM, flash memory bits wear out after 100,000 to 1,000,000 writes. Thus, file systems must keep track of the number of writes and have a strategy to avoid wearing out storage, such as by moving popular data. Chapter 5 describes disks and flash memory in more detail.

m a i n

m e m o r y

Also called primary memory. Memory used to hold programs while they are running; typically consists of DRAM in today’s computers.

s e c o n d a r y

m e m o r y

Nonvolatile memory used to store programs and data between runs; typically consists of flash memory in P MDs and magnetic disks in servers.

m a g n e t i c

d i s k

Also called hard disk. A form of nonvolatile secondary memory composed of rotating pla ers coated with a magnetic recording material. Because they are rotating mechanical devices, access times are about 5 to 20 milliseconds and cost per gigabyte in 2012 was $0.05 to $0.10.

fl a s h

m e m o r y

A nonvolatile semiconductor memory. It is cheaper and slower than DRAM but more expensive per bit and faster than magnetic disks. Access times are about 5 to 50 microseconds and cost per gigabyte in 2012 was $0.75 to $1.00.

Communicating with Other Computers
We’ve explained how we can input, compute, display, and save data, but there is still one missing item found in today’s computers: computer networks. Just as the processor shown in Figure 1.5 is connected to memory and I/O devices, networks interconnect whole computers, allowing computer users to extend the power of computing by including communication. Networks have become so popular that they are the backbone of current computer systems; a new personal mobile device or server without a network interface would be ridiculed. Networked computers have several major advantages: ■ Communication: Information is exchanged between computers at high speeds. ■ Resource sharing: Rather than each computer having its own I/O devices, computers on the network can share I/O devices. ■ Nonlocal access: By connecting computers over long distances, users need not be near the computer they are using. Networks vary in length and performance, with the cost of communication increasing according to both the speed of communication and the distance that information travels. Perhaps the most popular type of network is Ethernet. It can be up to a kilometer long and transfer at up to 40 gigabits per second. Its length and speed make Ethernet useful to connect computers on the same floor of a building;hence, it is an example of what is generically called a local area network. Local area networks are interconnected with switches that can also provide routing services and security. Wide area networks cross continents and are the backbone of the Internet, which supports the web. They are typically based on optical fibers and are leased from telecommunication companies.

l o c a l

a r e a

n e t w o r k

( L A N )

A network designed to carry data within a geographically confined area, typically within a single building.

w i d e

a r e a

n e t w o r k

( W A N )

A network extended over hundreds of kilometers that can span a continent.

Networks have changed the face of computing in the last 30 years, both by becoming much more ubiquitous and by making dramatic increases in performance. In the 1970s, very few individuals had access to electronic mail, the Internet and web did not exist, and physically mailing magnetic tapes was the primary way to transfer large amounts of data between two locations. Local area networks were almost nonexistent, and the few existing wide area networks had limited capacity and restricted access. As networking technology improved, it became much cheaper and had a much higher capacity. For example, the first standardized local area network technology, developed about 30 years ago, was a version of Ethernet that had a maximum capacity (also called bandwidth) of 10 million bits per second, typically shared by tens of, if not a hundred, computers. Today, local area network technology offers a capacity of from 1 to 40 gigabits per second, usually shared by at most a few computers. O ptical communications technology has allowed similar growth in the capacity of wide area networks, from hundreds of kilobits to gigabits and from hundreds of computers connected to a worldwide network to millions of computers connected. This combination of dramatic rise in deployment of networking combined with increases in capacity have made network technology central to the information revolution of the last 30 years. For the last decade another innovation in networking is reshaping the way computers communicate. Wireless technology is widespread, which enabled the P ostP C Era. The ability to make a radio in the same low-cost semiconductor technology (CMO S ) used for memory and microprocessors enabled a significant improvement in price, leading to an explosion in deployment. Currently available wireless technologies, called by the IEEE standard name 802.11, allow for transmission rates from 1 to nearly 100 million bits per second. Wireless technology is quite a bit different from wire-based networks, since all users in an immediate area share the airwaves.

C h e c k

Y o u r s e l f

■ Semiconductor DRAM memory, flash memory, and disk storage differ significantly. For each technology, list its volatility,

approximate relative access time, and approximate relative cost compared to DRAM.

1.5 Technologies for Building Processors and Memory
P rocessors and memory have improved at an incredible rate, because computer designers have long embraced the latest in electronic technology to try to win the race to design a be er computer. Figure 1.10 shows the technologies that have been used over time, with an estimate of the relative performance per unit cost for each technology. S ince this technology shapes what computers will be able to do and how quickly they will evolve, we believe all computer professionals should be familiar with the basics of integrated circuits.

FIGURE 1.10 Relative performance per unit cost of technologies used in computers over time. Source: Computer Museum, Boston, with 2013 extrapolated by the authors. See
Section 1.12.

A transistor is simply an on/off switch controlled by electricity. The integrated circuit (IC) combined dozens to hundreds of transistors into a single chip. When Gordon Moore predicted the continuous doubling of resources, he was predicting the growth rate of the number of transistors per chip. To describe the tremendous increase in the number of transistors from hundreds to millions, the adjective very large scale is added to the term, creating the abbreviation VLSI, for very large-scale integrated circuit.

t r a n s i s t o r
An on/off switch controlled by an electric signal.

v e r y

l a r g e - s c a l e

i n t e g r a t e d

( V

A device containing hundreds of thousands to millions of transistors.

This rate of increasing integration has been remarkably stable. Figure 1.11 shows the growth in DRAM capacity since 1977. For 35 years, the industry has consistently quadrupled capacity every 3 years, resulting in an increase in excess of 16,000 times!

FIGURE 1.11 Growth of capacity per DRAM chip over time. The y-axis is measured in kibibits (210 bits). The DRAM industry quadrupled capacity almost every three years, a 60% increase per year, for 20 years. In recent years, the rate has slowed down and is somewhat closer to doubling every two years to three years.

To understand how manufacture integrated circuits, we start at the beginning. The manufacture of a chip begins withsilicon, a substance found in sand. Because silicon does not conduct electricity well, it is called a semiconductor. With a special chemical process, it is possible to add materials to silicon that allow tiny areas to transform into one of three devices: ■ Excellent conductors of electricity (using either microscopic copper or aluminum wire) ■ Excellent insulators from electricity (like plastic sheathing or glass) ■ Areas that can conduct or insulate under special conditions (as a switch)

s i l i c o n
A natural element that is a semiconductor.

s e m i c o n d u c t o r

A substance that does not conduct electricity well.

Transistors fall in the last category. A VLS I circuit, then, is just billions of combinations of conductors, insulators, and switches manufactured in a single small package. The manufacturing process for integrated circuits is critical to the cost of the chips and hence important to computer designers. Figure 1.12 shows that process. The process starts with a silicon crystal ingot, which looks like a giant sausage. Today, ingots are 8–12 inches in diameter and about 12–24 inches long. An ingot is finely sliced into wafers no more than 0.1 inches thick. These wafers then go through a series of processing steps, during which patterns of chemicals are placed on each wafer, creating the transistors, conductors, and insulators discussed earlier. Today’s integrated circuits contain only one layer of transistors but may have from two to eight levels of metal conductor, separated by layers of insulators.

s i l i c o n

c r y s t a l

i n g o t

A rod composed of a silicon crystal that is between 8 and 12 inches in diameter and about 12 to 24 inches long.

w a f e r
A slice from a silicon ingot no more than 0.1 inches thick, used to create chips.

FIGURE 1.12 The chip manufacturing process. After being sliced from the silicon ingot, blank wafers are put through 20 to 40 steps to create patterned wafers (see Figure 1.13). These patterned wafers are then tested with a wafer tester, and a map of the good parts is made. Then, the wafers are diced into dies (see Figure 1.9). In this figure, one wafer produced 20 dies, of which 17 passed testing. (X means the die is bad.) The yield of good dies in this case was 17/20, or 85%. These good dies are then bonded into packages and tested one more time before shipping the packaged parts to customers. One bad packaged part was found in this final test.

A single microscopic flaw in the wafer itself or in one of the dozens of patterning steps can result in that area of the wafer failing. These defects, as they are called, make it virtually impossible to manufacture a perfect wafer. The simplest way to cope with imperfection is to place many independent components on a single wafer. The pa erned wafer is then chopped up, or diced, into these components, called dies and more informally known as chips. Figure 1.13 shows a photograph of a wafer containing microprocessors before they have been diced; earlier, Figure 1.9 shows an individual microprocessor die.

d e f e c t
A microscopic flaw in a wafer or in patterning steps that can result in the failure of the die containing that defect.

d i e
The individual rectangular sections that are cut from a wafer, more informally known as chips.

FIGURE 1.13 A 12-inch (300 mm) wafer of Intel Core i7 (Courtesy Intel). The number of dies on this 300 mm (12 inch) wafer at 100% yield is 280, each 20.7 by 10.5 mm. The several dozen partially rounded chips at the boundaries of the wafer are useless; they are included because it’s easier to create the masks used to pattern the silicon. This die uses a 32-nanometer technology, which means that the smallest features are approximately 32 nm in size, although they are typically somewhat smaller than the actual feature size, which refers to the size of the transistors as “drawn” versus the final manufactured size.

Dicing enables you to discard only those dies that were unlucky enough to contain the flaws, rather than the whole wafer. This concept is quantified by the yield of a process, which is defined as the percentage of good dies from the total number of dies on the wafer.

y i e l d
The percentage of good dies from the total number of dies on the wafer.

The cost of an integrated circuit rises quickly as the die size increases, due both to the lower yield and the smaller number of dies that fit on a wafer. To reduce the cost, using the next generation process shrinks a large die as it uses smaller sizes for both transistors and wires. This improves the yield and the die count per wafer. A 32-nanometer (nm) process was typical in 2012, which means essentially that the smallest feature size on the die is 32 nm. O nce you’ve found good dies, they are connected to the input/output pins of a package, using a process called bonding. These packaged parts are tested a final time, since mistakes can occur in packaging, and then they are shipped to customers.

E l a b o r a t i o n
The cost of an integrated circuit can be expressed in three simple equations:

image
The first equation is straightforward to derive. The second is an approximation, since it does not subtract the area near the border of the round wafer that cannot accommodate the rectangular dies (see Figure 1.13). The final equation is based on empirical observations of yields at integrated circuit factories, with the exponent related to the number of critical processing steps. Hence, depending on the defect rate and the size of the die and wafer, costs are generally not linear in the die area.

C h e c k

Y o u r s e l f

A key factor in determining the cost of an integrated circuit is volume. Which of the following are reasons why a chip made in high volume should cost less? 1. With high volumes, the manufacturing process can be tuned to a particular design, increasing the yield. 2. It is less work to design a high-volume part than a low-volume part. 3. The masks used to make the chip are expensive, so the cost per chip is lower for higher volumes. 4. Engineering development costs are high and largely independent of volume; thus, the development cost per die is lower with high-volume parts. 5. High-volume parts usually have smaller die sizes than low-volume parts and therefore have higher yield per wafer.

1.6 Performance
Assessing the performance of computers can be quite challenging. The scale and intricacy of modern software systems, together with the wide range of performance improvement techniques employed by hardware designers, have made performance assessment much more difficult. When trying to choose among different computers, performance is an important a ribute. Accurately measuring and comparing different computers is critical to purchasers and therefore to designers. The people selling computers know this as well. O ften, salespeople would like you to see their computer in the best possible light, whether or not this light accurately reflects the needs of the purchaser’s application. Hence, understanding how best to measure performance and the limitations of performance measurements is important in selecting a computer. The rest of this section describes different ways in which performance can be determined; then, we describe the metrics for measuring performance from the viewpoint of both a computer user and a designer. We also look at how these metrics are related and present the classical processor performance equation, which we will use throughout the text.

Defining Performance
When we say one computer has be er performance than another, what do we mean? Although this question might seem simple, an analogy with passenger airplanes shows how subtle the question of performance can be. Figure 1.14 lists some typical passenger airplanes, together with their cruising speed, range, and capacity. If we wanted to know which of the planes in this table had the best performance, we would first need to define performance. For example, considering different measures of performance, we see that the plane with the highest cruising speed was the Concorde (retired from service in 2003), the plane with the longest range is the DC-8, and the plane with the largest capacity is the 747.

FIGURE 1.14 The capacity, range, and speed for a number of commercial airplanes. The last column shows the rate at which the airplane transports passengers, which is the capacity times the cruising speed (ignoring range and takeoff and landing times).

Let’s suppose we define performance in terms of speed. This still leaves two possible definitions. You could define the fastest plane as the one with the highest cruising speed, taking a single passenger from one point to another in the least time. If you were interested in transporting 450 passengers from one point to another, however, the 747 would clearly be the fastest, as the last column of the figure shows. S imilarly, we can define computer performance in several different ways. If you were running a program on two different desktop computers, you’d say that the faster one is the desktop computer that gets the job done first. If you were running a datacenter that had several servers running jobs submi ed by many users, you’d say that the faster computer was the one that completed the most jobs during a day. As an individual computer user, you are interested in reducing response time—the time between the start and completion of a task—also referred to as execution time. Datacenter managers are often interested in increasing throughput or bandwidth—the total amount of work done in a given time. Hence, in most cases, we will need different performance metrics as well as different sets of applications to benchmark personal mobile devices, which are more focused on response time, versus servers, which are more focused on throughput.

r e s p o n s e

t i m e

Also called execution time. The total time required for the computer to complete a task, including disk accesses, memory accesses, I/O activities, operating system overhead, CPU execution time, and so on.

t h r o u g h p u t
Also called bandwidth. Another measure of performance, it is the number of tasks completed per unit time.

T h r o u g h p u t Example

a n d

R e s p o n s e

T i m e

Do the following changes to a computer system increase throughput, decrease response time, or both? 1. Replacing the processor in a computer with a faster version 2. Adding additional processors to a system that uses multiple processors for separate tasks—for example, searching the web

Answer
Decreasing response time almost always improves throughput. Hence, in case 1, both response time and throughput are improved. In case 2, no one task gets work done faster, so only throughput increases. If, however, the demand for processing in the second case was almost as large as the throughput, the system might force requests to queue up. In this case, increasing the throughput could also improve response time, since it would reduce the waiting time in the queue. Thus, in many real computer systems, changing either execution time or throughput often affects the other.

In discussing the performance of computers, we will be primarily concerned with response time for the first few chapters. To maximize performance, we want to minimize response time or execution time for some task. Thus, we can relate performance and execution time for a computer X:

This means that for two computers X and Y, if the performance of X is greater than the performance of Y, we have

That is, the execution time on Y is longer than that on X, if X is faster than Y. In discussing a computer design, we often want to relate the performance of two different computers quantitatively. We will use the phrase “X isn times faster than Y”—or equivalently “X is n times as fast as Y”—to mean

If X is n times as fast as Y, then the execution time on Y is n times as long as it is on X:

R e l a t i v e Example Answer
We know that A is n times as fast as B if

P e r f o r m a n c e

If computer A runs a program in 10 seconds and computer B runs the same program in 15 seconds how much faster is A than B?

image
Thus the performance ratio is

and A is therefore 1.5 times as fast as B.

In the above example, we could also say that computer B is 1.5 times slower than computer A, since

means that

For simplicity, we will normally use the terminology as fast as when we try to compare computers quantitatively. Because performance and execution time are reciprocals, increasing performance requires decreasing execution time. To avoid the potential confusion between the terms increasing and decreasing, we usually say “improve performance” or “improve execution time” when we mean “increase performance” and “decrease execution time.”

Measuring Performance
Time is the measure of computer performance: the computer that performs the same amount of work in the least time is the fastest. P rogram execution time is measured in seconds per program. However, time can be defined in different ways, depending on what we count. The most straightforward definition of time is called wall clock time, response time, or elapsed time. These terms mean the total time to complete a task, including disk accesses, memory accesses, input/output (I/O) activities, operating system overhead—everything. Computers are often shared, however, and a processor may work on several programs simultaneously. In such cases, the system may try to

Computers are often shared, however, and a processor may work on several programs simultaneously. In such cases, the system may try to optimize throughput rather than a empt to minimize the elapsed time for one program. Hence, we often want to distinguish between the elapsed time and the time over which the processor is working on our behalf. CPU execution time or simply CPU time, which recognizes this distinction, is the time the CP U spends computing for this task and does not include time spent waiting for I/O or running other programs. (Remember, though, that the response time experienced by the user will be the elapsed time of the program, not the CP U time.) CP U time can be further divided into the CP U time spent in the program, calleduser CPU time, and the CP U time spent in the operating system performing tasks on behalf of the program, called system CPU time. Differentiating between system and user CP U time is difficult to do accurately, because it is often hard to assign responsibility for operating system activities to one user program rather than another and because of the functionality differences among operating systems.

C P U

e x e c u t i o n

t i m e

Also called CPU time. The actual time the CPU spends computing for a specific task.

u s e r

C P U

t i m e

The CPU time spent in a program itself.

s y s t e m

C P U

t i m e

The CPU time spent in the operating system performing tasks on behalf of the program.

For consistency, we maintain a distinction between performance based on elapsed time and that based on CP U execution time. We will use the term system performance to refer to elapsed time on an unloaded system and CPU performanceto refer to user CP U time. We will focus on CP U performance in this chapter, although our discussions of how to summarize performance can be applied to either elapsed time or CP U time measurements.

U n d e r s t a n d i n g

P r o g r a m

P e r f o r

Different applications are sensitive to different aspects of the performance of a computer system. Many applications, especially those running on servers, depend as much on I/O performance, which, in turn, relies on both hardware and software. Total elapsed time measured by a wall clock is the measurement of interest. In some application environments, the user may care about throughput, response time, or a complex combination of the two (e.g., maximum throughput with a worst-case response time). To improve the performance of a program, one must have a clear definition of what performance metric ma ers and then proceed to look for performance bo lenecks by measuring program execution and looking for the likely bo lenecks. In the following chapters, we will describe how to search for bottlenecks and improve performance in various parts of the system.

Although as computer users we care about time, when we examine the details of a computer it’s convenient to think about performance in other metrics. In particular, computer designers may want to think about a computer by using a measure that relates to how fast the hardware can perform basic functions. Almost all computers are constructed using a clock that determines when events take place in the hardware. These discrete time intervals are called clock cycles (or ticks, clock ticks, clock periods, clocks, cycles). Designers refer to the length of a clock period both as the time for a complete clock cycle (e.g., 250 picoseconds, or 250 ps) and as the clock rate (e.g., 4 gigaher , or 4 GHz), which is the inverse of the clock period. In the next subsection, we will formalize the relationship between the clock cycles of the hardware designer and the seconds of the computer user.

c l o c k

c y c l e

Also called tick, clock tick, clock period, clock, or cycle. The time for one clock period, usually of the processor clock, which runs at a constant rate.

c l o c k

p e r i o d

The length of each clock cycle.

C h e c k

Y o u r s e l f

1. Suppose we know that an application that uses both personal mobile devices and the Cloud is limited by network performance. For the following changes, state whether only the throughput improves, both response time and throughput improve, or neither improves. a. An extra network channel is added between the PMD and the Cloud, increasing the total network throughput and reducing the delay to obtain network access (since there are now two channels). b. The networking software is improved, thereby reducing the network communication delay, but not increasing throughput. c. More memory is added to the computer. 2. Computer C’s performance is 4 times as fast as the performance of computer B, which runs a given application in 28 seconds. How long will computer C take to run that application?

CPU Performance and Its Factors
U sers and designers often examine performance using different metrics. If we could relate these different metrics, we could determine the effect of a design change on the performance as experienced by the user. S ince we are confining ourselves to CP U performance at this point, the bo om-line performance measure is CPU execution time. A simple formula relates the most basic metrics (clock cycles and clock cycle time) to CPU time:

Alternatively, because clock rate and clock cycle time are inverses,

This formula makes it clear that the hardware designer can improve performance by reducing the number of clock cycles required for a program or the length of the clock cycle. As we will see in later chapters, the designer often faces a trade-off between the number of clock cycles needed for a program and the length of each cycle. Many techniques that decrease the number of clock cycles may also increase the clock cycle time.

I m p r o v i n g Example

P e r f o r m a n c e

O ur favorite program runs in 10 seconds on computer A, which has a 2 GHz clock. We are trying to help a computer designer build a computer, B, which will run this program in 6 seconds. The designer has determined that a substantial increase in the clock rate is possible, but this increase will affect the rest of the CP U design, causing computer B to require 1.2 times as many clock cycles as computer A for this program. What clock rate should we tell the designer to target?

Answer
Let’s first find the number of clock cycles required for the program on A:

image
CPU time for B can be found using this equation:

image
To run the program in 6 seconds, B must have twice the clock rate of A.

Instruction Performance

The performance equations above did not include any reference to the number of instructions needed for the program. However, since the compiler clearly generated instructions to execute, and the computer had to execute the instructions to run the program, the execution time must depend on the number of instructions in a program. O ne way to think about execution time is that it equals the number of instructions executed multiplied by the average time per instruction. Therefore, the number of clock cycles required for a program can be written as

The term clock cycles per instruction, which is the average number of clock cycles each instruction takes to execute, is often abbreviated as CPI. S ince different instructions may take different amounts of time depending on what they do, CP I is an average of all the instructions executed in the program. CP I provides one way of comparing two different implementations of the same instruction set architecture, since the number of instructions executed for a program will, of course, be the same.

c l o c k

c y c l e s

p e r

i n s t r u c t i o n

(

Average number of clock cycles per instruction for a program or program fragment.

U s i n g Example

t h e

P e r f o r m a n c e

E q u a t i o

S uppose we have two implementations of the same instruction set architecture. Computer A has a clock cycle time of 250 ps and a CP I of 2.0 for some program, and computer B has a clock cycle time of 500 ps and a CP I of 1.2 for the same program. Which computer is faster for this program and by how much?

Answer
We know that each computer executes the same number of instructions for the program; let’s call this number I. First, find the number of processor clock cycles for each computer:

Now we can compute the CPU time for each computer:

image
Likewise, for B:

image
Clearly, computer A is faster. The amount faster is given by the ratio of the execution times:

image
We can conclude that computer A is 1.2 times as fast as computer B for this program.

The Classic CPU Performance Equation
We can now write this basic performance equation in terms of instruction count (the number of instructions executed by the program), CP I, and clock cycle time:

or, since the clock rate is the inverse of clock cycle time:

i n s t r u c t i o n
The number of instructions executed by the program.

c o u n t

These formulas are particularly useful because they separate the three key factors that affect performance. We can use these formulas to compare two different implementations or to evaluate a design alternative if we know its impact on these three parameters.

C o m p a r i n g Example Image

C o d e

S e g m e n t s

A compiler designer is trying to decide between two code sequences for a particular computer. The hardware designers have supplied the following facts:

For a particular high-level language statement, the compiler writer is considering two code sequences that require the following instruction counts:

Image
Which code sequence executes the most instructions? Which will be faster? What is the CPI for each sequence?

Answer
S equence 1 executes 2+1+2=5 instructions. S equence 2 executes 4+1+1=6 instructions. Therefore, sequence 1 executes fewer instructions. We can use the equation for CP U clock cycles based on instruction count and CP I to find the total number of clock cycles for each sequence:

This yields

image image
So code sequence 2 is faster, even though it executes one extra instruction. Since code sequence 2 takes fewer overall clock cycles but has more instructions, it must have a lower CPI. The CPI values can be computed by

T h e

B I G

P i c t u r e

Figure 1.15 shows the basic measurements at different levels in the computer and what is being measured in each case. We can see how these factors are combined to yield execution time measured in seconds per program:

image
Always bear in mind that the only complete and reliable measure of computer performance is time. For example, changing the instruction set to lower the instruction count may lead to an organization with a slower clock cycle time or higher CP I that offsets the improvement in instruction count. S imilarly, because CP I depends on type of instructions executed, the code that executes the fewest number of instructions may not be the fastest.

FIGURE 1.15 The basic components of performance and how each is measured.

How can we determine the value of these factors in the performance equation? We can measure the CP U execution time by running the program, and the clock cycle time is usually published as part of the documentation for a computer. The instruction count and CP I can be more difficult to obtain. Of course, if we know the clock rate and CPU execution time, we need only one of the instruction count or the CPI to determine the other. We can measure the instruction count by using software tools that profile the execution or by using a simulator of the architecture. Alternatively, we can use hardware counters, which are included in most processors, to record a variety of measurements, including the number of instructions executed, the average CP I, and often, the sources of performance loss. S ince the instruction count depends on the architecture, but not on the exact implementation, we can measure the instruction count without knowing all the details of the implementation. The CP I, however, depends on a wide variety of design details in the computer, including both the memory system and the processor structure (as we will see in Chapter 4 and Chapter 5), as well as on the mix of instruction types executed in an application. Thus, CP I varies by application, as well as among implementations with the same instruction set. The above example shows the danger of using only one factor (instruction count) to assess performance. When comparing two computers, you must look at all three components, which combine to form execution time. If some of the factors are identical, like the clock rate in the above example, performance can be determined by comparing all the nonidentical factors. S ince CP I varies byinstruction mix, both instruction count and CP I must be compared, even if clock rates are identical. S everal exercises at the end of this chapter ask you to evaluate a series of computer and compiler enhancements that affect clock rate, CP I, and instruction count. In Section 1.10, we’ll examine a common performance measurement that does not incorporate all the terms and can thus be misleading.

i n s t r u c t i o n

m i x

A measure of the dynamic frequency of instructions across one or many programs.

U n d e r s t a n d i n g
Hardware or software component

P r o g r a m
Affects what?

P e r f o r
How?

The performance of a program depends on the algorithm, the language, the compiler, the architecture, and the actual hardware. The following table summarizes how these components affect the factors in the CPU performance equation.
Algorithm Instruction count, possibly CPI The algorithm determines the number of source program instructions executed and hence the number of processor instructions executed. The algorithm may also affect the CPI, by favoring slower or faster instructions. For example, if the algorithm uses more divides, it will tend to have a higher CPI. The programming language certainly affects the instruction count, since statements in the language are translated to processor instructions, which determine instruction count. The language may also affect the CPI because of its features; for example, a language with heavy support for data abstraction (e.g., Java) will require indirect calls, which will use higher CPI instructions. The efficiency of the compiler affects both the instruction count and average cycles per instruction, since the compiler determines the translation of the source language instructions into computer instructions. The compiler’s role can be very complex and affect the CPI in complex ways. The instruction set architecture affects all three aspects of CPU performance, since it affects the instructions needed for a function, the cost in cycles of each instruction, and the overall clock rate of the processor.

Programming language

Instruction count, CPI

Compiler

Instruction count, CPI

Instruction set architecture

Instruction count, clock rate, CPI

E l a b o r a t i o n
Although you might expect that the minimum CP I is 1.0, as we’ll see inChapter 4, some processors fetch and execute multiple instructions per clock cycle. To reflect that approach, some designers invert CP I to talk aboutIPC, or instructions per clock cycle. If a processor executes on average 2 instructions per clock cycle, then it has an IPC of 2 and hence a CPI of 0.5.

E l a b o r a t i o n
Although clock cycle time has traditionally been fixed, to save energy or temporarily boost performance, today’s processors can vary their clock rates, so we would need to use the average clock rate for a program. For example, the Intel Core i7 will temporarily increase clock rate by about 10% until the chip gets too warm. Intel calls this Turbo mode.

C h e c k

Y o u r s e l f

A given application wri en in Java runs 15 seconds on a desktop processor. A new Java compiler is released that requires only 0.6 as many instructions as the old compiler. U nfortunately, it increases the CP I by 1.1. How fast can we expect the application to run using this new compiler? Pick the right answer from the three choices below: a. b. 15×0.6×1.1=9.9 sec c.

1.7 The Power Wall
Figure 1.16 shows the increase in clock rate and power of eight generations of Intel microprocessors over 30 years. Both clock rate and power increased rapidly for decades, and then fla ened off recently. The reason they grew together is that they are correlated, and the reason for their recent slowing is that we have run into the practical power limit for cooling commodity microprocessors.

FIGURE 1.16 Clock rate and Power for Intel x86 microprocessors over eight generations and 25 years. The Pentium 4 made a dramatic jump in clock rate and power but less so in performance. The Prescott thermal problems led to the abandonment of the Pentium 4 line. The Core 2 line reverts to a simpler pipeline with lower clock rates and multiple processors per chip. The Core i5 pipelines follow in its footsteps.

Although power provides a limit to what we can cool, in the P ostP C Era the really valuable resource is energy. Ba ery life can trump performance in the personal mobile device, and the architects of warehouse scale computers try to reduce the costs of powering and cooling 100,000 servers as the costs are high at this scale. Just as measuring time in seconds is a safer measure of program performance than a rate like MIP S (see Section 1.10), the energy metric joules is a better measure than a power rate like watts, which is just joules/second. The dominant technology for integrated circuits is called CMO S (complementary metal oxide semiconductor). For CMO S , the primary source of energy consumption is so-called dynamic energy—that is, energy that is consumed when transistors switch states from 0 to 1 and vice versa. The dynamic energy depends on the capacitive loading of each transistor and the voltage applied:

This equation is the energy of a pulse during the logic transition of 0 → 1 → 0 or 1 → 0 → 1. The energy of a single transition is then

The power required per transistor is just the product of energy of a transition and the frequency of transitions:

Frequency switched is a function of the clock rate. The capacitive load per transistor is a function of both the number of transistors connected to an output (called the fanout) and the technology, which determines the capacitance of both wires and transistors. With regard to Figure 1.16, how could clock rates grow by a factor of 1000 while power grew by only a factor of 30? Energy and thus power can be reduced by lowering the voltage, which occurred with each new generation of technology, and power is a function of the voltage squared. Typically, the voltage was reduced about 15% per generation. In 20 years, voltages have gone from 5 V to 1 V, which is why the increase in power is only 30 times.

R e l a t i v e Example

P o w e r

S uppose we developed a new, simpler processor that has 85% of the capacitive load of the more complex older processor. Further, assume that it has adjustable voltage so that it can reduce voltage 15% compared to processor B, which results in a 15% shrink in frequency. What is the impact on dynamic power?

Answer image
Thus the power ratio is

Hence, the new processor uses about half the power of the old processor.

The problem today is that further lowering of the voltage appears to make the transistors too leaky, like water faucets that cannot be completely shut off. Even today about 40% of the power consumption in server chips is due to leakage. If transistors started leaking more, the whole process could become unwieldy. To try to address the power problem, designers have already a ached large devices to increase cooling, and they turn off parts of the chip that are not used in a given clock cycle. Although there are many more expensive ways to cool chips and thereby raise their power to, say, 300 wa s, these techniques are generally too expensive for personal computers and even servers, not to mention personal mobile devices. S ince computer designers slammed into a power wall, they needed a new way forward. They chose a different way from the way they designed microprocessors for their first 30 years.

E l a b o r a t i o n
Although dynamic energy is the primary source of energy consumption in CMO S , static energy consumption occurs because of leakage current that flows even when a transistor is off. In servers, leakage is typically responsible for 40% of the energy consumption. Thus, increasing the number of transistors increases power dissipation, even if the transistors are always off. A variety of design techniques and technology innovations are being deployed to control leakage, but it’s hard to lower voltage further.

E l a b o r a t i o n
P ower is a challenge for integrated circuits for two reasons. First, power must be brought in and distributed around the chip; modern microprocessors use hundreds of pins just for power and ground! S imilarly, multiple levels of chip interconnect are used solely for power and ground distribution to portions of the chip. S econd, power is dissipated as heat and must be removed. S erver chips can burn more than 100 wa s, and cooling the chip and the surrounding system is a major expense in Warehouse S cale Computers (see Chapter 6).

1.8 The Sea Change: The Switch from Uniprocessors to Multiprocessors
Up to now, most software has been like music written for a solo performer; with the current generation of chips we’re getting a little experience with duets and quartets and other small ensembles; but scoring a work for large orchestra and chorus is a different kind of challenge.
Brian Hayes, Computing in a Parallel Universe, 2007.

The power limit has forced a dramatic change in the design of microprocessors. Figure 1.17 shows the improvement in response time of programs for desktop microprocessors over time. Since 2002, the rate has slowed from a factor of 1.5 per year to a factor of 1.2 per year.

FIGURE 1.17 Growth in processor performance since the mid-1980s. This chart plots performance relative to the VAX 11/780 as measured by the SPECint benchmarks (see Section 1.10). Prior to the mid-1980s, processor performance growth was largely technology-driven and averaged about 25% per year. The increase in growth to about 52% since then is attributable to more advanced architectural and organizational ideas. The higher annual performance improvement of 52% since the mid-1980s meant performance was about a factor of seven higher in 2002 than it would have been had it stayed at 25%. Since 2002, the limits of power, available instruction-level parallelism, and long memory latency have slowed uniprocessor performance recently, to about 22% per year.

Rather than continuing to decrease the response time of a single program running on the single processor, as of 2006 all desktop and server companies are shipping microprocessors with multiple processors per chip, where the benefit is often more on throughput than on response time. To reduce confusion between the words processor and microprocessor, companies refer to processors as “cores,” and such microprocessors are generically called multicore microprocessors. Hence, a “quadcore” microprocessor is a chip that contains four processors or four cores. In the past, programmers could rely on innovations in hardware, architecture, and compilers to double performance of their programs every 18 months without having to change a line of code. Today, for programmers to get significant improvement in response time, they need to rewrite their program s to take advantage of multiple processors. Moreover, to get the historic benefit of running faster on new microprocessors, programmers will have to continue to improve performance of their code as the number of cores increases. To reinforce how the software and hardware systems work hand in hand, we use a special section, Hardware/Software Interface throughout the , book, with the first one appearing below. These elements summarize important insights at this critical interface.

H a r d w a r e / S o f t w a r e

I n t e r f a c e

Parallelism has always been critical to performance in computing, but it was often hidden. Chapter 4 will explain pipelining, an elegant technique that runs programs faster by overlapping the execution of instructions. This is one example of instruction-level parallelism, where the parallel nature of the hardware is abstracted away so the programmer and compiler can think of the hardware as executing instructions sequentially.

Forcing programmers to be aware of the parallel hardware and to explicitly rewrite their programs to be parallel had been the “third rail” of computer architecture, for companies in the past that depended on such a change in behavior failed (see Section 6.15). From this historical perspective, it’s startling that the whole IT industry has bet its future that programmers will finally successfully switch to explicitly parallel programming.

Why has it been so hard for programmers to write explicitly parallel programs? The first reason is that parallel programming is by definition performance programming, which increases the difficulty of programming. Not only does the program need to be correct, solve an important problem, and provide a useful interface to the people or other programs that invoke it, the program must also be fast. O therwise, if you don’t need performance, just write a sequential program. The second reason is that fast for parallel hardware means that the programmer must divide an application so that each processor has roughly the same amount to do at the same time, and that the overhead of scheduling and coordination doesn’t fri er away the potential performance benefits of parallelism. As an analogy, suppose the task was to write a newspaper story. Eight reporters working on the same story could potentially write a story eight times faster. To achieve this increased speed, one would need to break up the task so that each reporter had something to do at the same time. Thus, we must schedule the sub-tasks. If anything went wrong and just one reporter took longer than the seven others did, then the benefits of having eight writers would be diminished. Thus, we must balance the load evenly to get the desired speedup. Another danger would be if reporters had to spend a lot of time talking to each other to write their sections. You would also fall short if one part of the story, such as the conclusion, couldn’t be wri en until all of the other parts were completed. Thus, care must be taken to reduce communication and synchronization overhead. For both this analogy and parallel programming, the challenges include scheduling, load balancing, time for synchronization, and overhead for communication between the parties. As you might guess, the challenge is stiffer with more reporters for a newspaper story and more processors for parallel programming. To reflect this sea change in the industry, the next five chapters in this edition of the book each have a section on the implications of the parallel revolution to that chapter: ■ Chapter 2, Section 2.11: Parallelism and Instructions: Synchronization. Usually independent parallel tasks need to coordinate at times, such as to say when they have completed their work. This chapter explains the instructions used by multicore processors to synchronize tasks. ■ Chapter 3, Section 3.6: Parallelism and Computer Arithmetic: Subword Parallelism. Perhaps the simplest form of parallelism to build involves computing on elements in parallel, such as when multiplying two vectors. Subword parallelism takes advantage of the resources supplied by Moore’s Law to provider wider arithmetic units that can operate on many operands simultaneously.

■ Chapter 4, Section 4.10: Parallelism via Instructions. Given the difficulty of explicitly parallel programming, tremendous effort was invested in

the 1990s in having the hardware and the compiler uncover implicit parallelism, initially vice pipelining. This chapter describes some of these aggressive techniques, including fetching and executing multiple instructions simultaneously and guessing on the outcomes of decisions, and executing instructions speculatively using prediction.

■ Chapter 5, Section 5.10: Parallelism and Memory Hierarchies: Cache Coherence. One way to lower the cost of communication is to have all

processors use the same address space, so that any processor can read or write any data. Given that all processors today use caches to keep a temporary copy of the data in faster memory near the processor, it’s easy to imagine that parallel programming would be even more difficult if the caches associated with each processor had inconsistent values of the shared data. This chapter describes the mechanisms that keep the data in all caches consistent.

■ Chapter 5, Section 5.11: Parallelism and Memory Hierarchy: Redundant Arrays of Inexpensive Disks. This section describes how using many disks

in conjunction can offer much higher throughput, which was the original inspiration of Redundant Arrays of Inexpensive Disks (RAID). The real popularity of RAID proved to be to the much greater dependability offered by including a modest number of redundant disks. The section explains the differences in performance, cost, and dependability between the different RAID levels. In addition to these sections, there is a full chapter on parallel processing. Chapter 6 goes into more detail on the challenges of parallel programming; presents the two contrasting approaches to communication of shared addressing and explicit message passing; describes a restricted model of parallelism that is easier to program; discusses the difficulty of benchmarking parallel processors; introduces a new simple performance model for multicore microprocessors; and, finally, describes and evaluates four examples of multicore microprocessors using this model. As mentioned above, Chapters 3 to 6 use matrix vector multiply as a running example to show how each type of parallelism can significantly increase performance. Appendix C describes an increasingly popular hardware component that is included with desktop computers, the graphics processing unit (GP U ). Invented to accelerate graphics, GP U s are becoming programming platforms in their own right. As you might expect, given these times, GPUs rely on parallelism.

Appendix C describes the NVIDIA GPU and highlights parts of its parallel programming environment.

1.9 Real Stuff: Benchmarking the Intel Core i7
I thought [computers] would be a universally applicable idea, like a book is. But I didn’t think it would develop as fast as it did, because I didn’t envision we’d be able to get as many parts on a chip as we finally got. The transistor came along unexpectedly. It all happened much faster than we expected.
J. Presper Eckert, coinventor of ENIAC, speaking in 1991

Each chapter has a section entitled “Real S tuff” that ties the concepts in the book with a computer you may use every day. These sections cover the technology underlying modern computers. For this first “Real S tuff” section, we look at how integrated circuits are manufactured and how performance and power are measured, with the Intel Core i7 as the example.

SPEC CPU Benchmark
A computer user who runs the same programs day in and day out would be the perfect candidate to evaluate a new computer. The set of programs run would form a workload. To evaluate two computer systems, a user would simply compare the execution time of the workload on the two computers. Most users, however, are not in this situation. Instead, they must rely on other methods that measure the performance of a candidate computer, hoping that the methods will reflect how well the computer will perform with the user’s workload. This alternative is usually followed by evaluating the computer using a set of benchmarks—programs specifically chosen to measure performance. The benchmarks form a workload that the user hopes will predict the performance of the actual workload. As we noted above, to make the common case fast, you first need to know accurately which case is common, so benchmarks play a critical role in computer architecture.

w o r k l o a d
A set of programs run on a computer that is either the actual collection of applications run by a user or constructed from real programs to approximate such a mix. A typical workload specifies both the programs and the relative frequencies.

b e n c h m a r k
A program selected for use in comparing computer performance.

SPEC (System Performance Evaluation Cooperative) is an effort funded and supported by a number of computer vendors to create standard sets of benchmarks for modern computer systems. In 1989, S P EC originally created a benchmark focusing on processor performance (now called set S P EC89), which has evolved through five generations. The latest is S P EC CP U 2006, which consists of a set of 12 integer benchmarks (CINT200 and 17 floating-point benchmarks (CFP 2006). The integer benchmarks vary from part of a C compiler to a chess program to a quantum computer simulation. The floating-point benchmarks include structured grid codes for finite element modeling, particle method codes for molecular dynamics, and sparse linear algebra codes for fluid dynamics. Figure 1.18 describes the S P EC integer benchmarks and their execution time on the Intel Core i7 and shows the factors that explain execution time: instruction count, CPI, and clock cycle time. Note that CPI varies by more than a factor of 5.

FIGURE 1.18 SPECINTC2006 benchmarks running on a 2.66 GHz Intel Core i7 920. As the equation on page 35 explains, execution time is the product of the three factors in this table: instruction count in billions, clocks per instruction (CPI), and clock cycle time in nanoseconds. SPECratio is simply the reference time, which is supplied by SPEC, divided by the measured execution time. The single number quoted as SPECINTC2006 is the geometric mean of the SPECratios.

To simplify the marketing of computers, S P EC decided to report a single number to summarize all 12 integer benchmarks. Dividing the execution time of a reference processor by the execution time of the measured computer normalizes the execution time measurements; this normalization yields a measure, called the SPECratio, which has the advantage that bigger numeric results indicate faster performance. That is, the S P ECratio is the inverse of execution time. A CINT2006 or CFP2006 summary measurement is obtained by taking the geometric mean of the SPECratios.

E l a b o r a t i o n
When comparing two computers using S P ECratios, use the geometric mean so that it gives the same relative answer no ma er what computer is used to normalize the results. If we averaged the normalized execution time values with an arithmetic mean, the results would vary depending on the computer we choose as the reference. The formula for the geometric mean is

where Execution time ratioi is the execution time, normalized to the reference computer, for the ith program of a total of n in the workload, and

SPEC Power Benchmark
Given the increasing importance of energy and power, S P EC added a benchmark to measure power. It reports power consumption of servers at different workload levels, divided into 10% increments, over a period of time. Figure 1.19 shows the results for a server using Intel Nehalem processors similar to the above.

FIGURE 1.19 SPECpower_ssj2008 running on a dual socket 2.66 GHz Intel Xeon X5650 with 16 GB of DRAM and one 100 GB SSD disk.

S P ECpower started with another S P EC benchmark for Java business applications (S P ECJBB2005), which exercises the processors, caches, an main memory as well as the Java virtual machine, compiler, garbage collector, and pieces of the operating system. Performance is measured in throughput, and the units are business operations per second. O nce again, to simplify the marketing of computers, S P EC these numbers down boils to a single number, called “overall ssj_ops per watt.” The formula for this single summarizing metric is

where ssj_opsi is performance at each 10% increment and poweri is power consumed at each performance level.

1.10 Fallacies and Pitfalls
Science must begin with myths, and the criticism of myths.
Sir Karl Popper, The Philosophy of Science, 1957

The purpose of a section on fallacies and pitfalls, which will be found in every chapter, is to explain some commonly held misconceptions that you might encounter. We call them fallacies. When discussing a fallacy, we try to give a counterexample. We also discusspitfalls, or easily made mistakes. O ften pitfalls are generalizations of principles that are true in a limited context. The purpose of these sections is to help you avoid making these mistakes in the computers you may design or use. Cost/performance fallacies and pitfalls have ensnared many a computer architect, including us. Accordingly, this section suffers no shortage of relevant examples. We start with a pitfall that traps many designers and reveals an important relationship in computer design.

Pitfall: Expecting the improvement of one aspect of a computer to increase overall performance by an amount proportional to the size of the improvement.
The great idea of making the common case fast has a demoralizing corollary that has plagued designers of both hardware and software. It reminds us that the opportunity for improvement is affected by how much time the event consumes.

A simple design problem illustrates it well. S uppose a program runs in 100 seconds on a computer, with multiply operations responsible for 80 seconds of this time. How much do I have to improve the speed of multiplication if I want my program to run five times faster? The execution time of the program after making the improvement is given by the following simple equation known as Amdahl’s Law:

For this problem:

A m d a h l ’ s

L a w

A rule stating that the performance enhancement possible with a given improvement is limited by the amount that the improved feature is used. It is a quantitative version of the law of diminishing returns.

Since we want the performance to be five times faster, the new execution time should be 20 seconds, giving

That is, there is no amount by which we can enhance-multiply to achieve a fivefold increase in performance, if multiply accounts for only 80% of the workload. The performance enhancement possible with a given improvement is limited by the amount that the improved feature is used. In everyday life this concept also yields what we call the law of diminishing returns. We can use Amdahl’s Law to estimate performance improvements when we know the time consumed for some function and its potential speedup. Amdahl’s Law, together with the CP U performance equation, is a handy tool for evaluating potential enhancements. Amdahl’s Law is explored in more detail in the exercises. Amdahl’s Law is also used to argue for practical limits to the number of parallel processors. We examine this argument in the Fallacies and P itfalls section of Chapter 6.

Fallacy: Computers at low utilization use little power.
P ower efficiency ma ers at low utilizations because server workloads vary. U tilization of servers in Google’s warehouse scale computer, for example, is between 10% and 50% most of the time and at 100% less than 1% of the time. Even given five years to learn how to run the S P ECpower benchmark well, the specially configured computer with the best results in 2012 still uses 33% of the peak power at 10% of the load. S ystems in the field that are not configured for the SPECpower benchmark are surely worse. S ince servers’ workloads vary but use a large fraction of peak power, Luiz Barroso and U rs Hölzle [2007] argue that we should redesign hardware to achieve “energy-proportional computing.” If future servers used, say, 10% of peak power at 10% workload, we could reduce the electricity bill of datacenters and become good corporate citizens in an era of increasing concern about CO2 emissions.

Fallacy: Designing for performance and designing for energy efficiency are unrelated goals.

S ince energy is power over time, it is often the case that hardware or software optimizations that take less time save energy overall even if the optimization takes a bit more energy when it is used. O ne reason is that all of the rest of the computer is consuming energy while the program is running, so even if the optimized portion uses a little more energy, the reduced time can save the energy of the whole system.

Pitfall: Using a subset of the performance equation as a performance metric.
We have already shown the fallacy of predicting performance based on simply one of clock rate, instruction count, or CP I. Another common mistake is to use only two of the three factors to compare performance. Although using two of the three factors may be valid in a limited context, the concept is also easily misused. Indeed, nearly all proposed alternatives to the use of time as the performance metric have led eventually to misleading claims, distorted results, or incorrect interpretations. One alternative to time is MIPS (million instructions per second). For a given program, MIPS is simply

m i l l i o n

i n s t r u c t i o n s

p e r

s e c o

A measurement of program execution speed based on the number of millions of instructions. MIP S is computed as the instruction count divided by the product of the execution time and 106.

S ince MIP S is an instruction execution rate, MIP S specifies performance inversely to execution time; faster computers have a higher MIP S rating The good news about MIPS is that it is easy to understand, and faster computers mean bigger MIPS, which matches intuition. There are three problems with using MIP S as a measure for comparing computers. First, MIP S specifies the instruction execution rate but does not take into account the capabilities of the instructions. We cannot compare computers with different instruction sets using MIP S , since the instruction counts will certainly differ. S econd, MIP S varies between programs on the same computer; thus, a computer cannot have a single MIP S rating. For example, by substituting for execution time, we see the relationship between MIPS, clock rate, and CPI:

The CP I varied by a factor of 5 for S P EC CP U 2006 on an Intel Core i7 computerFigure 1.18, so MIP S does as well. Finally, and most in importantly, if a new program executes more instructions but each instruction is faster, MIPS can vary independently from performance!

C h e c k
Measurement Computer A

Y o u r s e l f
Computer B

Consider the following performance measurements for a program:
Instruction count 10 billion Clock rate 4 GHz CPI 1.0 8 billion 4 GHz 1.1

a. Which computer has the higher MIPS rating? b. Which computer is faster?

1.11 Concluding Remarks
Where … the ENIAC is equipped with 18,000 vacuum tubes and weighs 30 tons, computers in the future may have 1,000 vacuum tubes and perhaps weigh just 1½ tons.
Popular Mechanics, March 1949

Although it is difficult to predict exactly what level of cost/performance computers will have in the future, it’s a safe bet that they will be much be er than they are today. To participate in these advances, computer designers and programmers must understand a wider variety of issues. Both hardware and software designers construct computer systems in hierarchical layers, with each lower layer hiding details from the level above. This great idea of abstraction is fundamental to understanding today’s computer systems, but it does not mean that designers can limit themselves to knowing a single abstraction. Perhaps the most important example of abstraction is the interface between hardware and low-level software, called the instruction set architecture. Maintaining the instruction set architecture as a constant enables many implementations of that architecture—presumably varying in cost and performance—to run identical software. O n the downside, the architecture may preclude introducing innovations that require the interface to change.

There is a reliable method of determining and reporting performance by using the execution time of real programs as the metric. This execution time is related to other important measurements we can make by the following equation:

We will use this equation and its constituent factors many times. Remember, though, that individually the factors do not determine performance: only the product, which equals execution time, is a reliable measure of performance.

T h e

B I G

P i c t u r e

Execution time is the only valid and unimpeachable measure of performance. Many other metrics have been proposed and found wanting. S ometimes these metrics are flawed from the start by not reflecting execution time; other times a metric that is valid in a limited context is extended and used beyond that context or without the additional clarification needed to make it valid.

The key hardware technology for modern processors is silicon. Equal in importance to an understanding of integrated circuit technology is an understanding of the expected rates of technological change, as predicted by Moore’s Law. While silicon fuels the rapid advance of hardware, new ideas in the organization of computers have improved price/performance. Two of the key ideas are exploiting parallelism in the program, typically today via multiple processors, and exploiting locality of accesses to a memory hierarchy, typically via caches.

Energy efficiency has replaced die area as the most critical resource of microprocessor design. Conserving power while trying to increase performance has forced the hardware industry to switch to multicore microprocessors, thereby forcing the software industry to switch to programming parallel hardware. Parallelism is now required for performance.

Computer designs have always been measured by cost and performance, as well as other important factors such as energy, dependability, cost of ownership, and scalability. Although this chapter has focused on cost, performance, and energy, the best designs will strike the appropriate balance for a given market among all the factors.

Road Map for This Book
At the bo om of these abstractions are the five classic components of a computer: datapath, control, memory, input, and output (refer to Figure 1.5). These five components also serve as the framework for the rest of the chapters in this book: ■ Datapath: Chapter 3, Chapter 4, Chapter 6, and Appendix C ■ Control: Chapter 4, Chapter 6, and Appendix C ■ Memory: Chapter 5 ■ Input: Chapters 5 and 6 ■ Output: Chapters 5 and 6 As mentioned above, Chapter 4 describes how processors exploit implicit parallelism, Chapter 6 describes the explicitly parallel multicore microprocessors that are at the heart of the parallel revolution, and Appendix C describes the highly parallel graphics processor chip. Chapter 5 describes how a memory hierarchy exploits locality. Chapter 2 describes instruction sets—the interface between compilers and the computer—and emphasizes the role of compilers and programming languages in using the features of the instruction set. Appendix A provides a reference for the instruction set of Chapter 2. Chapter 3 describes how computers handle arithmetic data. Appendix B introduces logic design.

Historical Perspective and Further Reading
An active field of science is like an immense anthill; the individual almost vanishes into the mass of minds tumbling over each other, carrying information from place to place, passing it around at the speed of light.
Lewis Thomas, “Natural Science,” in The Lives of a Cell, 1974

For each chapter in the text, a section devoted to a historical perspective can be found online on a site that accompanies this book. We may trace the development of an idea through a series of computers or describe some important projects, and we provide references in case you are interested in probing further. The historical perspective for this chapter provides a background for some of the key ideas presented in this opening chapter. Its purpose is to give you the human story behind the technological advances and to place achievements in their historical context. By understanding the past, you may be be er able to understand the forces that will shape computing in the future. Each Historical Perspective section online ends with suggestions for further reading, which are also collected separately online under the section “Further Reading.” The rest of Section 1.12 is found online.

1.13 Exercises
The relative time ratings of exercises are shown in square brackets after each exercise number. O n average, an exercise rated [10] will take you twice as long as one rated [5]. S ections of the text that should be read before a empting an exercise will be given in angled brackets; for example,means you should have read Section 1.4, Under the Covers, to help you solve this exercise. 1.1 [2] Aside from the smart cell phones used by a billion people, list and describe four other types of computers. 1.2 [5] The eight great ideas in computer architecture are similar to ideas from other fields. Match the eight ideas from computer architecture, “Design for Moore’s Law”, “Use Abstraction to Simplify Design”, “Make the Common Case Fast”, “Performance via Parallelism”, “Performance via Pipelining”, “Performance via Prediction”, “Hierarchy of Memories”, and “Dependability via Redundancy” to the following ideas from other fields: a. Assembly lines in automobile manufacturing b. Suspension bridge cables c. Aircraft and marine navigation systems that incorporate wind information d. Express elevators in buildings e. Library reserve desk f. Increasing the gate area on a CMOS transistor to decrease its switching time g. Adding electromagnetic aircraft catapults (which are electrically-powered as opposed to current steam-powered models), allowed by the increased power generation offered by the new reactor technology h. Building self-driving cars whose control systems partially rely on existing sensor systems already installed into the base vehicle, such as lane departure systems and smart cruise control systems 1.3 [2] Describe the steps that transform a program written in a high-level language such as C into a representation that is directly executed by a computer processor. 1.4 [2] Assume a color display using 8 bits for each of the primary colors (red, green, blue) per pixel and a frame size of 1280×1024. a. What is the minimum size in bytes of the frame buffer to store a frame? b. How long would it take, at a minimum, for the frame to be sent over a 100 Mbit/s network? 1.5 [4] Consider three different processors P1, P2, and P3 executing the same instruction set. P1 has a 3 GHz clock rate and a CPI of 1.5. P2 has a 2.5 GHz clock rate and a CPI of 1.0. P3 has a 4.0 GHz clock rate and has a CPI of 2.2. a. Which processor has the highest performance expressed in instructions per second? b. If the processors each execute a program in 10 seconds, find the number of cycles and the number of instructions. c. We are trying to reduce the time by 30% but this leads to an increase of 20% in the CPI. What clock rate should we have to get this time reduction? 1.6 [20] Consider two different implementations of the same instruction set architecture. The instructions can be divided into four classes according to their CPI (class A, B, C, and D). P1 with a clock rate of 2.5 GHz and CPIs of 1, 2, 3, and 3, and P2 with a clock rate of 3 GHz and CPIs of 2, 2, 2, and 2. Given a program with a dynamic instruction count of 1.0E6 instructions divided into classes as follows: 10% class A, 20% class B, 50% class C, and 20% class D, which implementation is faster? a. What is the global CPI for each implementation? b. Find the clock cycles required in both cases. 1.7 [15] Compilers can have a profound impact on the performance of an application. Assume that for a program, compiler A results in a dynamic instruction count of 1.0E9 and has an execution time of 1.1 s, while compiler B results in a dynamic instruction count of 1.2E9 and an execution time of 1.5 s. a. Find the average CPI for each program given that the processor has a clock cycle time of 1 ns. b. Assume the compiled programs run on two different processors. If the execution times on the two processors are the same, how much faster is the clock of the processor running compiler A’s code versus the clock of the processor running compiler B’s code? c. A new compiler is developed that uses only 6.0E8 instructions and has an average CPI of 1.1. What is the speedup of using this new compiler versus using compiler A or B on the original processor? 1.8 The Pentium 4 Prescott processor, released in 2004, had a clock rate of 3.6 GHz and voltage of 1.25 V. Assume that, on average, it consumed 10 W of static power and 90 W of dynamic power. The Core i5 Ivy Bridge, released in 2012, has a clock rate of 3.4 GHz and voltage of 0.9 V. Assume that, on average, it consumed 30 W of static power and 40 W of dynamic power. 1.8.1 [5] For each processor find the average capacitive loads. 1.8.2 [5] Find the percentage of the total dissipated power comprised by static power and the ratio of static power to dynamic power for each technology. 1.8.3 [15] If the total dissipated power is to be reduced by 10%, how much should the voltage be reduced to maintain the same leakage current? Note: power is defined as the product of voltage and current. 1.9 Assume for arithmetic, load/store, and branch instructions, a processor has CPIs of 1, 12, and 5, respectively. Also assume that on a single processor a program requires the execution of 2.56E9 arithmetic instructions, 1.28E9 load/store instructions, and 256 million branch instructions. Assume that each processor has a 2 GHz clock frequency. Assume that, as the program is parallelized to run over multiple cores, the number of arithmetic and load/store instructions per processor is divided by 0.7 x p (where p is the number of processors) but the number of branch instructions per processor remains the same. 1.9.1 [5] Find the total execution time for this program on 1, 2, 4, and 8 processors, and show the relative speedup of the 2, 4, and 8 processor result relative to the single processor result. 1.9.2 [10] If the CPI of the arithmetic instructions was doubled, what would the impact be on the execution time of the program on 1, 2, 4, or 8 processors? 1.9.3 [10] To what should the CPI of load/store instructions be reduced in order for a single processor to match the performance of four processors using the original CPI values? 1.10 Assume a 15 cm diameter wafer has a cost of 12, contains 84 dies, and has 0.020 defects/cm2. Assume a 20 cm diameter wafer has a cost of 15, contains 100 dies, and has 0.031 defects/cm2. 1.10.1 [10] Find the yield for both wafers. 1.10.2 [5] Find the cost per die for both wafers. 1.10.3 [5] If the number of dies per wafer is increased by 10% and the defects per area unit increases by 15%, find the die area and yield. 1.10.4 [5] Assume a fabrication process improves the yield from 0.92 to 0.95. Find the defects per area unit for each version of the technology given a die area of 200 mm2. 1.11 The results of the SPEC CPU2006 bzip2 benchmark running on an AMD Barcelona has an instruction count of 2.389E12, an execution time of

1.11 The results of the SPEC CPU2006 bzip2 benchmark running on an AMD Barcelona has an instruction count of 2.389E12, an execution time of 750 s, and a reference time of 9650 s. 1.11.1 [5] Find the CPI if the clock cycle time is 0.333 ns. 1.11.2 [5] Find the SPECratio. 1.11.3 [5] Find the increase in CPU time if the number of instructions of the benchmark is increased by 10% without affecting the CPI. 1.11.4 [5] Find the increase in CPU time if the number of instructions of the benchmark is increased by 10% and the CPI is increased by 5%. 1.11.5 [5] Find the change in the SPECratio for this change. 1.11.6 [10] Suppose that we are developing a new version of the AMD Barcelona processor with a 4 GHz clock rate. We have added some additional instructions to the instruction set in such a way that the number of instructions has been reduced by 15%. The execution time is reduced to 700 s and the new SPECratio is 13.7. Find the new CPI. 1.11.7 [10] This CPI value is larger than obtained in 1.11.1 as the clock rate was increased from 3 GHz to 4 GHz. Determine whether the increase in the CPI is similar to that of the clock rate. If they are dissimilar, why? 1.11.8 [5] By how much has the CPU time been reduced? 1.11.9 [10] For a second benchmark, libquantum, assume an execution time of 960 ns, CPI of 1.61, and clock rate of 3 GHz. If the execution time is reduced by an additional 10% without affecting to the CPI and with a clock rate of 4 GHz, determine the number of instructions. 1.11.10 [10] Determine the clock rate required to give a further 10% reduction in CPU time while maintaining the number of instructions and with the CPI unchanged. 1.11.11 [10] Determine the clock rate if the CPI is reduced by 15% and the CPU time by 20% while the number of instructions is unchanged. 1.12 Section 1.10 cites as a pitfall the utilization of a subset of the performance equation as a performance metric. To illustrate this, consider the following two processors. P1 has a clock rate of 4 GHz, average CPI of 0.9, and requires the execution of 5.0E9 instructions. P2 has a clock rate of 3 GHz, an average CPI of 0.75, and requires the execution of 1.0E9 instructions. 1.12.1 [5] One usual fallacy is to consider the computer with the largest clock rate as having the largest performance. Check if this is true for P1 and P2. 1.12.2 [10] Another fallacy is to consider that the processor executing the largest number of instructions will need a larger CPU time. Considering that processor P1 is executing a sequence of 1.0E9 instructions and that the CPI of processors P1 and P2 do not change, determine the number of instructions that P2 can execute in the same time that P1 needs to execute 1.0E9 instructions. 1.12.3 [10] A common fallacy is to use MIPS (millions of instructions per second) to compare the performance of two different processors, and consider that the processor with the largest MIPS has the largest performance. Check if this is true for P1 and P2. 1.12.4 [10] Another common performance figure is MFLOPS (millions of floating-point operations per second), defined as

but this figure has the same problems as MIPS. Assume that 40% of the instructions executed on both P1 and P2 are floating-point instructions. Find the MFLOPS figures for the programs. 1.13 Another pitfall cited in Section 1.10 is expecting to improve the overall performance of a computer by improving only one aspect of the computer. Consider a computer running a program that requires 250 s, with 70 s spent executing FP instructions, 85 s executed L/S instructions, and 40 s spent executing branch instructions. 1.13.1 [5] By how much is the total time reduced if the time for FP operations is reduced by 20%? 1.13.2 [5] By how much is the time for INT operations reduced if the total time is reduced by 20%? 1.13.3 [5] Can the total time can be reduced by 20% by reducing only the time for branch instructions? 1.14 Assume a program requires the execution of 50×106 FP instructions, 110×106 INT instructions, 80×106 L/S instructions, and 16×106 branch instructions. The CPI for each type of instruction is 1, 1, 4, and 2, respectively. Assume that the processor has a 2 GHz clock rate. 1.14.1 [10] By how much must we improve the CPI of FP instructions if we want the program to run two times faster? 1.14.2 [10] By how much must we improve the CPI of L/S instructions if we want the program to run two times faster? 1.14.3 [5] By how much is the execution time of the program improved if the CPI of INT and FP instructions is reduced by 40% and the CPI of L/S and Branch is reduced by 30%? 1.15 [5] When a program is adapted to run on multiple processors in a multiprocessor system, the execution time on each processor is comprised of computing time and the overhead time required for locked critical sections and/or to send data from one processor to another. Assume a program requires t=100 s of execution time on one processor. When run p processors, each processor requires t/p s, as well as an additional 4 s of overhead, irrespective of the number of processors. Compute the per-processor execution time for 2, 4, 8, 16, 32, 64, and 128 processors. For each case, list the corresponding speedup relative to a single processor and the ratio between actual speedup versus ideal speedup (speedup if there was no overhead).

Instructions
Language of the Computer
This chapter describes instructions, the language of the computer. It explains the two principles of the stored-program computer: the use of instructions that are indistinguishable from numbers, and the use of alterable memory for programs. The “instruction set architecture” is an abstract interface between the hardware and the lowest-level software that encompasses all the information necessary to write a machine language program that will run correctly. Above this machine level is assembly language, a language that humans can read. The assembler translates the language into the binary numbers that machines can understand, and it even “extends” the instruction set by creating symbolic instructions that aren’t in the hardware. Each category of MIPS instructions is associated with constructs that appear in programming languages. The popularity of a few instructions dominates the many. The varying popularity of these instructions plays an important role in the chapters about datapath, control, and pipelining.

Keywords

operand; signed number; unsigned number; instructions; logical operations; procedures; MIP S ; synchronization; C S ort; arrays; pointers; compiling C; C; interpreting Java; Java; ARM instructions; ARM; x86 instructions; instruction set; stored-program concept; word; data transfer instruction; address; alignment restriction; binary digit; binary bit; least significant bit; most significant bit; one’s complement; biased notation; instruction format; machine language; hexadecimal; opcode; AND; O R; NO T; NO R; conditional branch; basic block; jump address table; jump table procedure; jump-and-link instruction; return address; caller; callee; program counter; P C; stack; stack pointer; push; pop; global pointer; procedure frame; activation record; frame pointer; text segment; P C-relative addressing; addressing mode; data race; assembly language; pseudoinstruction; symbol table; linker; link editor; executable file; loader; dynamically linked libraries; DLL; Jave bytecode; Java Virtual Machine; JVM; Just In Tim compiler; JIT; object oriented language; MIPS; ARM; x86; general-purpose register; GPR

I speak Spanish to God, Italian to women, French to men, and German to my horse.
Charles V, Holy Roman Emperor (1500–1558) 2.1 Introduction 2.2 Operations of the Computer Hardware 2.3 Operands of the Computer Hardware 2.4 Signed and Unsigned Numbers 2.5 Representing Instructions in the Computer 2.6 Logical Operations 2.7 Instructions for Making Decisions 2.8 Supporting Procedures in Computer Hardware 2.9 Communicating with People 2.10 MIPS Addressing for 32-bit Immediates and Addresses 2.11 Parallelism and Instructions: Synchronization 2.12 Translating and Starting a Program 2.13 A C Sort Example to Put It All Together 2.14 Arrays versus Pointers 2.15 Advanced Material: Compiling C and Interpreting Java 2.16 Real Stuff: ARMv7 (32-bit) Instructions 2.17 Real Stuff: x86 Instructions 2.18 Real Stuff: ARMv8 (64-bit) Instructions 2.19 Fallacies and Pitfalls 2.20 Concluding Remarks 2.21 Historical Perspective and Further Reading 2.22 Exercises

The Five Classic Components of a Computer

2.1 Introduction
To command a computer’s hardware, you must speak its language. The words of a computer’s language are calledinstructions, and its vocabulary is called an instruction set. In this chapter, you will see the instruction set of a real computer, both in the form wri en by people and in the form read by the computer. We introduce instructions in a top-down fashion. S tarting from a notation that looks like a restricted programming language, we refine it step-by-step until you see the real language of a real computer. Chapter 3 continues our downward descent, unveiling the hardware for arithmetic and the representation of floating-point numbers.

i n s t r u c t i o n

s e t

The vocabulary of commands understood by a given architecture.

You might think that the languages of computers would be as diverse as those of people, but in reality computer languages are quite similar, more like regional dialects than like independent languages. Hence, once you learn one, it is easy to pick up others. The chosen instruction set comes from MIP S Technologies, which is an elegant example of the instruction sets designed since the 1980s. To demostrate how easy it is to pick up other instruction sets, we will take a quick look at three other popular instruction sets. 1. ARMv7 is similar to MIPS. More than 9 billion chips with ARM processors were manufactured in 2011, making it the most popular instruction set in the world. 2. The second example is the Intel x86, which powers both the PC and the cloud of the PostPC Era. 3. The third example is ARMv8, which extends the address size of the ARMv7 from 32 bits to 64 bits. Ironically, as we shall see, this 2013 instruction set is closer to MIPS than it is to ARMv7. This similarity of instruction sets occurs because all computers are constructed from hardware technologies based on similar underlying principles and because there are a few basic operations that all computers must provide. Moreover, computer designers have a common goal: to find a language that makes it easy to build the hardware and the compiler while maximizing performance and minimizing cost and energy. This goal is time honored; the following quote was written before you could buy a computer, and it is as true today as it was in 1947:

It is easy to see by formal-logical methods that there exist certain [instruction sets] that are in abstract adequate to control and cause the execution of any sequence of operations. … The really decisive considerations from the present point of view, in selecting an [instruction set], are more of a practical nature: simplicity of the equipment demanded by the [instruction set], and the clarity of its application to the actually important problems together with the speed of its handling of those problems.
Burks, Goldstine, and von Neumann, 1947

The “simplicity of the equipment” is as valuable a consideration for today’s computers as it was for those of the 1950s. The goal of this chapter is to teach an instruction set that follows this advice, showing both how it is represented in hardware and the relationship between high-level programming languages and this more primitive one. O ur examples are in the C programming language; Section 2.15 shows how these would change for an object-oriented language like Java. By learning how to represent instructions, you will also discover the secret of computing: the stored-program concept. Moreover, you will exercise your “foreign language” skills by writing programs in the language of the computer and running them on the simulator that comes with this book. You will also see the impact of programming languages and compiler optimization on performance. We conclude with a look at the historical evolution of instruction sets and an overview of other computer dialects.

s t o r e d - p r o g r a m

c o n c e p t

The idea that instructions and data of many types can be stored in memory as numbers, leading to the stored program computer.

We reveal our first instruction set a piece at a time, giving the rationale along with the computer structures. This top-down, step-by-step tutorial weaves the components with their explanations, making the computer’s language more palatable. Figure 2.1 gives a sneak preview of the instruction set covered in this chapter.

FIGURE 2.1 MIPS assembly language revealed in this chapter. This information is also found in Column 1 of the MIPS Reference Data Card at the front of this book.

2.2 Operations of the Computer Hardware
There must certainly be instructions for performing the fundamental arithmetic operations.
Burks, Goldstine, and von Neumann, 1947

Every computer must be able to perform arithmetic. The MIPS assembly language notation add a, b, c

instructs a computer to add the two variables b and c and to put their sum in a. This notation is rigid in that each MIP S arithmetic instruction performs only one operation and must always have exactly three variables. For example, suppose we want to place the sum of four variables b, c, d, and e into variable a. (In this section we are being deliberately vague about what a “variable” is; in the next section we’ll explain in detail.) The following sequence of instructions adds the four variables:

Thus, it takes three instructions to sum the four variables. The words to the right of the sharp symbol (#) on each line above are comments for the human reader, and the computer ignores them. Note that unlike other programming languages, each line of this language can contain at most one instruction. Another difference from C is that comments always terminate at the end of a line. The natural number of operands for an operation like addition is three: the two numbers being added together and a place to put the sum. Requiring every instruction to have exactly three operands, no more and no less, conforms to the philosophy of keeping the hardware simple: hardware for a variable number of operands is more complicated than hardware for a fixed number. This situation illustrates the first of three underlying principles of hardware design:

Design Principle 1: Simplicity favors regularity.
We can now show, in the two examples that follow, the relationship of programs wri en in higher-level programming languages to programs in this more primitive notation.

C o m p i l i n g Example a = b + c; d = a – e;

T w o

C

A s s i g n m e n t

S

This segment of a C program contains the five variablesa, b, c, d, and e. S ince Java evolved from C, this example and the next few work for either high-level programming language: The translation from C to MIP S assembly language instructions is performed by the compiler. S how the MIP S code produced by a compiler.

Answer
A MIP S instruction operates on two source operands and places the result in one destination operand. Hence, the two simple statements above compile directly into these two MIPS assembly language instructions: add a, b, c sub d, a, e

C o m p i l i n g Example f = (g + h) – (i + j);

a

C o m p l e x

C

A s s i g n

A somewhat complex statement contains the five variables f, g, h, i, and j: What might a C compiler produce?

Answer
The compiler must break this statement into several assembly instructions, since only one operation is performed per MIP S instruction. The first MIP S instruction calculates the sum of and h. We must place the result somewhere, so the compiler creates a g temporary variable, called t0: add t0,g,h # temporary variable t0 contains g + h

Although the next operation is subtract, we need to calculate the sum of i and j before we can subtract. Thus, the second instruction places the sum of i and j in another temporary variable created by the compiler, called t1: add t1,i,j # temporary variable t1 contains i + j

Finally, the subtract instruction subtracts the second sum from the first and places the difference in the variable f, completing the compiled code: sub f,t0,t1 # f gets t0 –t1, which is (g + h) – (i + j)

C h e c k

Y o u r s e l f

For a given function, which programming language likely takes the most lines of code? Put the three representations below in order. 1. Java 2. C 3. MIPS assembly language

E l a b o r a t i o n
To increase portability, Java was originally envisioned as relying on a software interpreter. The instruction set of this interpreter is called Java bytecodes (see Section 2.15), which is quite different from the MIP S instruction set. To get performance close to the equivalent C program, Java systems today typically compile Java bytecodes into the native instruction sets like MIP S . Because this compilation is normally done much later than for C programs, such Java compilers are often calledJust In Time(JIT) compilers. S ection 2.12 shows how JITs are used later than C compilers in the start-up process, andS ection 2.13 shows the performance consequences of compiling versus interpreting Java programs.

2.3 Operands of the Computer Hardware
U nlike programs in high-level languages, the operands of arithmetic instructions are restricted; they must be from a limited number of special locations built directly in hardware called registers. Registers are primitives used in hardware design that are also visible to the programmer when the computer is completed, so you can think of registers as the bricks of computer construction. The size of a register in the MIP S architecture is 32 bits; groups of 32 bits occur so frequently that they are given the name word in the MIPS architecture.

w o r d
The natural unit of access in a computer, usually a group of 32 bits; corresponds to the size of a register in the MIPS architecture.

O ne major difference between the variables of a programming language and registers is the limited number of registers, typically 32 on current computers, like MIP S . (S ee Section 2.21 for the history of the number of registers.) Thus, continuing in our top-down, stepwise evolution of the symbolic representation of the MIP S language, in this section we have added the restriction that the three operands of MIP S arithmetic instructions must each be chosen from one of the 32 32-bit registers. The reason for the limit of 32 registers may be found in the second of our three underlying design principles of hardware technology:

Design Principle 2: Smaller is faster.
A very large number of registers may increase the clock cycle time simply because it takes electronic signals longer when they must travel farther. Guidelines such as “smaller is faster” are not absolutes; 31 registers may not be faster than 32. Yet, the truth behind such observations causes computer designers to take them seriously. In this case, the designer must balance the craving of programs for more registers with the designer’s desire to keep the clock cycle fast. Another reason for not using more than 32 is the number of bits it would take in the instruction format, as Section 2.5 demonstrates. Chapter 4 shows the central role that registers play in hardware construction; as we shall see in this chapter, effective use of registers is critical to program performance. Although we could simply write instructions using numbers for registers, from 0 to 31, the MIP S convention is to use two-character names following a dollar sign to represent a register. Section 2.8 will explain the reasons behind these names. For now, we will use $s0, $s1,… for registers that correspond to variables in C and Java programs and$t0, $t1,… for temporary registers needed to compile the program into MIP S instructions.

C o m p i l i n g Example f = (g + h) – (i + j); The variables f, g, h, i, and j

a

C

A s s i g n m e n t

U s i

It is the compiler’s job to associate program variables with registers. Take, for instance, the assignment statement from our earlier example: are assigned to the registers $s0, $s1, $s2, $s3, and $s4, respectively. What is the compiled MIP S code?

Answer
The compiled program is very similar to the prior example, except we replace the variables with the register names mentioned above plus two temporary registers, $t0 and $t1, which correspond to the temporary variables above:

Memory Operands

P rogramming languages have simple variables that contain single data elements, as in these examples, but they also have more complex data structures—arrays and structures. These complex data structures can contain many more data elements than there are registers in a computer. How can a computer represent and access such large structures? Recall the five components of a computer introduced in Chapter 1 and repeated on page 61. The processor can keep only a small amount of data in registers, but computer memory contains billions of data elements. Hence, data structures (arrays and structures) are kept in memory. As explained above, arithmetic operations occur only on registers in MIP S instructions; thus, MIP S must include instructions that transfer data between memory and registers. S uch instructions are called data transfer instructions. To access a word in memory, the instruction must supply the memory address. Memory is just a large, single-dimensional array, with the address acting as the index to that array, starting at 0. For example, in Figure 2.2, the address of the third data element is 2, and the value of Memory[2] is 10.

d a t a

t r a n s f e r

i n s t r u c t i o n

A command that moves data between memory and registers.

a d d r e s s
A value used to delineate the location of a specific data element within a memory array.

FIGURE 2.2 Memory addresses and contents of memory at those locations. If these elements were words, these addresses would be incorrect, since MIPS actually uses byte addressing, with each word representing four bytes. Figure 2.3 shows the memory addressing for sequential word addresses.

The data transfer instruction that copies data from memory to a register is traditionally called load. The format of the load instruction is the name of the operation followed by the register to be loaded, then a constant and register used to access memory. The sum of the constant portion of the instruction and the contents of the second register forms the memory address. The actual MIP S name for this instruction is lw, standing for load word.

C o m p i l i n g Example g = h + A[8];

a n

A s s i g n m e n t

W h e n

Let’s assume that A is an array of 100 words and that the compiler has associated the variables g and h with the registers $s1 and $s2 as before. Let’s also assume that the starting address, or base address, of the array is in $s3. Compile this C assignment statement:

Answer
Although there is a single operation in this assignment statement, one of the operands is in memory, so we must first transfer A[8] to a register. The address of this array element is the sum of the base of the array A, found in register $s3, plus the number to select element 8. The data should be placed in a temporary register for use in the next instruction. Based onFigure 2.2, the first compiled instruction is lw $t0,8($s3) # Temporary reg $t0 gets A[8]

(O n the next page we’ll make a slight adjustment to this instruction, but we’ll use this simplified version for now.) The following instruction can operate on the value in $t0 (which equals A[8]) since it is in a register. The instruction must add h (contained in ($2) to A[8] (contained in $t0) and put the sum in the register corresponding to g (associated with $s1): add $s1,$s2,$t0 # g = h + A[8]

The constant in a data transfer instruction (8) is called the offset, and the register added to form the address ($s3) is called the base register.

H a r d w a r e / S o f t w a r e

I n t e r f a c e

In addition to associating variables with registers, the compiler allocates data structures like arrays and structures to locations in memory. The compiler can then place the proper starting address into the data transfer instructions. Since 8-bit bytes are useful in many programs, virtually all architectures today address individual bytes. Therefore, the address of a word matches the address of one of the 4 bytes within the word, and addresses of sequential words differ by 4. For example, Figure 2.3 shows the actual MIPS addresses for the words in Figure 2.2; the byte address of the third word is 8.

FIGURE 2.3 Actual MIPS memory addresses and contents of memory for those words. The changed addresses are highlighted to contrast with Figure 2.2. Since MIPS addresses each byte, word addresses are multiples of 4: there are 4 bytes in a word.

In MIP S , words must start at addresses that are multiples of 4. This requirement is called an alignment restriction, and many architectures have it. (Chapter 4 suggests why alignment leads to faster data transfers.)

a l i g n m e n t

r e s t r i c t i o n

A requirement that data be aligned in memory on natural boundaries.

Computers divide into those that use the address of the leftmost or “big end” byte as the word address versus those that use the rightmost or “li le end” byte. MIP S is in the big-endian camp. S ince the order ma ers only if you access the identical data both as a word and as four bytes, few need to be aware of the endianess. (Appendix A, shows the two options to number bytes in a word.) Byte addressing also affects the array index. To get the proper byte address in the code above,the offset to be added to the base register$s3must be 4×8, or 32, so that the load address will select A[8] and not A[8/4]. (S ee the related pitfall on page 175 of Section 2.19.)

The instruction complementary to load is traditionally called store; it copies data from a register to memory. The format of a store is similar to that of a load: the name of the operation, followed by the register to be stored, then offset to select the array element, and finally the base register. O nce again, the MIP S address is specified in part by a constant and in part by the contents of a register. The actual MIP S namesw, standing for store is word.

H a r d w a r e / S o f t w a r e

I n t e r f a c e

As the addresses in loads and stores are binary numbers, we can see why the DRAM for main memory comes in binary sizes rather than in decimal sizes. That is, in gebibytes (230) or tebibytes (240), not in giabytes (109) or terabytes (1012); see Figure 1.1.

C o m p i l i n g Example
A[12] = h + A[8];

U s i n g

L o a d

a n d

S t o

Assume variable h is associated with register $s2 and the base address of the array A is in $s3. What is the MIP S assembly code for the C assignment statement below?

Answer
Although there is a single operation in the C statement, now two of the operands are in memory, so we need even more MIP S instructions. The first two instructions are the same as the prior example, except this time we use the proper offset for byte addressing in the load word instruction to select A[8], and the add instruction places the sum in $t0: lw $t0,32($s3) # Temporary reg $t0 gets A[8] add $t0,$s2,$t0 # Temporary reg $t0 gets h + A[8] The final instruction stores the sum into A[12], using 48 (4×12) as the offset and register $s3 as the base register. sw $t0,48($s3) # Stores h + A[8] back into A[12]

Load word and store word are the instructions that copy words between memory and registers in the MIP S architecture. O ther brands of computers use other instructions along with load and store to transfer data. An architecture with such alternatives is the Intel x86, described in Section 2.17.

H a r d w a r e / S o f t w a r e

I n t e r f a c e

Many programs have more variables than computers have registers. Consequently, the compiler tries to keep the most frequently

Many programs have more variables than computers have registers. Consequently, the compiler tries to keep the most frequently used variables in registers and places the rest in memory, using loads and stores to move variables between registers and memory. The process of putting less commonly used variables (or those needed later) into memory is called spilling registers. The hardware principle relating size and speed suggests that memory must be slower than registers, since there are fewer registers. This is indeed the case; data accesses are faster if data is in registers instead of memory. Moreover, data is more useful when in a register. A MIP S arithmetic instruction can read two registers, operate on them, and write the result. A MIPS data transfer instruction only reads one operand or writes one operand, without operating on it. Thus, registers take less time to access and have higher throughput than memory, making data in registers both faster to access and simpler to use. Accessing registers also uses less energy than accessing memory. To achieve highest performance and conserve energy, instruction set architecture must have a sufficient number of registers and compilers must use registers efficiently.

Constant or Immediate Operands
Many times a program will use a constant in an operation—for example, incrementing an index to point to the next element of an array. In fact, more than half of the MIPS arithmetic instructions have a constant as an operand when running the SPEC CPU2006 benchmarks. U sing only the instructions we have seen so far, we would have to load a constant from memory to use one. (The constants would have been placed in memory when the program was loaded.) For example, to add the constant 4 to register $s3, we could use the code lw $t0, AddrConstant4($s1) # $t0 = constant 4 add $s3,$s3,$t0 # $s3 = $s3 + $t0 ($t0 == 4) assuming that $s1 + AddrConstant4 is the memory address of the constant 4.

An alternative that avoids the load instruction is to offer versions of the arithmetic instructions in which one operand is a constant. This quick add instruction with one constant operand is called add immediate or addi. To add 4 to register $s3, we just write addi $s3,$s3,4 # $s3 = $s3 + 4

Constant operands occur frequently, and by including constants inside arithmetic instructions, operations are much faster and use less energy than if constants were loaded from memory. The constant zero has another role, which is to simplify the instruction set by offering useful variations. For example, the move operation is just an add instruction where one operand is zero. Hence, MIP S dedicates a register $zero to be hard-wired to the value zero. (As you might expect, it is register number 0.) Using frequency to justify the inclusions of constants is another example of the great idea of making the common case fast.

C h e c k

Y o u r s e l f

Given the importance of registers, what is the rate of increase in the number of registers in a chip over time? 1. Very fast: They increase as fast as Moore’s law, which predicts doubling the number of transistors on a chip every 18 months. 2. Very slow: Since programs are usually distributed in the language of the computer, there is inertia in instruction set architecture, and so the number of registers increases only as fast as new instruction sets become viable.

E l a b o r a t i o n
Although the MIP S registers in this book are 32 bits wide, there is a 64-bit version of the MIP S instruction set with 32 64-bit registers. To keep them straight, they are officially called MIP S -32 and MIP S -64. In this chapter, we use a subset of MIP S -32. Appendix E shows the differences between MIP S -32 and MIP S -64.ections 2.16 and 2.18 show the much more dramatic difference between the S 32-bit address ARMv7 and its 64-bit successor, ARMv8.

E l a b o r a t i o n
The MIP S offset plus base register addressing is an excellent match to structures as well as arrays, since the register can point to the beginning of the structure and the offset can select the desired element. We’ll see such an example in Section 2.13.

E l a b o r a t i o n
The register in the data transfer instructions was originally invented to hold an index of an array with the offset used for the starting address of an array. Thus, the base register is also called the index register. Today’s memories are much larger and the software model of data allocation is more sophisticated, so the base address of the array is normally passed in a register since it won’t fit in the offset, as we shall see.

E l a b o r a t i o n
Since MIPS supports negative constants, there is no need for subtract immediate in MIPS.

2.4 Signed and Unsigned Numbers
First, let’s quickly review how a computer represents numbers. Humans are taught to think in base 10, but numbers may be represented in any base. For example, 123 base 10 5 1111011 base 2. Numbers are kept in computer hardware as a series of high and low electronic signals, and so they are considered base 2 numbers. (Just as base 10 numbers are called decimal numbers, base 2 numbers are called binary numbers.) A single digit of a binary number is thus the “atom” of computing, since all information is composed of binary digits or bits. This fundamental building block can be one of two values, which can be thought of as several alternatives: high or low, on or off, true or false, or 1 or 0.

b i n a r y

d i g i t

Also called binary bit. One of the two numbers in base 2, 0 or 1, that are the components of information.

Generalizing the point, in any number base, the value of ith digit d is

where i starts at 0 and increases from right to left. This representation leads to an obvious way to number the bits in the word: simply use the power of the base for that bit. We subscript decimal numbers with ten and binary numbers with two. For example,
1011two

represents

We number the bits 0, 1, 2, 3, … from right to left in a word. The drawing below shows the numbering of bits within a MIP S word and the placement of the number 1011two:

Since words are drawn vertically as well as horizontally, leftmost and rightmost may be unclear. Hence, the phrase least significant bit is used to refer to the rightmost bit (bit 0 above) and most significant bit to the leftmost bit (bit 31).

l e a s t

s i g n i fi c a n t

b i t

The rightmost bit in a MIPS word.

m o s t

s i g n i fi c a n t

b i t

The leftmost bit in a MIPS word.

The MIP S word is 32 bits long, so we can represent 32 different 32-bit pa erns. It is natural to let these combinations represent the numbers from 2 0 to 232−1 (4,294,967,295ten):

That is, 32-bit binary numbers can be represented in terms of the bit value times a power of 2 (here xi means the ith bit of x):

For reasons we will shortly see, these positive numbers are called unsigned numbers.

H a r d w a r e / S o f t w a r e

I n t e r f a c e

Base 2 is not natural to human beings; we have 10 fingers and so find base 10 natural. Why didn’t computers use decimal? In fact, the first commercial computer did offer decimal arithmetic. The problem was that the computer still used on and off signals, so a decimal digit was simply represented by several binary digits. Decimal proved so inefficient that subsequent computers reverted to all binary, converting to base 10 only for the relatively infrequent input/output events.

Keep in mind that the binary bit pa erns above are simply representatives of numbers. Numbers really have an infinite number of digits, with almost all being 0 except for a few of the rightmost digits. We just don’t normally show leading 0s. Hardware can be designed to add, subtract, multiply, and divide these binary bit pa erns. If the number that is the proper result of such operations cannot be represented by these rightmost hardware bits, overflow is said to have occurred. It’s up to the programming language, the operating system, and the program to determine what to do if overflow occurs. Computer programs calculate both positive and negative numbers, so we need a representation that distinguishes the positive from the negative. The most obvious solution is to add a separate sign, which conveniently can be represented in a single bit; the name for this representation is sign and magnitude. Alas, sign and magnitude representation has several shortcomings. First, it’s not obvious where to put the sign bit. To the right? To the left? Early computers tried both. S econd, adders for sign and magnitude may need an extra step to set the sign because we can’t know in advance what the proper sign will be. Finally, a separate sign bit means that sign and magnitude has both a positive and a negative zero, which can lead to problems for inattentive programmers. As a result of these shortcomings, sign and magnitude representation was soon abandoned. In the search for a more a ractive alternative, the question arose as to what would be the result for unsigned numbers if we tried to subtract a large number from a small one. The answer is that it would try to borrow from a string of leading 0s, so the result would have a string of leading 1s. Given that there was no obvious be er alternative, the final solution was to pick the representation that made the hardware simple: leading 0s mean positive, and leading 1s mean negative. This convention for representing signed binary numbers is called two’s complement representation:

The positive half of the numbers, from 0 to 2,147,483,647ten (231−1), use the same representation as before. The following bit pa ern (1000 … 0000two) represents the most negative number −2,147,483,648ten (−231). It is followed by a declining set of negative numbers: −2,147,483,647ten (1000 … 0001two) down to −1ten (1111 … 1111two). Two’s complement does have one negative number, −2,147,483,648ten, that has no corresponding positive number. S uch imbalance was also a worry to the ina entive programmer, but sign and magnitude had problems for both the programmer and the hardware designer. Consequently, every computer today uses two’s complement binary representations for signed numbers. Two’s complement representation has the advantage that all negative numbers have a 1 in the most significant bit. Consequently, hardware needs to test only this bit to see if a number is positive or negative (with the number 0 considered positive). This bit is often called the sign bit. By recognizing the role of the sign bit, we can represent positive and negative 32-bit numbers in terms of the bit value times a power of 2:

The sign bit is multiplied by −231, and the rest of the bits are then multiplied by positive versions of their respective base values.

B i n a r y Example Answer

t o

D e c i m a l

C o n v e r s i o n

What is the decimal value of this 32-bit two’s complement number?
1111 1111 1111 1111 1111 1111 1111 1100two

Substituting the number’s bit values into the formula above:

image
We’ll see a shortcut to simplify conversion from negative to positive soon.

Just as an operation on unsigned numbers can overflow the capacity of hardware to represent the result, so can an operation on two’s complement numbers. O verflow occurs when the leftmost retained bit of the binary bit pa ern is not the same as the infinite number of digits to the left (the sign bit is incorrect): a 0 on the left of the bit pattern when the number is negative or a 1 when the number is positive.

H a r d w a r e / S o f t w a r e

I n t e r f a c e

S igned versus unsigned applies to loads as well as to arithmetic. Thefunction of a signed load is to copy the sign repeatedly to fill the rest of the register—called sign extension—but its purpose is to place a correct representation of the number within that register. Unsigned loads simply fill with 0 s to the left of the data, since the number represented by the bit pattern is unsigned. When loading a 32-bit word into a 32-bit register, the point is moot; signed and unsigned loads are identical. MIP S does offer two flavors of byte loads: load byte (lb) treats the byte as a signed number and thus sign-extends to fill the 24 leftmost bits of the register, while load byte unsigned (lbu) works with unsigned integers. S ince C programs almost always use bytes to represent characters rather than consider bytes as very short signed integers, lbu is used practically exclusively for byte loads.

H a r d w a r e / S o f t w a r e

I n t e r f a c e

U nlike the numbers discussed above, memory addresses naturally start at 0 and continue to the largest address. P ut another way, negative addresses make no sense. Thus, programs want to deal sometimes with numbers that can be positive or negative and sometimes with numbers that can be only positive. S ome programming languages reflect this distinction. C, for example, names the former integers (declared as int in the program) and the la er unsigned integers (unsigned int). S ome C style guides even recommend declaring the former as signed int to keep the distinction clear.

Let’s examine two useful shortcuts when working with two’s complement numbers. The first shortcut is a quick way to negate a two’s complement binary number. S imply invert every 0 to 1 and every 1 to 0, then add one to the result. This shortcut is based on the observation that the sum of a number and its inverted representation must be 111 … 111two, which represents −1. S ince , therefore or . (We use the notation to mean invert every bit in x from 0 to 1 and vice versa.)

N e g a t i o n Example Answer

S h o r t c u t

Negate 2ten, and then check the result by negating −2ten. 2ten = 0000 0000 0000 0000 0000 0000 0000 0010two Negating this number by inverting the bits and adding one,

Going the other direction,
1111 1111 1111 1111 1111 1111 1111 1110two

is first inverted and then incremented:

O ur next shortcut tells us how to convert a binary number represented in n bits to a number represented with more than n bits. For example, the immediate field in the load, store, branch, add, and set on less than instructions contains a two’s complement 16-bit number, representing −32,768 ten (−215) to 32,767ten (215−1). To add the immediate field to a 32-bit register, the computer must convert that 16-bit number to its 32-bit equivalent. The shortcut is to take the most significant bit from the smaller quantity—the sign bit—and replicate it to fill the new bits of the larger quantity. The old nonsign bits are simply copied into the right portion of the new word. This shortcut is commonly called sign extension.

S i g n Example Answer

E x t e n s i o n

S h o r t c u t

Convert 16-bit binary versions of 2ten and −2ten to 32-bit binary numbers. The 16-bit binary version of the number 2 is
0000 0000 0000 0010two = 2ten

It is converted to a 32-bit number by making 16 copies of the value in the most significant bit (0) and placing that in the left-hand half of the word. The right half gets the old value:
0000 0000 0000 0000 0000 0000 0000 0010two = 2ten

Let’s negate the 16-bit version of 2 using the earlier shortcut. Thus,
0000 0000 0000 0010two

becomes

Creating a 32-bit version of the negative number means copying the sign bit 16 times and placing it on the left:
1111 1111 1111 1111 1111 1111 1111 1110two = –2ten

This trick works because positive two’s complement numbers really have an infinite number of 0s on the left and negative two’s complement numbers have an infinite number of 1s. The binary bit pa ern representing a number hides leading bits to fit the width of the hardware; sign extension simply restores some of them.

Summary
The main point of this section is that we need to represent both positive and negative integers within a computer word, and although there are pros and cons to any option, the unanimous choice since 1965 has been two’s complement.

E l a b o r a t i o n
For signed decimal numbers, we used “−” to represent negative because there are no limits to the size of a decimal number. Given a fixed word size, binary and hexadecimal (see Figure 2.4) bit strings can encode the sign; hence we do not normally use “+” or “−” with binary or hexadecimal notation.

C h e c k

Y o u r s e l f

What is the decimal value of this 64-bit two’s complement number?

E l a b o r a t i o n
Two’s complement gets its name from the rule that the unsigned sum of an n-bit number and its n-bit negative is 2n; hence, the negation or complement of a number x is 2n−x, or its “two’s complement.” A third alternative representation to two’s complement and sign and magnitude is called one’s complement. The negative of a one’s complement is found by inverting each bit, from 0 to 1 and from 1 to 0, or . This relation helps explain its name since the complement of x is 2n−×−1. It was also an a empt to be a be er solution than sign and magnitude, and several early scientific computers did use the notation. This representation is similar to two’s complement except that it also has two 0s: 00 … 00two is positive 0 and 11 … 11two is negative 0. The most negative number, 10 … 000two, represents −2,147,483,647ten, and so the positives and negatives are balanced. O ne’s complement adders did need an extra step to subtract a number, and hence two’s complement dominates today.

o n e ’ s

c o m p l e m e n t

A notation that represents the most negative value by 10 … 000two and the most positive value by 01 … 11two, leaving an equal number of negatives and positives but ending up with two zeros, one positive (00 … 00two) and one negative (11 …11two). The term is also used to mean the inversion of every bit in a pattern: 0 to 1 and 1 to 0.

A final notation, which we will look at when we discuss floating point in Chapter 3, is to represent the most negative value by 00 … 000two and the most positive value by 11 … 11two, with 0 typically having the value 10 … 00two. This is called a biased notation, since it biases the number such that the number plus the bias has a nonnegative representation.

b i a s e d

n o t a t i o n

A notation that represents the most negative value by 00 … 000two and the most positive value by 11 … 11two, with 0 typically having the value 10 … 00two, thereby biasing the number such that the number plus the bias has a nonnegative representation.

2.5 Representing Instructions in the Computer
We are now ready to explain the difference between the way humans instruct computers and the way computers see instructions. Instructions are kept in the computer as a series of high and low electronic signals and may be represented as numbers. In fact, each piece of an instruction can be considered as an individual number, and placing these numbers side by side forms the instruction. S ince registers are referred to by almost all instructions, there must be a convention to map register names into numbers. In MIP S assembly language, registers $s0 to $s7 map onto registers 16 to 23, and registers $t0 to $t7 map onto registers 8 to 15. Hence, $s0 means register 16, $s1 means register 17, $s2 means register 18,…, $t0 means register 8, $t1 means register 9, and so on. We’ll describe the convention for the rest of the 32 registers in the following sections.

T r a n s l a t i n g Example add $t0,$s1,$s2

a

M I P S

A s s e m b l y

Let’s do the next step in the refinement of the MIP S language as an example. We’ll show the real MIP S language version of the instruction represented symbolically as first as a combination of decimal numbers and then of binary numbers.

Answer
The decimal representation is

Each of these segments of an instruction is called a field. The first and last fields (containing 0 and 32 in this case) in combination tell the MIP S computer that this instruction performs addition. The second field gives the number of the register that is the first source operand of the addition operation (17=$s1), and the third field gives the other source operand for the addition (18=$s2). The fourth field contains the number of the register that is to receive the sum (8=$t0). The fifth field is unused in this instruction, so it is set to 0. Thus, this instruction adds register $s1 to register $s2 and places the sum in register $t0. This instruction can also be represented as fields of binary numbers as opposed to decimal:

This layout of the instruction is called the instruction format. As you can see from counting the number of bits, this MIP S instruction takes exactly 32 bits—the same size as a data word. In keeping with our design principle that simplicity favors regularity, all MIP S instructions are 32 bits long.

i n s t r u c t i o n

f o r m a t

A form of representation of an instruction composed of fields of binary numbers.

To distinguish it from assembly language, we call the numeric version of instructions machine language and a sequence of such instructions machine code.

m a c h i n e

l a n g u a g e

Binary representation used for communication within a computer system.

It would appear that you would now be reading and writing long, tedious strings of binary numbers. We avoid that tedium by using a higher base than binary that converts easily into binary. S ince almost all computer data sizes are multiples of 4, hexadecimal (base 16) numbers are popular. S ince base 16 is a power of 2, we can trivially convert by replacing each group of four binary digits by a single hexadecimal digit, and vice versa. Figure 2.4 converts between hexadecimal and binary.

h e x a d e c i m a l

Numbers in base 16.

FIGURE 2.4 The hexadecimal-binary conversion table. Just replace one hexadecimal digit by the corresponding four binary digits, and vice versa. If the length of the binary number is not a multiple of 4, go from right to left.

Because we frequently deal with different number bases, to avoid confusion we will subscript decimal numbers with ten, binary numbers with two, and hexadecimal numbers with hex. (If there is no subscript, the default is base 10.) By the way, C and Java use the notation 0xnnn for n hexadecimal numbers.

B i n a r y Example

t o

H e x a d e c i m a l

a n d

B a c

Convert the following hexadecimal and binary numbers into the other base: eca8 6420hex 0001 0011 0101 0111 1001 1011 1101 1111two

Answer
Using Figure 2.4, the answer is just a table lookup one way:

And then the other direction:

MIPS Fields
MIPS fields are given names to make them easier to discuss:

Here is the meaning of each name of the fields in MIPS instructions:
■ op: Basic operation of the instruction, traditionally called the opcode. ■ rs: The first register source operand. ■ rt: The second register source operand. ■ rd: The register destination operand. It gets the result of the operation.

■ rd: The register destination operand. It gets the result of the operation. ■ shamt: Shift amount. (Section 2.6 explains shift instructions and this term; it will not be used until then, and hence the field contains zero in this

section.)
■ funct: Function. This field, often called the function code, selects the specific variant of the operation in the op field.

o p c o d e
The field that denotes the operation and format of an instruction.

A problem occurs when an instruction needs longer fields than those shown above. For example, the load word instruction must specify two registers and a constant. If the address were to use one of the 5-bit fields in the format above, the constant within the load word instruction would be limited to only 25 or 32. This constant is used to select elements from arrays or data structures, and it often needs to be much larger than 32. This 5bit field is too small to be useful. Hence, we have a conflict between the desire to keep all instructions the same length and the desire to have a single instruction format. This leads us to the final hardware design principle:

Design Principle 3: Good design demands good compromises.
The compromise chosen by the MIP S designers is to keep all instructions the same length, thereby requiring different kinds of instruction formats for different kinds of instructions. For example, the format above is called R-type (for register) or R-format. A second type of instruction format is called I-type (for immediate) or I-format and is used by the immediate and data transfer instructions. The fields of I-format are

The 16-bit address means a load word instruction can load any word within a region of ±215 or 32,768 bytes (±213 or 8192 words) of the address in the base register rs. S imilarly, add immediate is limited to constants no larger than ±215. We see that more than 32 registers would be difficult in this format, as the rs and rt fields would each need another bit, making it harder to fit everything in one word. Let’s look at the load word instruction from page 71: lw $t0,32($s3) # Temporary reg $t0 gets A[8] Here, 19 (for $s3) is placed in the rs field, 8 (for $t0) is placed in the rt field, and 32 is placed in the address field. Note that the meaning of the rt

field has changed for this instruction: in a load word instruction, the rt field specifies the destination register, which receives the result of the load. Although multiple formats complicate the hardware, we can reduce the complexity by keeping the formats similar. For example, the first three fields of the R-type and I-type formats are the same size and have the same names; the length of the fourth field in I-type is equal to the sum of the lengths of the last three fields of R-type. In case you were wondering, the formats are distinguished by the values in the first field: each format is assigned a distinct set of values in the first field (op) so that the hardware knows whether to treat the last half of the instruction as three fields (R-type) or as a single field (I-type).Figure 2.5 shows the numbers used in each field for the MIPS instructions covered so far.

FIGURE 2.5 MIPS instruction encoding. In the table above, “reg” means a register number between 0 and 31, “address” means a 16-bit address, and “n.a.” (not applicable) means this field does not appear in this format. Note that add and sub instructions have the same value in the op field; the hardware uses the funct field to decide the variant of the operation: add (32) or subtract (34).

T r a n s l a t i n g Example
A[300] = h + A[300];

M I P S

A s s e m b l y

L a

We can now take an example all the way from what the programmer writes to what the computer executes. If$t1 has the base of the array A and $s2 corresponds to h, the assignment statement is compiled into

What is the MIPS machine language code for these three instructions?

Answer
For convenience, let’s first represent the machine language instructions using decimal numbers. FromFigure 2.5, we can determine the three machine language instructions:

The lw instruction is identified by 35 (see Figure 2.5) in the first field (op). The base register 9 ($t1) is specified in the second field (rs), and the destination register 8 ($t0) is specified in the third field (rt). The offset to select A[300] (1200=300×4) is found in the final field (address). The add instruction that follows is specified with 0 in the first field (op) and 32 in the last field (funct). The three register operands (18, 8, and 8) are found in the second, third, and fourth fields and correspond to $s2, $t0, and $t0. The sw instruction is identified with 43 in the first field. The rest of this final instruction is identical to the lw instruction. Since 1200ten=0000 0100 1011 0000two, the binary equivalent to the decimal form is:

Note the similarity of the binary representations of the first and last instructions. The only difference is in the third bit from the left, which is highlighted here.

H a r d w a r e / S o f t w a r e

I n t e r f a c e

The desire to keep all instructions the same size is in conflict with the desire to have as many registers as possible. Any increase in the number of registers uses up at least one more bit in every register field of the instruction format. Given these constraints and the design princple that smaller is faster, most instruction sets today have 16 or 32 general purpose registers.

Figure 2.6 summarizes the portions of MIP S machine language described in this section. As we shall see in Chapter 4, the similarity of the binary representations of related instructions simplifies hardware design. These similarities are another example of regularity in the MIPS architecture.

FIGURE 2.6 MIPS architecture revealed through Section 2.5. The two MIPS instruction formats so far are R and I. The first 16 bits are the same: both contain an op field, giving the base operation; an rs field, giving one of the sources; and the rt field, which specifies the other source operand, except for load word, where it specifies the destination register. R-format divides the last 16 bits into an rd field, specifying the destination register; the shamt field, which Section 2.6 explains; and the funct field, which specifies the specific operation of R-format instructions. I-format combines the last 16 bits into a single address field.

T h e

B I G

P i c t u r e

Today’s computers are built on two key principles: 1. Instructions are represented as numbers. 2. Programs are stored in memory to be read or written, just like numbers. These principles lead to the stored-program concept; its invention let the computing genie out of its bo le. Figure 2.7 shows the power of the concept; specifically, memory can contain the source code for an editor program, the corresponding compiled machine code, the text that the compiled program is using, and even the compiler that generated the machine code.

FIGURE 2.7 The stored-program concept. Stored programs allow a computer that performs accounting to become, in the blink of an eye, a computer that helps an author write a book. The switch happens simply by loading memory with programs and data and then telling the computer to begin executing at a given location in memory. Treating instructions in the same way as data greatly simplifies both the memory hardware and the software of computer systems. Specifically, the memory technology needed for data can also be used for programs, and programs like compilers, for instance, can translate code written in a notation far more convenient for humans into code that the computer can understand.

O ne consequence of instructions as numbers is that programs are often shipped as files of binary numbers. The commercial implication is that computers can inherit ready-made software provided they are compatible with an existing instruction set. S uch “binary compatibility” often leads industry to align around a small number of instruction set architectures.

C h e c k

Y o u r s e l f

What MIPS instruction does this represent? Chose from one of the four options below.

1. sub 2. add 3. sub 4. sub

$t0, $t2, $t2, $t2,

$t1, $t0, $t1, $t0,

$t2 $t1 $t0 $t1

2.6 Logical Operations
“Contrariwise,” continued Tweedledee, “if it was so, it might be; and if it were so, it would be; but as it isn’t, it ain’t. That’s logic.”
Lewis Carroll, Alice’s Adventures in Wonderland, 1865

Although the first computers operated on full words, it soon became clear that it was useful to operate on fields of bits within a word or even on individual bits. Examining characters within a word, each of which is stored as 8 bits, is one example of such an operation (see Section 2.9). It follows that operations were added to programming languages and instruction set architectures to simplify, among other things, the packing and unpacking of bits into words. These instructions are called logical operations. Figure 2.8 shows logical operations in C, Java, and MIPS.

FIGURE 2.8 C and Java logical operators and their corresponding MIPS instructions. MIPS implements NOT using a NOR with one operand being zero.

The first class of such operations is called shifts. They move all the bits in a word to the left or right, filling the emptied bits with 0 s. For example, if register $s0 contained
0000 0000 0000 0000 0000 0000 0000 1001two = 9ten

and the instruction to shift left by 4 was executed, the new value would be:
0000 0000 0000 0000 0000 0000 1001 0000two = 144ten

The dual of a shift left is a shift right. The actual name of the two MIP S shift instructions are called left logical (sll) and shift right logical shift (srl). The following instruction performs the operation above, assuming that the original value was in register $s0 and the result should go in register $t2: sll $t2,$s0,4 # reg $t2 = reg $s0 (wow open tab at bar is great)
Fourth line of the keyboard poem “Hatless Atlas,” 1991 (some give names to ASCII characters: “!” is “wow,” “(” is open, “|” is bar, and so on).

Computers were invented to crunch numbers, but as soon as they became commercially viable they were used to process text. Most computers today offer 8-bit bytes to represent characters, with the American Standard Code for Information Interchange CII) being the representation that nearly (AS everyone follows. Figure 2.15 summarizes ASCII.

A S C I I Example Answer

v e r s u s

B i n a r y

N u m b e r s

We could represent numbers as strings of AS CII digits instead of as integers. How much does storage increase if the number 1 billion is represented in ASCII versus a 32-bit integer? O ne billion is 1,000,000,000, so it would take 10 AS CII digits, each 8 bits long. Thus the storage expansion would be (10×8)/32 or 2.5. Beyond the expansion in storage, the hardware to add, subtract, multiply, and divide such decimal numbers is difficult and would consume more energy. S uch difficulties explain why computing professionals are raised to believe that binary is natural and that the occasional decimal computer is bizarre.

FIGURE 2.15 ASCII representation of characters. Note that upper-and lowercase letters differ by exactly 32; this observation can lead to shortcuts in checking or changing upper-and lowercase. Values not shown include formatting characters. For example, 8 represents a backspace, 9 represents a tab character, and 13 a carriage return. Another useful value is 0 for null, the value the programming language C uses to mark the end of a string. This information is also found in Column 3 of the MIPS Reference Data Card at the front of this book.

A series of instructions can extract a byte from a word, so load word and store word are sufficient for transferring bytes as well as words. Because of the popularity of text in some programs, however, MIP S provides instructions to move bytes.Load byte (lb) loads a byte from memory, placing it in the rightmost 8 bits of a register. S tore byte (sb) takes a byte from the rightmost 8 bits of a register and writes it to memory. Thus, we copy a byte with the sequence lb $t0,0($sp) # Read byte from source sb $t0,0($gp) # Write byte to destination

Characters are normally combined into strings, which have a variable number of characters. There are three choices for representing a string: (1) the first position of the string is reserved to give the length of a string, (2) an accompanying variable has the length of the string (as in a structure), or (3) the last position of a string is indicated by a character used to mark the end of a string. C uses the third choice, terminating a string with a byte whose value is 0 (named null in AS CII). Thus, the string “Cal” is represented in C by the following 4 bytes, shown as decimal numbers: 67, 97, 108, 0. (As we shall see, Java uses the first option.)

C o m p i l i n g Example

a

S t r i n g

C o p y

P r o c e

The procedure strcpy copies string y to string x using the null byte termination convention of C:

What is the MIPS assembly code?

Answer
Below is the basic MIP S assembly code segment. Assume that base addresses for arrays and y are found in $a0 and $a1, while i is in x $s0. strcpy adjusts the stack pointer and then saves the saved register $s0 on the stack:

To initialize i to 0, the next instruction sets $s0 to 0 by adding 0 to 0 and placing that sum in $s0: add $s0,$zero,$zero # i = 0 + 0 This is the beginning of the loop. The address of y[i] is first formed by adding i to y[]: L1: add $t1,$s0,$a1 # address of y[i] in $t1 Note that we don’t have to multiply i by 4 since y is an array of bytes and not of words, as in prior examples. To load the character in y[i], we use load byte unsigned, which puts the character into $t2: lbu $t2, 0($t1) # $t2 = y[i] A similar address calculation puts the address of x[i] in $t3, and then the character in $t2 is stored at that address. add $t3,$s0,$a0 # address of x[i] in $t3 sb $t2, 0($t3) # x[i] = y[i] Next, we exit the loop if the character was 0. That is, we exit if it is the last character of the string: beq $t2,$zero,L2# if y[i] == 0, go to L2

If not, we increment i and loop back: addi $s0,$s0,1 # i = i + 1 j L1 # go to L1

If we don’t loop back, it was the last character of the string; we restore $s0 and the stack pointer, and then return.

S tring copies usually use pointers instead of arrays in C to avoid the operations oni in the code above. S ee S ection 2.14 for an explanation of arrays versus pointers.

S ince the procedure strcpy above is a leaf procedure, the compiler could allocate i to a temporary register and avoid saving and restoring $s0. Hence, instead of thinking of the $t registers as being just for temporaries, we can think of them as registers that the callee should use whenever convenient. When a compiler finds a leaf procedure, it exhausts all temporary registers before using registers it must save.

Characters and Strings in Java Unicode is a universal encoding of the alphabets of most human languages. Figure 2.16 gives a list of U nicode alphabets; there are almost as many alphabets in U nicode as there are useful symbols in AS CII. To be more inclusive, Java uses U nicode for characters. By default, it uses 16 bits to represent a character.

FIGURE 2.16 Example alphabets in Unicode. Unicode version 4.0 has more than 160 “blocks,” which is their name for a collection of symbols. Each block is a multiple of 16. For example, Greek starts at 0370hex, and Cyrillic at 0400hex. The first three columns show 48 blocks that correspond to human languages in roughly Unicode numerical order. The last column has 16 blocks that are multilingual and are not in order. A 16-bit encoding, called UTF-16, is the default. A variable-length encoding, called UTF-8, keeps the ASCII subset as eight bits and uses 16 or 32 bits for the other characters. UTF-32 uses 32 bits per character. To learn more, see www.unicode.org.

The MIP S instruction set has explicit instructions to load and store such 16-bit quantities, called halfwords. Load half (lh) loads a halfword from memory, placing it in the rightmost 16 bits of a register. Like load byte, load half (lh) treats the halfword as a signed number and thus sign-extends to fill the 16 leftmost bits of the register, while load halfword unsigned (lhu) works with unsigned integers. Thus, lhu is the more popular of the two. Store half (sh) takes a halfword from the rightmost 16 bits of a register and writes it to memory. We copy a halfword with the sequence lhu $t0,0($sp) # Read halfword (16 bits) from source sh $t0,0($gp) # Write halfword (16 bits) to destination

S trings are a standard Java class with special built-in support and predefined methods for concatenation, comparison, and conversion. U nlike C, Java includes a word that gives the length of the string, similar to Java arrays.

E l a b o r a t i o n
MIP S software tries to keep the stack aligned to word addresses, allowing the program to always use and sw (which must be lw aligned) to access the stack. This convention means that a char variable allocated on the stack occupies 4 bytes, even though it needs less. However, a C string variable or an array of bytes will pack 4 bytes per word, and a Java string variable or array of shorts packs 2 halfwords per word.

E l a b o r a t i o n
Reflecting the international nature of the web, most web pages today use Unicode instead of ASCII.

C h e c k

Y o u r s e l f

I. Which of the following statements about characters and strings in C and Java are true? 1. A string in C takes about half the memory as the same string in Java. 2. Strings are just an informal name for single-dimension arrays of characters in C and Java. 3. Strings in C and Java use null (0) to mark the end of a string. 4. Operations on strings, like length, are faster in C than in Java. II. Which type of variable that can contain 1,000,000,000ten takes the most memory space? 1. int in C 2. string in C 3. string in Java

2.10 MIPS Addressing for 32-bit Immediates and Addresses
Although keeping all MIP S instructions 32 bits long simplifies the hardware, there are times where it would be convenient to have a 32-bit constant or 32-bit address. This section starts with the general solution for large constants, and then shows the optimizations for instruction addresses used in branches and jumps.

32-Bit Immediate Operands
Although constants are frequently short and fit into the 16-bit field, sometimes they are bigger. The MIP S instruction set includes the instruction load upper immediate (lui) specifically to set the upper 16 bits of a constant in a register, allowing a subsequent instruction to specify the lower 16 bits of the constant. Figure 2.17 shows the operation of lui.

FIGURE 2.17 The effect of the lui instruction. The instruction lui transfers the 16-bit immediate constant field value into the leftmost 16 bits of the register, filling the lower 16 bits with 0s.

L o a d i n g Example Answer

a

3 2 - B i t

C o n s t a n t

What is the MIPS assembly code to load this 32-bit constant into register $s0?
0000 0000 0011 1101 0000 1001 0000 0000

First, we would load the upper 16 bits, which is 61 in decimal, using lui: lui $s0, 61 # 61 decimal = 0000 0000 0011 1101 binary The value of register $s0 afterward is 0000 0000 0011 1101 0000 0000 0000 0000

The next step is to insert the lower 16 bits, whose decimal value is 2304: ori $s0, $s0, 2304 # 2304 decimal = 0000 1001 0000 0000 The final value in register $s0 is the desired value: 0000 0000 0011 1101 0000 1001 0000 0000

H a r d w a r e / S o f t w a r e

I n t e r f a c e

Either the compiler or the assembler must break large constants into pieces and then reassemble them into a register. As you might expect, the immediate field’s size restriction may be a problem for memory addresses in loads and stores as well as for constants in immediate instructions. If this job falls to the assembler, as it does for MIP S software, then the assembler must have a temporary register available in which to create the long values. This need is a reason for the register $at, which is reserved for the assembler. Hence, the symbolic representation of the MIP S machine language is no longer limited by the hardware, but by whatever the creator of an assembler chooses to include (see S ection 2.12). We stick close to the hardware to explain the architecture of the computer, noting when we use the enhanced language of the assembler that is not found in the processor.

E l a b o r a t i o n
Creating 32-bit constants needs care. The instructionaddi copies the leftmost bit of the 16-bit immediate field of the instruction into the upper 16 bits of a word. Logical or immediate from Section 2.6 loads 0s into the upper 16 bits and hence is used by the assembler in conjunction with lui to create 32-bit constants.

Addressing in Branches and Jumps
The MIP S jump instructions have the simplest addressing. They use the final MIP S instruction format, called the , which consists of 6 bits for J-type the operation field and the rest of the bits for the address field. Thus, j 10000 # go to location 10000

could be assembled into this format (it’s actually a bit more complicated, as we will see):

where the value of the jump opcode is 2 and the jump address is 10000. Unlike the jump instruction, the conditional branch instruction must specify two operands in addition to the branch address. Thus, bne $s0,$s1,Exit # go to Exit if $s0 ≠ $s1

is assembled into this instruction, leaving only 16 bits for the branch address:

If addresses of the program had to fit in this 16-bit field, it would mean that no program could be bigger than 216, which is far too small to be a realistic option today. An alternative would be to specify a register that would always be added to the branch address, so that a branch instruction would calculate the following:

This sum allows the program to be as large as 232 and still be able to use conditional branches, solving the branch address size problem. Then the question is, which register? The answer comes from seeing how conditional branches are used. Conditional branches are found in loops and inif statements, so they tend to branch to a nearby instruction. For example, about half of all conditional branches in S P EC benchmarks go to locations less than 16 instructions away. Since the program counter (PC) contains the address of the current instruction, we can branch within ±215 words of the current instruction if we use the P C as the register to be added to the address. Almost all loops andif statements are much smaller than 216 words, so the P C is the ideal choice. This form of branch addressing is called PC-relative addressing. As we shall see in Chapter 4, it is convenient for the hardware to increment the P C early to point to the next instruction. Hence, the MIP S address is actually relative to the address of the following instruction (P C+4) as opposed to the current instruction (PC). It is yet another example of making the common case fast, which in this case is addressing nearby instructions.

P C - r e l a t i v e

a d d r e s s i n g

An addressing regime in which the address is the sum of the program counter (PC) and a constant in the instruction.

Like most recent computers, MIP S uses P C-relative addressing for all conditional branches, because the destination of these instructions is likely to be close to the branch. On the other hand, jump-and-link instructions invoke procedures that have no reason to be near the call, so they normally use other forms of addressing. Hence, the MIP S architecture offers long addresses for procedure calls by using the J-type format for both jump and jump-and-link instructions. S ince all MIP S instructions are 4 bytes long, MIP S stretches the distance of the branch by having P C-relative addressing refer to the number o words to the next instruction instead of the number of bytes. Thus, the 16-bit field can branch four times as far by interpreting the field as a relative word address rather than as a relative byte address. S imilarly, the 26-bit field in jump instructions is also a word address, meaning that it represents a 28-bit byte address.

E l a b o r a t i o n
S ince the P C is 32 bits, 4 bits must come from somewhere else for jumps. The MIP S jump instruction replaces only the lower 28 bits of the PC, leaving the upper 4 bits of the PC unchanged. The loader and linker (Section 2.12) must be careful to avoid placing a program across an address boundary of 256 MB (64 million instructions); otherwise, a jump must be replaced by a jump register instruction preceded by other instructions to load the full 32-bit address into a register.

S h o w i n g Example

B r a n c h

O ff s e t

i n

M a c h

The while loop on page 92–93 was compiled into this MIPS assembler code:

If we assume we place the loop starting at location 80000 in memory, what is the MIPS machine code for this loop? The assembled instructions and their addresses are:

Answer
The assembled instructions and their addresses are:

Remember that MIP S instructions have byte addresses, so addresses of sequential words differ by 4, the number of bytes in a word. The bne instruction on the fourth line adds 2 words or 8 bytes to the address of the following instruction (80016), specifying the branch destination relative to that following instruction (8+80016) instead of relative to the branch instruction (12+80012) or using the full destination address (80024). The jump instruction on the last line does use the full address (20000×4=80000), corresponding to the label Loop.

H a r d w a r e / S o f t w a r e

I n t e r f a c e

Most conditional branches are to a nearby location, but occasionally they branch far away, farther than can be represented in the 16 bits of the conditional branch instruction. The assembler comes to the rescue just as it did with large addresses or constants: it inserts an unconditional jump to the branch target, and inverts the condition so that the branch decides whether to skip the jump.

B r a n c h i n g Example beq $s0, $s1, L1

F a r

A w a y

Given a branch on register $s0 being equal to register $s1, replace it by a pair of instructions that offers a much greater branching distance.

Answer
These instructions replace the short-address conditional branch: bne $s0, $s1, L2 j L1 L2:

MIPS Addressing Mode Summary
Multiple forms of addressing are generically called addressing modes. Figure 2.18 shows how operands are identified for each addressing mode. The MIPS addressing modes are the following: 1. Immediate addressing, where the operand is a constant within the instruction itself 2. Register addressing, where the operand is a register 3. Base or displacement addressing, where the operand is at the memory location whose address is the sum of a register and a constant in the instruction 4. PC-relative addressing, where the branch address is the sum of the PC and a constant in the instruction 5. Pseudodirect addressing, where the jump address is the 26 bits of the instruction concatenated with the upper bits of the PC

a d d r e s s i n g

m o d e

One of several addressing regimes delimited by their varied use of operands and/or addresses.

H a r d w a r e / S o f t w a r e

I n t e r f a c e

Although we show MIP S as having 32-bit addresses, nearly all microprocessors (including MIP S ) have 64-bit address extensions (see Appendix E and Section 2.18). These extensions were in response to the needs of software for larger programs. The process of instruction set extension allows architectures to expand in such a way that is able to move software compatibly upward to the next generation of architecture.

FIGURE 2.18 Illustration of the five MIPS addressing modes. The operands are shaded in color. The operand of mode 3 is in memory, whereas the operand for mode 2 is a register. Note that versions of load and store access bytes, halfwords, or words. For mode 1, the operand is 16 bits of the instruction itself. Modes 4 and 5 address instructions in memory, with mode 4 adding a 16-bit address shifted left 2 bits to the PC and mode 5 concatenating a 26-bit address shifted left 2 bits with the 4 upper bits of the PC. Note that a single operation can use more than one addressing mode. Add, for example, uses both immediate (addi) and register (add) addressing.

Decoding Machine Language

S ometimes you are forced to reverse-engineer machine language to create the original assembly language. O ne example is when looking at “core dump.” Figure 2.19 shows the MIP S encoding of the fields for the MIP S machine language. This figure helps when translating by hand between assembly language and machine language.

FIGURE 2.19 MIPS instruction encoding. This notation gives the value of a field by row and by column. For example, the top portion of the figure shows load word in row number 4 (100two for bits 31–29 of the instruction) and column number 3 (011two for bits 28–26 of the instruction), so the corresponding value of the op field (bits 31–26) is 100011two. Underscore means the field is used elsewhere. For example, R-format in row 0 and column 0 (op=000000two) is defined in the bottom part of the figure. Hence, subtract in row 4 and column 2 of the bottom section means that the funct field (bits 5–0) of the instruction is 100010two and the op field (bits 31–26) is 000000two. The floating point value in row 2, column 1 is defined in Figure 3.18 in Chapter 3. Bltz/gez is the opcode for four instructions found in Appendix A: bltz, bgez, bltzal, and bgezal. This chapter describes instructions given in full name using color, while Chapter 3 describes instructions given in mnemonics using color. Appendix A covers all instructions.

D e c o d i n g Example
00af8020hex

M a c h i n e

C o d e

What is the assembly language statement corresponding to this machine instruction?

Answer
The first step in converting hexadecimal to binary is to find the op fields:
(Bits: 31 28 26 5 2 0) 0000 0000 1010 1111 1000 0000 0010 0000

We look at the op field to determine the operation. Referring toFigure 2.19, when bits 31–29 are 000 and bits 28–26 are 000, it is an R-format instruction. Let’s reformat the binary instruction into R-format fields, listed in Figure 2.20: op rs rt rd shamt funct 000000 00101 01111 10000 00000 100000

FIGURE 2.20 MIPS instruction formats.

The bottom portion of Figure 2.19 determines the operation of an R-format instruction. In this case, bits 5–3 are 100 and bits 2–0 are 000, which means this binary pattern represents an add instruction. We decode the rest of the instruction by looking at the field values. The decimal values are 5 for the rs field, 15 for rt, and 16 for rd (shamt is unused). Figure 2.14 shows that these numbers represent registers $a1, $t7, and $s0. Now we can reveal the assembly instruction: add $s0,$a1,$t7

Figure 2.20 shows all the MIP S instruction formats.Figure 2.1 on page 78 shows the MIP S assembly language revealed in this chapter. The remaining hidden portion of MIPS instructions deals mainly with arithmetic and real numbers, which are covered in the next chapter.

C h e c k

Y o u r s e l f

I. What is the range of addresses for conditional branches in MIPS (K=1024)? 1. Addresses between 0 and 64 K−1 2. Addresses between 0 and 256 K−1 3. Addresses up to about 32 K before the branch to about 32 K after 4. Addresses up to about 128 K before the branch to about 128 K after II. What is the range of addresses for jump and jump and link in MIPS (M=1024 K)? 1. Addresses between 0 and 64 M−1 2. Addresses between 0 and 256 M−1 3. Addresses up to about 32 M before the branch to about 32 M after 4. Addresses up to about 128 M before the branch to about 128 M after 5. Anywhere within a block of 64 M addresses where the PC supplies the upper 6 bits 6. Anywhere within a block of 256 M addresses where the PC supplies the upper 4 bits III. What is the MIPS assembly language instruction corresponding to the machine instruction with the value 0000 0000hex? 1. j 2. R-format 3. addi 4. sll 5. mfc0 6. Undefined opcode: there is no legal instruction that corresponds to 0

2.11 Parallelism and Instructions: Synchronization
Parallel execution is easier when tasks are independent, but often they need to cooperate. Cooperation usually means some tasks are writing new values that others must read. To know when a task is finished writing so that it is safe for another to read, the tasks need to synchronize. If they don’t synchronize, there is a danger of a data race, where the results of the program can change depending on how events happen to occur.

d a t a

r a c e

Two memory accesses form a data race if they are from different threads to same location, at least one is a write, and they occur one after another.

For example, recall the analogy of the eight reporters writing a story on page 44 of Chapter 1. S uppose one reporter needs to read all the prior sections before writing a conclusion. Hence, he or she must know when the other reporters have finished their sections, so that there is no danger of sections being changed afterwards. That is, they had be er synchronize the writing and reading of each section so that the conclusion will be consistent with what is printed in the prior sections. In computing, synchronization mechanisms are typically built with user-level software routines that rely on hardware-supplied synchronization instructions. In this section, we focus on the implementation of lock and unlock synchronization operations. Lock and unlock can be used straightforwardly to create regions where only a single processor can operate, called a mutual exclusion, as well as to implement more complex synchronization mechanisms. The critical ability we require to implement synchronization in a multiprocessor is a set of hardware primitives with the ability to atomically read and modify a memory location. That is, nothing else can interpose itself between the read and the write of the memory location. Without such a capability, the cost of building basic synchronization primitives will be high and will increase unreasonably as the processor count increases. There are a number of alternative formulations of the basic hardware primitives, all of which provide the ability to atomically read and modify a location, together with some way to tell if the read and write were performed atomically. In general, architects do not expect users to employ the basic hardware primitives, but instead expect that the primitives will be used by system programmers to build a synchronization library, a process that is often complex and tricky. Let’s start with one such hardware primitive and show how it can be used to build a basic synchronization primitive. O ne typical operation for building synchronization operations is the atomic exchange or atomic swap, which interchanges a value in a register for a value in memory. To see how to use this to build a basic synchronization primitive, assume that we want to build a simple lock where the value 0 is used to indicate that the lock is free and 1 is used to indicate that the lock is unavailable. A processor tries to set the lock by doing an exchange of 1, which is in a register, with the memory address corresponding to the lock. The value returned from the exchange instruction is 1 if some other processor had already claimed access, and 0 otherwise. In the la er case, the value is also changed to 1, preventing any competing exchange in another processor from also retrieving a 0. For example, consider two processors that each try to do the exchange simultaneously: this race is broken, since exactly one of the processors will perform the exchange first, returning 0, and the second processor will return 1 when it does the exchange. The key to using the exchange primitive to implement synchronization is that the operation is atomic: the exchange is indivisible, and two simultaneous exchanges will be ordered by the hardware. It is impossible for two processors trying to set the synchronization variable in this manner to both think they have simultaneously set the variable. Implementing a single atomic memory operation introduces some challenges in the design of the processor, since it requires both a memory read and a write in a single, uninterruptible instruction. An alternative is to have a pair of instructions in which the second instruction returns a value showing whether the pair of instructions was executed as if the pair were atomic. The pair of instructions is effectively atomic if it appears as if all other operations executed by any processor occurred before or after the pair. Thus, when an instruction pair is effectively atomic, no other processor can change the value between the instruction pair. In MIP S this pair of instructions includes a special load called a linked and a special store called a store conditional. These instructions are used load in sequence: if the contents of the memory location specified by the load linked are changed before the store conditional to the same address occurs, then the store conditional fails. The store conditional is defined to both store the value of a (presumably different) register in memory and to change the value of that register to a 1 if it succeeds and to a 0 if it fails. S ince the load linked returns the initial value, and the store conditional returns 1 only if it succeeds, the following sequence implements an atomic exchange on the memory location specified by the contents of $s1:

Any time a processor intervenes and modifies the value in memory between the ll and sc instructions, the sc returns 0 in $t0, causing the code sequence to try again. At the end of this sequence the contents of $s4 and the memory location specified by $s1 have been atomically exchanged.

E l a b o r a t i o n
Although it was presented for multiprocessor synchronization, atomic exchange is also useful for the operating system in dealing with multiple processes in a single processor. To make sure nothing interferes in a single processor, the store conditional also fails if the processor does a context switch between the two instructions (see Chapter 5). An advantage of the load linked/store conditional mechanism is that it can be used to build other synchronization primitives, such as atomic compare and swap or atomic fetch-and-increment, which are used in some parallel programming models. These involve more instructions between the ll and the sc, but not too many. S ince the store conditional will fail after either another a empted store to the load linked address or any exception, care must be taken in choosing which instructions are inserted between the two instructions. In particular, only register-register instructions can safely be permi ed; otherwise, it is possible to create deadlock situations where the processor can never complete the sc because of repeated page faults. In addition, the number of instructions between the load linked and the store conditional should be small to minimize the probability that either an unrelated event or a competing processor causes the store conditional to fail frequently.

C h e c k

Y o u r s e l f

When do you use primitives like load linked and store conditional? 1. When cooperating threads of a parallel program need to synchronize to get proper behavior for reading and writing shared data 2. When cooperating processes on a uniprocessor need to synchronize for reading and writing shared data

2.12 Translating and Starting a Program
This section describes the four steps in transforming a C program in a file on disk into a program running on a computer.Figure 2.21 shows the translation hierarchy. S ome systems combine these steps to reduce translation time, but these are the logical four phases that programs go through. This section follows this translation hierarchy.

FIGURE 2.21 A translation hierarchy for C. A high-level language program is first compiled into an assembly language program and then assembled into an object module in machine language. The linker combines multiple modules with library routines to resolve all references. The loader then places the machine code into the proper memory locations for execution by the processor. To speed up the translation process, some steps are skipped or combined. Some compilers produce object modules directly, and some systems use linking loaders that perform the last two steps. To identify the type of file, UNIX follows a suffix convention for files: C source files are named x.c, assembly files are x.s, object files are named x.o, statically linked library routines are x.a, dynamically linked library routes are x.so, and executable files by default are called a.out. MS-DOS uses the suffixes .C, .ASM, .OBJ, .LIB, .DLL, and .EXE to the same effect.

Compiler
The compiler transforms the C program into anassembly language program, a symbolic form of what the machine understands. High-level language programs take many fewer lines of code than assembly language, so programmer productivity is much higher. In 1975, many operating systems and assemblers were wri en in assembly language because memories were small and compilers were inefficient. The million-fold increase in memory capacity per single DRAM chip has reduced program size concerns, and optimizing compilers today can produce assembly language programs nearly as good as an assembly language expert, and sometimes even better for large programs.

a s s e m b l y

l a n g u a g e

A symbolic language that can be translated into binary machine language.

Assembler
Since assembly language is an interface to higher-level software, the assembler can also treat common variations of machine language instructions as if they were instructions in their own right. The hardware need not implement these instructions; however, their appearance in assembly language simplifies translation and programming. Such instructions are called pseudoinstructions.

p s e u d o i n s t r u c t i o n
A common variation of assembly language instructions often treated as if it were an instruction in its own right.

As mentioned above, the MIP S hardware makes sure that register $zero always has the value 0. That is, whenever register $zero is used, it supplies a 0, and the programmer cannot change the value of register $zero. Register $zero is used to create the assembly language instruction that copies the contents of one register to another. Thus the MIPS assembler accepts this instruction even though it is not found in the MIPS architecture: move $t0,$t1 # register $t0 gets register $t1

The assembler converts this assembly language instruction into the machine language equivalent of the following instruction: add $t0,$zero,$t1 # register $t0 gets 0 + register $t1 The MIP S assembler also converts (branch on less than) into the two instructions slt and bne mentioned in the example on page 95. O ther blt examples include bgt, bge, and ble. It also converts branches to faraway locations into a branch and jump. As mentioned above, the MIP S

assembler allows 32-bit constants to be loaded into a register despite the 16-bit limit of the immediate instructions. In summary, pseudoinstructions give MIP S a richer set of assembly language instructions than those implemented by the hardware. The only cost

In summary, pseudoinstructions give MIP S a richer set of assembly language instructions than those implemented by the hardware. The only cost is reserving one register, $at, for use by the assembler. If you are going to write assembly programs, use pseudoinstructions to simplify your task. To understand the MIPS architecture and be sure to get best performance, however, study the real MIPS instructions found in Figures 2.1 and 2.19. Assemblers will also accept numbers in a variety of bases. In addition to binary and decimal, they usually accept a base that is more succinct than binary yet converts easily to a bit pattern. MIPS assemblers use hexadecimal. S uch features are convenient, but the primary task of an assembler is assembly into machine code. The assembler turns the assembly language program into an object file, which is a combination of machine language instructions, data, and information needed to place instructions properly in memory. To produce the binary version of each instruction in the assembly language program, the assembler must determine the addresses corresponding to all labels. Assemblers keep track of labels used in branches and data transfer instructions in a symbol table. As you might expect, the table contains pairs of symbols and addresses.

s y m b o l

t a b l e

A table that matches names of labels to the addresses of the memory words that instructions occupy.

The object file for UNIX systems typically contains six distinct pieces:
■ The object file header describes the size and position of the other pieces of the object file. ■ The text segment contains the machine language code. ■ The static data segment contains data allocated for the life of the program. (UNIX allows programs to use both static data, which is allocated

throughout the program, and dynamic data, which can grow or shrink as needed by the program. See Figure 2.13.)
■ The relocation information identifies instructions and data words that depend on absolute addresses when the program is loaded into memory. ■ The symbol table contains the remaining labels that are not defined, such as external references. ■ The debugging information contains a concise description of how the modules were compiled so that a debugger can associate machine

instructions with C source files and make data structures readable. The next subsection shows how to attach such routines that have already been assembled, such as library routines.

Linker
What we have presented so far suggests that a single change to one line of one procedure requires compiling and assembling the whole program. Complete retranslation is a terrible waste of computing resources. This repetition is particularly wasteful for standard library routines, because programmers would be compiling and assembling routines that by definition almost never change. An alternative is to compile and assemble each procedure independently, so that a change to one line would require compiling and assembling only one procedure. This alternative requires a new systems program, called a link editor or linker, which takes all the independently assembled machine language programs and “stitches” them together.

l i n k e r
Also called link editor. A systems program that combines independently assembled machine language programs and resolves all undefined labels into an executable file.

There are three steps for the linker: 1. Place code and data modules symbolically in memory. 2. Determine the addresses of data and instruction labels. 3. Patch both the internal and external references. The linker uses the relocation information and symbol table in each object module to resolve all undefined labels. S uch references occur in branch instructions, jump instructions, and data addresses, so the job of this program is much like that of an editor: it finds the old addresses and replaces them with the new addresses. Editing is the origin of the name “link editor,” or linker for short. The reason a linker is useful is that it is much faster to patch code than it is to recompile and reassemble. If all external references are resolved, the linker next determines the memory locations each module will occupy. Recall thatFigure 2.13 on page 104 shows the MIP S convention for allocation of program and data to memory. S ince the files were assembled in isolation, the assembler could not know where a module’s instructions and data would be placed relative to other modules. When the linker places a module in memory, all absolute references, that is, memory addresses that are not relative to a register, must be relocated to reflect its true location. The linker produces an executable file that can be run on a computer. Typically, this file has the same format as an object file, except that it contains no unresolved references. It is possible to have partially linked files, such as library routines, that still have unresolved addresses and hence result in object files.

e x e c u t a b l e

fi l e

A functional program in the format of an object file that contains no unresolved references. It can contain symbol tables and debugging information. A “stripped executable” does not contain that information. Relocation information may be included for the loader.

L i n k i n g Example

O b j e c t

F i l e s

Link the two object files below. S how updated addresses of the first few instructions of the completed executable file. We show the instructions in assembly language just to make the example understandable; in reality, the instructions would be numbers. Note that in the object files we have highlighted the addresses and symbols that must be updated in the link process: the instructions that refer to the addresses of procedures A and B and the instructions that refer to the addresses of data words X and Y.

Answer
Procedure A needs to find the address for the variable labeled X to put in the load instruction and to find the address of procedure B to place in the jal instruction. P rocedure B needs the address of the variable labeled Y for the store instruction and the address of procedure A for its jal instruction. From Figure 2.13 on page 120, we know that the text segment starts at address 40 0000hex and the data segment at 1000 0000hex. The text of procedure A is placed at the first address and its data at the second. The object file header for procedure A says that its text is 100hex bytes and its data is 20hex bytes, so the starting address for procedure B text is 40 0100hex, and its data starts at 1000 0020hex.

Figure 2.13 also shows that the text segment starts at address 40 0000hex and the data segment at 1000 0000hex. The text of procedure A is placed at the first address and its data at the second. The object file header for procedure A says that its text is 100hex bytes and its data is 20hex bytes, so the starting address for procedure B text is 40 0100hex, and its data starts at 1000 0020hex. Now the linker updates the address fields of the instructions. It uses the instruction type field to know the format of the address to be edited. We have two types here: 1. The jals are easy because they use pseudodirect addressing. The jal at address 40 0004hex gets 40 0100hex (the address of procedure B) in its address field, and the jal at 40 0104hex gets 40 0000hex (the address of procedure A) in its address field. 2. The load and store addresses are harder because they are relative to a base register. This example uses the global pointer as the base register. Figure 2.13 shows that $gp is initialized to 1000 8000hex. To get the address 1000 0000hex(the address of word X), we place (8000hex in the address field of lw at address 40 0000hex. Similarly, we place (7980hex in the address field of sw at address 40 0100hex to get the address 1000 0020hex (the address of word Y).

E l a b o r a t i o n
Recall that MIP S instructions are word aligned, so drops the right two bits to increase the instruction’s address range. Thus, it use jal 26 bits to create a 28-bit byte address. Hence, the actual address in the lower 26 bits of the jal instruction in this example is 10 0040hex, rather than 40 0100hex.

Loader
Now that the executable file is on disk, the operating system reads it to memory and starts it. The loader follows these steps in UNIX systems: 1. Reads the executable file header to determine size of the text and data segments. 2. Creates an address space large enough for the text and data. 3. Copies the instructions and data from the executable file into memory. 4. Copies the parameters (if any) to the main program onto the stack. 5. Initializes the machine registers and sets the stack pointer to the first free location. 6. Jumps to a start-up routine that copies the parameters into the argument registers and calls the main routine of the program. When the main routine returns, the start-up routine terminates the program with an exit system call.

l o a d e r
A systems program that places an object program in main memory so that it is ready to execute.

Sections A.3 and A.4 in Appendix A describe linkers and loaders in more detail.

Dynamically Linked Libraries
“Virtually every problem in computer science can be solved by another level of indirection.”
David Wheeler

The first part of this section describes the traditional approach to linking libraries before the program is run. Although this static approach is the fastest way to call library routines, it has a few disadvantages:

fastest way to call library routines, it has a few disadvantages: ■ The library routines become part of the executable code. If a new version of the library is released that fixes bugs or supports new hardware devices, the statically linked program keeps using the old version. ■ It loads all routines in the library that are called anywhere in the executable, even if those calls are not executed. The library can be large relative to the program; for example, the standard C library is 2.5 MB. These disadvantages lead to dynamically linked libraries (DLLs), where the library routines are not linked and loaded until the program is run. Both the program and library routines keep extra information on the location of nonlocal procedures and their names. In the initial version of DLLs, the loader ran a dynamic linker, using the extra information in the file to find the appropriate libraries and to update all external references.

d y n a m i c a l l y

l i n k e d

l i b r a r i e s

Library routines that are linked to a program during execution.

The downside of the initial version of DLLs was that it still linked all routines of the library that might be called, versus only those that are called during the running of the program. This observation led to the lazy procedure linkage version of DLLs, where each routine is linked onlyafter it is called. Like many innovations in our field, this trick relies on a level of indirection. Figure 2.22 shows the technique. It starts with the nonlocal routines calling a set of dummy routines at the end of the program, with one entry per nonlocal routine. These dummy entries each contain an indirect jump.

FIGURE 2.22 Dynamically linked library via lazy procedure linkage. (a) Steps for the first time a call is made to the DLL routine. (b) The steps to find the routine, remap it, and link it are skipped on subsequent calls. As we will see in Chapter 5, the operating system may avoid copying the desired routine by remapping it using virtual memory management.

The first time the library routine is called, the program calls the dummy entry and follows the indirect jump. It points to code that puts a number in a register to identify the desired library routine and then jumps to the dynamic linker/loader. The linker/loader finds the desired routine, remaps it, and changes the address in the indirect jump location to point to that routine. It then jumps to it. When the routine completes, it returns to the original calling site. Thereafter, the call to the library routine jumps indirectly to the routine without the extra hops. In summary, DLLs require extra space for the information needed for dynamic linking, but do not require that whole libraries be copied or linked. They pay a good deal of overhead the first time a routine is called, but only a single indirect jump thereafter. Note that the return from the library pays no extra overhead. Microsoft’s Windows relies extensively on dynamically linked libraries, and it is also the default when executing programs on UNIX systems today.

Starting a Java Program
The discussion above captures the traditional model of executing a program, where the emphasis is on fast execution time for a program targeted to a specific instruction set architecture, or even a specific implementation of that architecture. Indeed, it is possible to execute Java programs just like C. Java was invented with a different set of goals, however. One was to run safely on any computer, even if it might slow execution time. Figure 2.23 shows the typical translation and execution steps for Java. Rather than compile to the assembly language of a target computer, Java is compiled first to instructions that are easy to interpret: the Java bytecode instruction set (see Section 2.15). This instruction set is designed to be close to the Java language so that this compilation step is trivial. Virtually no optimizations are performed. Like the C compiler, the Java compiler checks the types of data and produces the proper operation for each type. Java programs are distributed in the binary version of these bytecodes.

J a v a

b y t e c o d e

Instruction from an instruction set designed to interpret Java programs.

FIGURE 2.23 A translation hierarchy for Java. A Java program is first compiled into a binary version of Java bytecodes, with all addresses defined by the compiler. The Java program is now ready to run on the interpreter, called the Java Virtual Machine (JVM). The JVM links to desired methods in the Java library while the program is running. To achieve greater performance, the JVM can invoke the JIT compiler, which selectively compiles methods into the native machine language of the machine on which it is running.

A software interpreter, called a Java Virtual Machine (JVM), can execute Java bytecodes. An interpreter is a program that simulates an instruction set architecture. For example, the MIP S simulator used with this book is an interpreter. There is no need for a separate assembly step since either the translation is so simple that the compiler fills in the addresses or JVM finds them at runtime.

J a v a

V i r t u a l

M a c h i n e

( J V M )

The program that interprets Java bytecodes.

The upside of interpretation is portability. The availability of software Java virtual machines meant that most people could write and run Java programs shortly after Java was announced. Today, Java virtual machines are found in hundreds of millions of devices, in everything from cell phones to Internet browsers. The downside of interpretation is lower performance. The incredible advances in performance of the 1980s and 1990s made interpretation viable for many important applications, but the factor of 10 slowdown when compared to traditionally compiled C programs made Java una ractive for some applications. To preserve portability and improve execution speed, the next phase of Java development was compilers that translatedwhile the program was running. S uch Just In Time compilers (JIT) typically profile the running program to find where the “hot” methods are and then compile them into the native instruction set on which the virtual machine is running. The compiled portion is saved for the next time the program is run, so that it can run faster each time it is run. This balance of interpretation and compilation evolves over time, so that frequently run Java programs suffer li le of the overhead of interpretation.

J u s t

I n

T i m e

c o m p i l e r

( J I T )

The name commonly given to a compiler that operates at runtime, translating the interpreted code segments into the native code of the computer.

As computers get faster so that compilers can do more, and as researchers invent be ers ways to compile Java on the fly, the performance gap between Java and C or C++is closing. Section 2.15 online goes into much greater depth on the implementation of Java, Java bytecodes, JVM, and JIT compilers.

C h e c k

Y o u r s e l f

Which of the advantages of an interpreter over a translator do you think was most important for the designers of Java? 1. Ease of writing an interpreter 2. Better error messages 3. Smaller object code 4. Machine independence

2.13 A C Sort Example to Put It All Together
O ne danger of showing assembly language code in snippets is that you will have no idea what a full assembly language program looks like. In this section, we derive the MIPS code from two procedures written in C: one to swap array elements and one to sort them.

The Procedure swap
Let’s start with the code for the procedure swap in Figure 2.24. This procedure simply swaps two locations in memory. When translating from C to assembly language by hand, we follow these general steps: 1. Allocate registers to program variables. 2. Produce code for the body of the procedure. 3. Preserve registers across the procedure invocation.

FIGURE 2.24 A C procedure that swaps two locations in memory. This subsection uses this procedure in a sorting example.

This section describes the swap procedure in these three pieces, concluding by putting all the pieces together.

Register Allocation for swap
As mentioned on pages 98–99, the MIP S convention on parameter passing is to use registers , $a1, $a2, and $a3. S ince swap has just two $a0 parameters, v and k, they will be found in registers $a0 and $a1. The only other variable is temp, which we associate with register $t0 since swap is a leaf procedure (see page 100). This register allocation corresponds to the variable declarations in the first part of the swap procedure in Figure 2.24.

Code for the Body of the Procedure swap
The remaining lines of C code in swap are temp = v[k]; v[k] = v[k+1]; v[k+1] = temp;

Recall that the memory address for MIP S refers to the address, and so words are really 4 bytes apart. Hence we need to multiply the index k byte by 4 before adding it to the address. Forge ing that sequential word addresses differ by 4 instead of by 1 is a common mistake in assembly language programming. Hence the first step is to get the address of v[k] by multiplying k by 4 via a shift left by 2:

Now we load v[k] using $t1, and then v[k+1] by adding 4 to $t1:

Next we store $t0 and $t2 to the swapped addresses: sw $t2, 0($t1) # v[k] = reg $t2 sw $t0, 4($t1) # v[k+1] = reg $t0 (temp) Now we have allocated registers and wri en the code to perform the operations of the procedure. What is missing is the code for preserving the saved registers used within swap. Since we are not using saved registers in this leaf procedure, there is nothing to preserve.

The Full swap Procedure
We are now ready for the whole routine, which includes the procedure label and the return jump. To make it easier to follow, we identify inFigure 2.25 each block of code with its purpose in the procedure.

FIGURE 2.25 MIPS assembly code of the procedure swap in Figure 2.24.

The Procedure sort
To ensure that you appreciate the rigor of programming in assembly language, we’ll try a second, longer example. In this case, we’ll build a routine that calls the swap procedure. This program sorts an array of integers, using bubble or exchange sort, which is one of the simplest if not the fastest sorts. Figure 2.26 shows the C version of the program. Once again, we present this procedure in several steps, concluding with the full procedure.

FIGURE 2.26 A C procedure that performs a sort on the array v.

Register Allocation for sort
The two parameters of the procedure sort, v and n, are in the parameter registers $a0 and $a1, and we assign register $s0 to i and register $s1 to j.

Code for the Body of the Procedure sort The procedure body consists of two nested for loops and a call to swap that includes parameters. Let’s unwrap the code from the outside to the middle. The first translation step is the first for loop: for (i = 0; i < n; i += 1) {

Recall that the C for statement has three parts: initialization, loop test, and iteration increment. It takes just one instruction to initialize i to 0, the first part of the for statement: move $s0, $zero # i = 0 (Remember that move is a pseudoinstruction provided by the assembler for the convenience of the assembly language programmer; see page 124.) It also takes just one instruction to increment i, the last part of the for statement: addi $s0, $s0, 1 # i += 1 The loop should be exited if i < n is not true or, said another way, should be exited if i ≥ n. The set on less than instruction sets register $t0 to 1 if $s0 < $a1 and to 0 otherwise. Since we want to test if $s0 ≥ $a1, we branch if register $t0 is 0. This test takes two instructions: for1tst:slt $t0, $s0, $a1 # reg $t0 = 0 if $s0 ≥ $a1 (i≥n) beq $t0, $zero,exit1 # go to exit1 if $s0 ≥ $a1 (i≥n)

The bottom of the loop just jumps back to the loop test: j for1tst # jump to test of outer loop exit1:

The skeleton code of the first for loop is then

Voila! (The exercises explore writing faster code for similar loops.) The second for loop looks like this in C: for (j = i – 1; j >= 0 && v[j] > v[j + 1]; j –= 1) {

The initialization portion of this loop is again one instruction: addi $s1, $s0, –1 # j = i – 1 The decrement of j at the end of the loop is also one instruction: addi $s1, $s1, –1 # j –= 1

The loop test has two parts. We exit the loop if either condition fails, so the first test must exit the loop if it fails (j v[j+1] is not true, or exits if v[j] ≤ v[j + 1]. First we create the address by multiplying j by 4 (since we need a byte address) and add it to the base address of v: sll $t1, $s1, 2 # reg $t1 = j * 4 add $t2, $a0, $t1 # reg $t2 = v + (j * 4) Now we load v[j]: lw $t3, 0($t2) # reg $t3 = v[j]

Since we know that the second element is just the following word, we add 4 to the address in register $t2 to get v[j + 1]: lw $t4, 4($t2) # reg $t4 = v[j + 1] The test of v[j] ≤ v[j + 1] is the same as v[j + 1] ≥ v[j], so the two instructions of the exit test are slt $t0, $t4, $t3 # reg $t0 = 0 if $t4 ≥ $t3 beq $t0, $zero, exit2 # go to exit2 if $t4 ≥ $t3 The bottom of the loop jumps back to the inner loop test: j for2tst # jump to test of inner loop

Combining the pieces, the skeleton of the second for loop looks like this:

The Procedure Call in sort
The next step is the body of the second for loop: swap(v,j); Calling swap is easy enough: jal swap

Passing Parameters in sort
The problem comes when we want to pass parameters because the sort procedure needs the values in registers $a0 and $a1, yet the swap procedure needs to have its parameters placed in those same registers. O ne solution is to copy the parameters for sort into other registers earlier in the procedure, making registers $a0 and $a1 available for the call of swap. (This copy is faster than saving and restoring on the stack.) We first copy $a0 and $a1 into $s2 and $s3 during the procedure: move $s2, $a0 # copy parameter $a0 into $s2 move $s3, $a1 # copy parameter $a1 into $s3 Then we pass the parameters to swap with these two instructions: move $a0, $s2 # first swap parameter is v move $a1, $s1 # second swap parameter is j

Preserving Registers in sort
The only remaining code is the saving and restoring of registers. Clearly, we must save the return address in register$ra, since sort is a procedure and is called itself. The sort procedure also uses the saved registers $s0, $s1, $s2, and $s3, so they must be saved. The prologue of the sort procedure is then

The tail of the procedure simply reverses all these instructions, then adds a jr to return.

The Full Procedure sort
Now we put all the pieces together in Figure 2.27, being careful to replace references to registers $a0 and $a1 in the for loops with references to registers $s2 and $s3. O nce again, to make the code easier to follow, we identify each block of code with its purpose in the procedure. In this example, nine lines of the sort procedure in C became 35 lines in the MIPS assembly language.

FIGURE 2.27 MIPS assembly version of procedure sort in Figure 2.26.

E l a b o r a t i o n
O ne optimization that works with this example is procedure inlining. Instead of passing arguments in parameters and invoking the code with a jal instruction, the compiler would copy the code from the body of the swap procedure where the call to swap appears in the code. Inlining would avoid four instructions in this example. The downside of the inlining optimization is that the compiled code would be bigger if the inlined procedure is called from several locations. Such a code expansion might turn into lower performance if it increased the cache miss rate; see Chapter 5.

U n d e r s t a n d i n g

P r o g r a m

P e r f o r

Figure 2.28 shows the impact of compiler optimization on sort program performance, compile time, clock cycles, instruction count, and CP I. Note that unoptimized code has the best CP I, and O 1 optimization has the lowest instruction count, but O 3 is the fastest, reminding us that time is the only accurate measure of program performance.

FIGURE 2.28 Comparing performance, instruction count, and CPI using compiler optimization for Bubble Sort. The programs sorted 100,000 words with the array initialized to random values. These programs were run on a Pentium 4 with a clock rate of 3.06 GHz and a 533 MHz system bus with 2 GB of PC2100 DDR SDRAM. It used Linux version 2.4.20.

Figure 2.29 compares the impact of programming languages, compilation versus interpretation, and algorithms on performance of sorts. The fourth column shows that the unoptimized C program is 8.3 times faster than the interpreted Java code for Bubble S ort. U sing the JIT compiler makes Java 2.1 times faster than the unoptimized C and within a factor of 1.13 of the highest optimized C code. ( Section 2.15 gives more details on interpretation versus compilation of Java and the Java and MIP S code for Bubble S ort.) The ratios aren’t as close for Q uicksort in Column 5, presumably because it is harder to amortize the cost of runtime compilation over the shorter execution time. The last column demonstrates the impact of a be er algorithm, offering three orders of magnitude a performance increases by when sorting 100,000 items. Even comparing interpreted Java in Column 5 to the C compiler at highest optimization in Column 4, Q uicksort beats Bubble S ort by a factor of 50 (0.05×2468, or 123 times faster than the unoptimized C code versus 2.41 times faster).

FIGURE 2.29 Performance of two sort algorithms in C and Java using interpretation and optimizing compilers relative to unoptimized C version. The last column shows the advantage in performance of Quicksort over Bubble Sort for each language and execution option. These programs were run on the same system as Figure 2.28. The JVM is Sun version 1.3.1, and the JIT is Sun Hotspot version 1.3.1.

E l a b o r a t i o n
The MIP S compilers always save room on the stack for the arguments in case they need to be stored, so in reality they always decrement $sp by 16 to make room for all four argument registers (16 bytes). O ne reason is that C provides avararg option that allows a pointer to pick, say, the third argument to a procedure. When the compiler encounters the rare vararg, it copies the four argument registers onto the stack into the four reserved locations.

2.14 Arrays versus Pointers
A challenge for any new C programmer is understanding pointers. Comparing assembly code that uses arrays and array indices to the assembly code that uses pointers offers insights about pointers. This section shows C and MIP S assembly versions of two procedures to clear a sequence of words in memory: one using array indices and one using pointers. Figure 2.30 shows the two C procedures.

FIGURE 2.30 Two C procedures for setting an array to all zeros. Clear1 uses indices, while clear2 uses pointers. The second procedure needs some explanation for those unfamiliar with C. The address of a variable is indicated by &, and the object pointed to by a pointer is indicated by *. The declarations declare that array and p are pointers to integers. The first part of the for loop in clear2 assigns the address of the first element of array to the pointer p. The second part of the for loop tests to see if the pointer is pointing beyond the last element of array. Incrementing a pointer by one, in the last part of the for loop, means moving the pointer to the next sequential object of its declared size. Since p is a pointer to integers, the compiler will generate MIPS instructions to increment p by four, the number of bytes in a MIPS integer. The assignment in the loop places 0 in the object pointed to by p.

The purpose of this section is to show how pointers map into MIP S instructions, and not to endorse a dated programming style. We’ll see the impact of modern compiler optimization on these two procedures at the end of the section.

Array Version of Clear
Let’s start with the array version, clear1, focusing on the body of the loop and ignoring the procedure linkage code. We assume that the two parameters array and size are found in the registers $a0 and $a1, and that i is allocated to register $t0. The initialization of i, the first part of the for loop, is straightforward: move $t0,$zero # i = 0 (register $t0 = 0) To set array[i] to 0 we must first get its address. Start by multiplying i by 4 to get the byte address: loop1: sll $t1,$t0,2 # $t1 = i * 4

Since the starting address of the array is in a register, we must add it to the index to get the address of array[i] using an add instruction: add $t2,$a0,$t1 # $t2 = address of array[i]

Finally, we can store 0 in that address: sw $zero, 0($t2) # array[i] = 0

This instruction is the end of the body of the loop, so the next step is to increment i: addi $t0,$t0,1 # i = i + 1

The loop test checks if i is less than size:

We have now seen all the pieces of the procedure. Here is the MIPS code for clearing an array using indices:

(This code works as long as size is greater than 0; ANSI C requires a test of size before the loop, but we’ll skip that legality here.)

Pointer Version of Clear
The second procedure that uses pointers allocates the two parameters array and size to the registers $a0 and $a1 and allocates p to register $t0. The code for the second procedure starts with assigning the pointer p to the address of the first element of the array: move $t0,$a0 # p = address of array[0]

The next code is the body of the for loop, which simply stores 0 into p: loop2: sw $zero,0($t0) # Memory[p] = 0

This instruction implements the body of the loop, so the next code is the iteration increment, which changes p to point to the next word: addi $t0,$t0,4 # p = p + 4

Incrementing a pointer by 1 means moving the pointer to the next sequential object in C. S incep is a pointer to integers, each of which uses 4 bytes, the compiler increments p by 4. The loop test is next. The first step is calculating the address of the last element of array. Start with multiplying size by 4 to get its byte address: sll $t1,$a1,2 # $t1 = size * 4

and then we add the product to the starting address of the array to get the address of the first word after the array: add $t2,$a0,$t1 # $t2 = address of array[size]

add $t2,$a0,$t1 # $t2 = address of array[size] The loop test is simply to see if p is less than the last element of array: slt $t3,$t0,$t2 # $t3 = (p 3.65 days of repair/year Three nines: 99.9%=> 526 minutes of repair/year Four nines: 99.99%=> 52.6 minutes of repair/year Five nines: 99.999%=> 5.26 minutes of repair/year and so on. To increase MTTF, you can improve the quality of the components or design systems to continue operation in the presence of components that have failed. Hence, failure needs to be defined with respect to a context, as failure of a component may not lead to a failure of the system. To make this distinction clear, the term fault is used to mean failure of a component. Here are three ways to improve MTTF: 1. Fault avoidance: Preventing fault occurrence by construction. 2. Fault tolerance: Using redundancy to allow the service to comply with the service specification despite faults occurring. 3. Fault forecasting: Predicting the presence and creation of faults, allowing the component to be replaced before it fails.

The Hamming Single Error Correcting, Double Error Detecting Code (SEC/DED)
Richard Hamming invented a popular redundancy scheme for memory, for which he received the Turing Award in 1968. To invent redundant codes, it is helpful to talk about how “close” correct bit pa erns can be. What we call the Hamming distance is just the minimum number of bits that are different between any two correct bit pa erns. For example, the distance between 011011 and 001111 is two. What happens if the minimum distance between members of a codes is two, and we get a one-bit error? It will turn a valid pa ern in a code to an invalid one. Thus, if we can detect whether members of a code are valid or not, we can detect single bit errors, and can say we have a single bit error detection code.

e r r o r

d e t e c t i o n

c o d e

A code that enables the detection of an error in data, but not the precise location and, hence, correction of the error.

Hamming used a parity code for error detection. In a parity code, the number of 1 s in a word is counted; the word has odd parity if the number of 1 s is odd and even otherwise. When a word is wri en into memory, the parity bit is also wri en (1 for odd, 0 for even). That is, the parity of the N+1 bit word should always be even. Then, when the word is read out, the parity bit is read and checked. If the parity of the memory word and the stored parity bit do not match, an error has occurred.

Example
Calculate the parity of a byte with the value 31ten and show the pa ern stored to memory. Assume the parity bit is on the right. S uppose the most significant bit was inverted in memory, and then you read it back. Did you detect the error? What happens if the two most significant bits are inverted?

Answer
31ten is 00011111two, which has five 1 s. To make parity even, we need to write a 1 in the parity bit, or 000111111two. If the most significant bit is inverted when we read it back, we would see 100111111two which has seven 1 s. S ince we expect even parity and calculated odd parity, we would signal an error. If the two most significant bits are inverted, we would see 110111111two which has eight 1 s or even parity and we would not signal an error.

If there are 2 bits of error, then a 1-bit parity scheme will not detect any errors, since the parity will match the data with two errors. (Actually, a 1bit parity scheme can detect any odd number of errors; however, the probability of having 3 errors is much lower than the probability of having two, so, in practice, a 1-bit parity code is limited to detecting a single bit of error.) O f course, a parity code cannot correct errors, which Hamming wanted to do as well as detect them. If we used a code that had a minimum distance of 3, then any single bit error would be closer to the correct pa ern than to any other valid pa ern. He came up with an easy to understand mapping of data into a distance 3 code that we call Hamming Error Correction Code (ECC) in his honor. We use extraparity bits to allow the position identification of a single error. Here are the steps to calculate Hamming ECC 1. Start numbering bits from 1 on the left, as opposed to the traditional numbering of the rightmost bit being 0. 2. Mark all bit positions that are powers of 2 as parity bits (positions 1, 2, 4, 8, 16, …). 3. All other bit positions are used for data bits (positions 3, 5, 6, 7, 9, 10, 11, 12, 13, 14, 15, …). 4. The position of parity bit determines sequence of data bits that it checks (Figure 5.24 shows this coverage graphically) is: ■ Bit 1 (0001two) checks bits (1,3,5,7,9,11,…), which are bits where rightmost bit of address is 1 (0001two, 0011two, 0101two, 0111two, 1001two, 1011two,…). ■ Bit 2 (0010two) checks bits (2,3,6,7,10,11,14,15,…), which are the bits where the second bit to the right in the address is 1. ■ Bit 4 (0100two) checks bits (4–7, 12–15, 20–23,…), which are the bits where the third bit to the right in the address is 1. ■ Bit 8 (1000two) checks bits (8–15, 24–31, 40–47,…), which are the bits where the fourth bit to the right in the address is 1. Note that each data bit is covered by two or more parity bits.

FIGURE 5.24 Parity bits, data bits, and field coverage in a Hamming ECC code for eight data bits.

5. Set parity bits to create even parity for each group. In what seems like a magic trick, you can then determine whether bits are incorrect by looking at the parity bits. U sing the 12 bit code inFigure 5.24, if the value of the four parity calculations (p8,p4,p2,p1) was 0000, then there was no error. However, if the pa ern was, say, 1010, which is 10ten, then Hamming ECC tells us that bit 10 (d6) is an error. S ince the number is binary, we can correct the error just by inverting the value of bit 10.

Example
Assume one byte data value is 10011010two. First show the Hamming ECC code for that byte, and then invert bit 10 and show that the ECC code finds and corrects the single bit error.

Answer
Leaving spaces for the parity bits, the 12 bit pattern is_ 1 0 0 1_ 1 0 1 0. P osition 1 checks bits 1,3,5,7,9, and 11, which we highlight: __ 1_ 0 0 1_ 1 0 1 0. To make the group even parity, we should set bit 1 to 0. Position 2 checks bits 2,3,6,7,10,11, which is 0 _ 1_ 0 0 1_ 1 0 1 0 or odd parity, so we set position 2 to a 1. Position 4 checks bits 4,5,6,7,12, which is 0 1 1 _ 0 0 1_ 1 0 1, so we set it to a 1. Position 8 checks bits 8,9,10,11,12, which is 0 1 1 1 0 0 1 _ 1 0 1 0, so we set it to a 0. The final code word is 011100101010. Inverting bit 10 changes it to 011100101110. Parity bit 1 is 0 (011100101110 is four 1 s, so even parity; this group is OK). Parity bit 2 is 1 (011100101110 is five 1 s, so odd parity; there is an error somewhere). Parity bit 4 is 1 (011100101110 is two 1 s, so even parity; this group is OK). Parity bit 8 is 1 (011100101110 is three 1 s, so odd parity; there is an error somewhere). Parity bits 2 and 10 are incorrect. As 2+8=10, bit 10 must be wrong. Hence, we can correct the error by inverting bit 10: 011100101010. Voila!

Hamming did not stop at single bit error correction code. At the cost of one more bit, we can make the minimum Hamming distance in a code be 4. This means we can correct single bit errors and detect double bit errors. The idea is to add a parity bit that is calculated over the whole word. Let’s use a four-bit data word as an example, which would only need 7 bits for single bit error detection. Hamming parity bits H (p1 p2 p3) are computed (even parity as usual) plus the even parity over the entire word, p4: 12345678 p1 p2 d1 p3 d2 d3 d4 p4 Then the algorithm to correct one error and detect two is just to calculate parity over the ECC groups (H) as before plus one more over the whole group (p4). There are four cases: 1. H is even and p4 is even, so no error occurred. 2. H is odd and p4 is odd, so a correctable single error occurred. (p4 should calculate odd parity if one error occurred.) 3. H is even and p4 is odd, a single error occurred in p4 bit, not in the rest of the word, so correct the p4 bit. 4. H is odd and p4 is even, a double error occurred. (p4 should calculate even parity if two errors occurred.) S ingle Error CorrectingDouble Error Detecting (SEC ED) is common in memory for servers today. Conveniently, eight byte data blocks can get D SEC/DED with just one more byte, which is why many DIMMs are 72 bits wide.

E l a b o r a t i o n
To calculate how many bits are needed for S EC, letp be total number of parity bits and d number of data bits in p+d bit word. If p error correction bits are to point to error bit (p+d cases) plus one case to indicate that no error exists, we need:

image
For example, for 8 bits data means d=8 and 2p≥p+8+1, so p=4. Similarly, p=5 for 16 bits of data, 6 for 32 bits, 7 for 64 bits, and so on.

E l a b o r a t i o n
In very large systems, the possibility of multiple errors as well as complete failure of a single wide memory chip becomes significant. IBM introducedchipkill to solve this problem, and many very large systems use this technology. (Intel calls their version S DDC.)

IBM introducedchipkill to solve this problem, and many very large systems use this technology. (Intel calls their version S DDC.) S imilar in nature to the RAID approach used for disks (see Section 5.11), Chipkill distributes the data and ECC information, so that the complete failure of a single memory chip can be handled by supporting the reconstruction of the missing data from the remaining memory chips. Assuming a 10,000-processor cluster with 4 GiB per processor, IBM calculated the following rates of unrecoverable memory errors in three years of operation: ■ Parity only—about 90,000, or one unrecoverable (or undetected) failure every 17 minutes. ■ SEC/DED only—about 3500, or about one undetected or unrecoverable failure every 7.5 hours. ■ Chipkill—6, or about one undetected or unrecoverable failure every 2 months. Hence, Chipkill is a requirement for warehouse-scale computers.

E l a b o r a t i o n
While single or double bit errors are typical for memory systems, networks can have bursts of bit errors. O ne solution is calledCyclic Redundancy Check. For a block of k bits, a transmi er generates an n-k bit frame check sequence. It transmits n bits exactly divisible by some number. The receiver divides frame by that number. If there is no remainder, it assumes there is no error. If there is, the receiver rejects the message, and asks the transmi er to send again. As you might guess from Chapter 3, it is easy to calculate division for some binary numbers with a shift register, which made CRC codes popular even when hardware was more precious. Going even further, Reed-S olomon codes use Galois fields tocorrect multibit transmission errors, but now data is considered coefficients of a polynomials and the code space is values of a polynomial. The Reed-S olomon calculation is considerably more complicated than binary division!

5.6 Virtual Machines

Virtual Machines (VM) were first developed in the mid-1960s, and they have remained an important part of mainframe computing over the years. Although largely ignored in the single user PC era in the 1980s and 1990s, they have recently gained popularity due to ■ The increasing importance of isolation and security in modern systems ■ The failures in security and reliability of standard operating systems ■ The sharing of a single computer among many unrelated users, in particular for cloud computing ■ The dramatic increases in raw speed of processors over the decades, which makes the overhead of VMs more acceptable The broadest definition of VMs includes basically all emulation methods that provide a standard software interface, such as the Java VM. In this section, we are interested in VMs that provide a complete system-level environment at the binaryinstruction set architecture (IS A) level. Although some VMs run different IS As in the VM from the native hardware, we assume they always match the hardware. S uch VMs are called (O perating System Virtual Machines. IBM VM/370, VirtualBox, VMware ESX Server, and Xen are examples. S ystem virtual machines present the illusion that the users have an entire computer to themselves, including a copy of the operating system. A single computer runs multiple VMs and can support a number of differentoperating systems (O S es). O n a conventional platform, a single O S “owns” all the hardware resources, but with a VM, multiple OSes all share the hardware resources. The software that supports VMs is called a virtual machine monitor (VMM) or hypervisor; the VMM is the heart of virtual machine technology. The underlying hardware platform is called the host, and its resources are shared among the guest VMs. The VMM determines how to map virtual resources to physical resources: a physical resource may be time-shared, partitioned, or even emulated in software. The VMM is much smaller than a traditional OS; the isolation portion of a VMM is perhaps only 10,000 lines of code. Although our interest here is in VMs for improving protection, VMs provide two other benefits that are commercially significant: 1. Managing software. VMs provide an abstraction that can run the complete software stack, even including old operating systems like DOS. A typical deployment might be some VMs running legacy OSes, many running the current stable OS release, and a few testing the next OS release. 2. Managing hardware. One reason for multiple servers is to have each application running with the compatible version of the operating system on separate computers, as this separation can improve dependability. VMs allow these separate software stacks to run independently yet share hardware, thereby consolidating the number of servers. Another example is that some VMMs support migration of a running VM to a different computer, either to balance load or to evacuate from failing hardware.

H a r d w a r e / S o f t w a r e

I n t e r f a c e

Amazon Web Services (AWS) uses the virtual machines in its cloud computing offering EC2 for five reasons: 1. It allows AWS to protect users from each other while sharing the same server. 2. It simplifies software distribution within a warehouse scale computer. A customer installs a virtual machine image configured with the appropriate software, and AWS distributes it to all the instances a customer wants to use. 3. Customers (and AWS) can reliably “kill” a VM to control resource usage when customers complete their work. 4. Virtual machines hide the identity of the hardware on which the customer is running, which means AWS can keep using old servers and introduce new, more efficient servers. The customer expects performance for instances to match their ratings in “EC2 Compute Units,” which AWS defines: to “provide the equivalent CPU capacity of a 1.0–1.2 GHz 2007 AMD Opteron or 2007 Intel Xeon processor.” Thanks to Moore’s Law, newer servers clearly offer more EC2 Compute Units than older ones, but AWS can keep renting old servers as long as they are economical.

5. Virtual Machine Monitors can control the rate that a VM uses the processor, the network, and disk space, which allows AWS to offer many price points of instances of different types running on the same underlying servers. For example, in 2012 AWS offered 14 instance types, from small standard instances at $0.08 per hour to high I/O quadruple extra large instances at $3.10 per hour.

In general, the cost of processor virtualization depends on the workload. U ser-level processor-bound programs have zero virtualization overhead, because the O S is rarely invoked, so everything runs at native speeds. I/O -intensive workloads are generally also O S -intensive, executing many system calls and privileged instructions that can result in high virtualization overhead. O n the other hand, if the I/O -intensive workload is alsoI/Obound, the cost of processor virtualization can be completely hidden, since the processor is often idle waiting for I/O. The overhead is determined by both the number of instructions that must be emulated by the VMM and by how much time each takes to emulate them. Hence, when the guest VMs run the same IS A as the host, as we assume here, the goal the architecture and the VMM is to run almost all of instructions directly on the native hardware.

Requirements of a Virtual Machine Monitor
What must a VM monitor do? It presents a software interface to guest software, it must isolate the state of guests from each other, and it must protect itself from guest software (including guest OSes). The qualitative requirements are: ■ Guest software should behave on a VM exactly as if it were running on the native hardware, except for performance-related behavior or limitations of fixed resources shared by multiple VMs. ■ Guest software should not be able to change allocation of real system resources directly. To “virtualize” the processor, the VMM must control just about everything—access to privileged state, I/O , exceptions, and interrupts—even

To “virtualize” the processor, the VMM must control just about everything—access to privileged state, I/O , exceptions, and interrupts—even though the guest VM and OS currently running are temporarily using them. For example, in the case of a timer interrupt, the VMM would suspend the currently running guest VM, save its state, handle the interrupt, determine which guest VM to run next, and then load its state. Guest VMs that rely on a timer interrupt are provided with a virtual timer and an emulated timer interrupt by the VMM. To be in charge, the VMM must be at a higher privilege level than the guest VM, which generally runs in user mode; this also ensures that the execution of any privileged instruction will be handled by the VMM. The basic requirements of system virtual: ■ At least two processor modes, system and user. ■ A privileged subset of instructions that is available only in system mode, resulting in a trap if executed in user mode; all system resources must be controllable only via these instructions.

(Lack of) Instruction Set Architecture Support for Virtual Machines
If VMs are planned for during the design of the IS A, it’s relatively easy to reduce both the number of instructions that must be executed by a VMM and improve their emulation speed. An architecture that allows the VM to execute directly on the hardware earns the titlevirtualizable, and the IBM 370 architecture proudly bears that label. Alas, since VMs have been considered for P C and server applications only fairly recently, most instruction sets were created without virtualization in mind. These culprits include x86 and most RISC architectures, including ARMv7 and MIPS. Because the VMM must ensure that the guest system only interacts with virtual resources, a conventional guest O S runs as a user mode program on top of the VMM. Then, if a guest O S a empts to access or modify information related to hardware resources via a privileged instruction—for example, reading or writing a status bit that enables interrupts—it will trap to the VMM. The VMM can then effect the appropriate changes to corresponding real resources. Hence, if any instruction that tries to read or write such sensitive information traps when executed in user mode, the VMM can intercept it and support a virtual version of the sensitive information, as the guest OS expects. In the absence of such support, other measures must be taken. A VMM must take special precautions to locate all problematic instructions and ensure that they behave correctly when executed by a guest O S , thereby increasing the complexity of the VMM and reducing the performance of running the VM.

Protection and Instruction Set Architecture

P rotection is a joint effort of architecture and operating systems, but architects had to modify some awkward details of existing instruction set architectures when virtual memory became popular. For example, the x86 instruction P O P F loads the flag registers from the top of the stack in memory. O ne of the flags is the Interrupt Enable(IE) flag. If you run the P O P F instruction in user mode, rather than trap it, it simply changes all the flags except IE. In system mode, it does change the IE. Since a guest OS runs in user mode inside a VM, this is a problem, as it expects to see a changed IE. Historically, IBM mainframe hardware and VMM took three steps to improve performance of virtual machines: 1. Reduce the cost of processor virtualization. 2. Reduce interrupt overhead cost due to the virtualization. 3. Reduce interrupt cost by steering interrupts to the proper VM without invoking VMM. AMD and Intel tried to address the first point in 2006 by reducing the cost of processor virtualization. It will be interesting to see how many generations of architecture and VMM modifications it will take to address all three points, and how long before virtual machines of the 21st century will be as efficient as the IBM mainframes and VMMs of the 1970s.

5.7 Virtual Memory
… a system has been devised to make the core drum combination appear to the programmer as a single level store, the requisite transfers taking place automatically.
Kilburn et al., One-level storage system, 1962

In earlier sections, we saw how caches provided fast access to recently used portions of a program’s code and data. S imilarly, the main memory can act as a “cache” for the secondary storage, usually implemented with magnetic disks. This technique is called virtual memory. Historically, there were two major motivations for virtual memory: to allow efficient and safe sharing of memory among multiple programs, such as for the memory needed by multiple virtual machines for cloud computing, and to remove the programming burdens of a small, limited amount of main memory. Five decades after its invention, it’s the former reason that reigns today.

v i r t u a l

m e m o r y

A technique that uses main memory as a “cache” for secondary storage.

O f course, to allow multiple virtual machines to share the same memory, we must be able to protect the virtual machines from each other, ensuring that a program can only read and write the portions of main memory that have been assigned to it. Main memory need contain only the active portions of the many virtual machines, just as a cache contains only the active portion of one program. Thus, the principle of locality enables virtual memory as well as caches, and virtual memory allows us to efficiently share the processor as well as the main memory. We cannot know which virtual machines will share the memory with other virtual machines when we compile them. In fact, the virtual machines sharing the memory change dynamically while the virtual machines are running. Because of this dynamic interaction, we would like to compile each program into its own address space—a separate range of memory locations accessible only to this program. Virtual memory implements the translation of a program’s address space to physical addresses. This translation process enforces protection of a program’s address space from other virtual machines.

p h y s i c a l
An address in main memory.

a d d r e s s

p r o t e c t i o n
A set of mechanisms for ensuring that multiple processes sharing the processor, memory, or I/O devices cannot interfere, intentionally or unintentionally, with one another by reading or writing each other’s data. These mechanisms also isolate the operating system from a user process.

The second motivation for virtual memory is to allow a single user program to exceed the size of primary memory. Formerly, if a program became too large for memory, it was up to the programmer to make it fit. P rogrammers divided programs into pieces and then identified the pieces that were mutually exclusive. These overlays were loaded or unloaded under user program control during execution, with the programmer ensuring that the program never tried to access an overlay that was not loaded and that the overlays loaded never exceeded the total size of the memory. O verlays were traditionally organized as modules, each containing both code and data. Calls between procedures in different modules would lead to overlaying of one module with another. As you can well imagine, this responsibility was a substantial burden on programmers. Virtual memory, which was invented to relieve programmers of this difficulty, automatically manages the two levels of the memory hierarchy represented by main memory (sometimes called physical memory to distinguish it from virtual memory) and secondary storage. Although the concepts at work in virtual memory and in caches are the same, their differing historical roots have led to the use of different terminology. A virtual memory block is called a page, and a virtual memory miss is called a page fault. With virtual memory, the processor produces a virtual address, which is translated by a combination of hardware and software to a physical address, which in turn can be used to access main memory. Figure 5.25 shows the virtually addressed memory with pages mapped to main memory. This process is called address mapping or address translation. Today, the two memory hierarchy levels controlled by virtual memory are usually DRAMs and flash memory in personal mobile devices and DRAMs and magnetic disks in servers (see ection 5.2). If we return to our library analogy, we can think of a virtual S address as the title of a book and a physical address as the location of that book in the library, such as might be given by the Library of Congress call number.

p a g e

f a u l t

An event that occurs when an accessed page is not present in main memory.

v i r t u a l

a d d r e s s

An address that corresponds to a location in virtual space and is translated by address mapping to a physical address when memory is accessed.

a d d r e s s

t r a n s l a t i o n

Also called address mapping. The process by which a virtual address is mapped to an address used to access memory.

FIGURE 5.25 In virtual memory, blocks of memory (called pages) are mapped from one set of addresses (called virtual addresses) to another set (called physical addresses). The processor generates virtual addresses while the memory is accessed using physical addresses. Both the virtual memory and the physical memory are broken into pages, so that a virtual page is mapped to a physical page. Of course, it is also possible for a virtual page to be absent from main memory and not be mapped to a physical address; in that case, the page resides on disk. Physical pages can be shared by having two virtual addresses point to the same physical address. This capability is used to allow two different programs to share data or code.

Virtual memory also simplifies loading the program for execution by providing relocation. Relocation maps the virtual addresses used by a program to different physical addresses before the addresses are used to access memory. This relocation allows us to load the program anywhere in main memory. Furthermore, all virtual memory systems in use today relocate the program as a set of fixed-size blocks (pages), thereby eliminating the need to find a contiguous block of memory to allocate to a program; instead, the operating system need only find a sufficient number of pages in main memory. In virtual memory, the address is broken into a virtual page number and a page offset. Figure 5.26 shows the translation of the virtual page number to a physical page number. The physical page number constitutes the upper portion of the physical address, while the page offset, which is not changed, constitutes the lower portion. The number of bits in the page offset field determines the page size. The number of pages addressable with the virtual address need not match the number of pages addressable with the physical address. Having a larger number of virtual pages than physical pages is the basis for the illusion of an essentially unbounded amount of virtual memory.

FIGURE 5.26 Mapping from a virtual to a physical address. The page size is 212=4 KiB. The number of physical pages allowed in memory is 218, since the physical page number has 18 bits in it. Thus, main memory can have at most 1 GiB, while the virtual address space is 4 GiB.

Many design choices in virtual memory systems are motivated by the high cost of a page fault. A page fault to disk will take millions of clock cycles to process. (The table on page 378 shows that main memory latency is about 100,000 times quicker than disk.) This enormous miss penalty, dominated by the time to get the first word for typical page sizes, leads to several key decisions in designing virtual memory systems: ■ Pages should be large enough to try to amortize the high access time. Sizes from 4 KiB to 16 KiB are typical today. New desktop and server systems are being developed to support 32 KiB and 64 KiB pages, but new embedded systems are going in the other direction, to 1 KiB pages. ■ Organizations that reduce the page fault rate are attractive. The primary technique used here is to allow fully associative placement of pages in memory. ■ Page faults can be handled in software because the overhead will be small compared to the disk access time. In addition, software can afford to use clever algorithms for choosing how to place pages because even small reductions in the miss rate will pay for the cost of such algorithms.

use clever algorithms for choosing how to place pages because even small reductions in the miss rate will pay for the cost of such algorithms.
■ Write-through will not work for virtual memory, since writes take too long. Instead, virtual memory systems use write-back.

The next few subsections address these factors in virtual memory design.

E l a b o r a t i o n
We present the motivation for virtual memory as many virtual machines sharing the same memory, but virtual memory was originally invented so that many programs could share a computer as part of a timesharing system. S ince many readers today have no experience with timesharing systems, we use virtual machines to motivate this section.

E l a b o r a t i o n
For servers and even P Cs, 32-bit address processors are problematic. Although we normally think of virtual addresses as much larger than physical addresses, the opposite can occur when the processor address size is small relative to the state of the memory technology. No single program or virtual machine can benefit, but a collection of programs or virtual machines running at the same time can benefit from not having to be swapped to memory or by running on parallel processors.

E l a b o r a t i o n
The discussion of virtual memory in this book focuses on paging, which uses fixed-size blocks. There is also a variable-size block scheme called segmentation. In segmentation, an address consists of two parts: a segment number and a segment offset. The segment number is mapped to a physical address, and the offset is added to find the actual physical address. Because the segment can vary in size, a bounds check is also needed to make sure that the offset is within the segment. The major use of segmentation is to support more powerful methods of protection and sharing in an address space. Most operating system textbooks contain extensive discussions of segmentation compared to paging and of the use of segmentation to logically share the address space. The major disadvantage of segmentation is that it splits the address space into logically separate pieces that must be manipulated as a two-part address: the segment number and the offset. Paging, in contrast, makes the boundary between page number and offset invisible to programmers and compilers.

s e g m e n t a t i o n
A variable-size address mapping scheme in which an address consists of two parts: a segment number, which is mapped to a physical address, and a segment offset.

S egments have also been used as a method to extend the address space without changing the word size of the computer. S uch a empts have been unsuccessful because of the awkwardness and performance penalties inherent in a two-part address, of which programmers and compilers must be aware. Many architectures divide the address space into large fixed-size blocks that simplify protection between the operating system and user programs and increase the efficiency of implementing paging. Although these divisions are often called “segments,” this mechanism is much simpler than variable block size segmentation and is not visible to user programs; we discuss it in more detail shortly.

Placing a Page and Finding it Again
Because of the incredibly high penalty for a page fault, designers reduce page fault frequency by optimizing page placement. If we allow a virtual page to be mapped to any physical page, the operating system can then choose to replace any page it wants when a page fault occurs. For example, the operating system can use a sophisticated algorithm and complex data structures that track page usage to try to choose a page that will not be needed for a long time. The ability to use a clever and flexible replacement scheme reduces the page fault rate and simplifies the use of fully associative placement of pages. As mentioned in Section 5.4, the difficulty in using fully associative placement is in locating an entry, since it can be anywhere in the upper level of the hierarchy. A full search is impractical. In virtual memory systems, we locate pages by using a table that indexes the memory; this structure is called a page table, and it resides in memory. A page table is indexed with the page number from the virtual address to discover the corresponding physical page number. Each program has its own page table, which maps the virtual address space of that program to main memory. In our library analogy, the page table corresponds to a mapping between book titles and library locations. Just as the card catalog may contain entries for books in another library on campus rather than the local branch library, we will see that the page table may contain entries for pages not present in memory. To indicate the location of the page table in memory, the hardware includes a register that points to the start of the page table; we call this the page table register. Assume for now that the page table is in a fixed and contiguous area of memory.

p a g e

t a b l e

The table containing the virtual to physical address translations in a virtual memory system. The table, which is stored in memory, is typically indexed by the virtual page number; each entry in the table contains the physical page number for that virtual page if the page is currently in memory.

H a r d w a r e / S o f t w a r e

I n t e r f a c e

The page table, together with the program counter and the registers, specifies the state of a virtual machine. If we want to allow another virtual machine to use the processor, we must save this state. Later, after restoring this state, the virtual machine can continue execution. We often refer to this state as a process. The process is considered active when it is in possession of the processor; otherwise, it is considered inactive. The operating system can make a process active by loading the process’s state, including the program counter, which will initiate execution at the value of the saved program counter. The process’s address space, and hence all the data it can access in memory, is defined by its page table, which resides in memory. Rather than save the entire page table, the operating system simply loads the page table register to point to the page table of the process it wants to make active. Each process has its own page table, since different processes use the same virtual addresses. The operating system is responsible for allocating the physical memory and updating the page tables, so that the virtual address spaces of different processes do not collide. As we will see shortly, the use of separate page tables also provides protection of one process from another.

Figure 5.27 uses the page table register, the virtual address, and the indicated page table to show how the hardware can form a physical address. A valid bit is used in each page table entry, just as we did in a cache. If the bit is off, the page is not present in main memory and a page fault occurs. If the bit is on, the page is in memory and the entry contains the physical page number.

FIGURE 5.27 The page table is indexed with the virtual page number to obtain the corresponding portion of the physical address. We assume a 32-bit address. The page table pointer gives the starting address of the page table. In this figure, the page size is 212 bytes, or 4 KiB. The virtual address space is 232 bytes, or 4 GiB, and the physical address space is 230 bytes, which allows main memory of up to 1 GiB. The number of entries in the page table is 220, or 1 million entries. The valid bit for each entry indicates whether the mapping is legal. If it is off, then the page is not present in memory. Although the page table entry shown here need only be 19 bits wide, it would typically be rounded up to 32 bits for ease of indexing. The extra bits would be used to store additional information that needs to be kept on a per-page basis, such as protection.

Because the page table contains a mapping for every possible virtual page, no tags are required. In cache terminology, the index that is used to access the page table consists of the full block address, which is the virtual page number.

Page Faults
If the valid bit for a virtual page is off, a page fault occurs. The operating system must be given control. This transfer is done with the exception mechanism, which we saw in Chapter 4 and will discuss again later in this section. O nce the operating system gets control, it must find the page in the next level of the hierarchy (usually flash memory or magnetic disk) and decide where to place the requested page in main memory. The virtual address alone does not immediately tell us where the page is on disk. Returning to our library analogy, we cannot find the location of a library book on the shelves just by knowing its title. Instead, we go to the catalog and look up the book, obtaining an address for the location on the shelves, such as the Library of Congress call number. Likewise, in a virtual memory system, we must keep track of the location on disk of each page in virtual address space. Because we do not know ahead of time when a page in memory will be replaced, the operating system usually creates the space on flash memory or disk for all the pages of a process when it creates the process. This space is called the swap space. At that time, it also creates a data structure to record where each virtual page is stored on disk. This data structure may be part of the page table or may be an auxiliary data structure indexed in the same way as the page table. Figure 5.28 shows the organization when a single table holds either the physical page number or the disk address.

s w a p

s p a c e

The space on the disk reserved for the full virtual memory space of a process.

FIGURE 5.28 The page table maps each page in virtual memory to either a page in main memory or a page stored on disk, which is the next level in the hierarchy. The virtual page number is used to index the page table. If the valid bit is on, the page table supplies the physical page number (i.e., the starting address of the page in memory) corresponding to the virtual page. If the valid bit is off, the page currently resides only on disk, at a specified disk address. In many systems, the table of physical page addresses and disk page addresses, while logically one table, is stored in two separate data structures. Dual tables are justified in part because we must keep the disk addresses of all the pages, even if they are currently in main memory. Remember that the pages in main memory and the pages on disk are the same size.

The operating system also creates a data structure that tracks which processes and which virtual addresses use each physical page. When a page fault occurs, if all the pages in main memory are in use, the operating system must choose a page to replace. Because we want to minimize the number of page faults, most operating systems try to choose a page that they hypothesize will not be needed in the near future. U sing the past to predict the future, operating systems follow the least recently used (LRU ) replacement scheme, which we mentioned inS ection 5.4. The operating system searches for the least recently used page, assuming that a page that has not been used in a long time is less likely to be needed than a more recently accessed page. The replaced pages are wri en to swap space on the disk. In case you are wondering, the operating system is just another process, and these tables controlling memory are in memory; the details of this seeming contradiction will be explained shortly.

H a r d w a r e / S o f t w a r e

I n t e r f a c e

Implementing a completely accurate LRU scheme is too expensive, since it requires updating a data structure on every memory reference. Instead, most operating systems approximate LRU by keeping track of which pages have and which pages have not been recently used. To help the operating system estimate the LRU pages, some computers provide areference bit or use bit, which is set whenever a page is accessed. The operating system periodically clears the reference bits and later records them so it can determine which pages were touched during a particular time period. With this usage information, the operating system can select a page that is among the least recently referenced (detected by having its reference bit off). If this bit is not provided by the hardware, the operating system must find another way to estimate which pages have been accessed.

r e f e r e n c e

b i t

Also called use bit. A field that is set whenever a page is accessed and that is used to implement LRU or other replacement schemes.

E l a b o r a t i o n
With a 32-bit virtual address, 4 KiB pages, and 4 bytes per page table entry, we can compute the total page table size:

image
That is, we would need to use 4 MiB of memory for each program in execution at any time. This amount is not so bad for a single process. What if there are hundreds of processes running, each with their own page table? And how should we handle 64-bit addresses, which by this calculation would need 252 words?

addresses, which by this calculation would need 2 words? A range of techniques is used to reduce the amount of storage required for the page table. The five techniques below aim at reducing the total maximum storage required as well as minimizing the main memory dedicated to page tables: 1. The simplest technique is to keep a limit register that restricts the size of the page table for a given process. If the virtual page number becomes larger than the contents of the limit register, entries must be added to the page table. This technique allows the page table to grow as a process consumes more space. Thus, the page table will only be large if the process is using many pages of virtual address space. This technique requires that the address space expand in only one direction. 2. Allowing growth in only one direction is not sufficient, since most languages require two areas whose size is expandable: one area holds the stack and the other area holds the heap. Because of this duality, it is convenient to divide the page table and let it grow from the highest address down, as well as from the lowest address up. This means that there will be two separate page tables and two separate limits. The use of two page tables breaks the address space into two segments. The high-order bit of an address usually determines which segment and thus which page table to use for that address. Since the high-order address bit specifies the segment, each segment can be as large as one-half of the address space. A limit register for each segment specifies the current size of the segment, which grows in units of pages. This type of segmentation is used by many architectures, including MIPS. Unlike the type of segmentation discussed in the second elaboration on page 431, this form of segmentation is invisible to the application program, although not to the operating system. The major disadvantage of this scheme is that it does not work well when the address space is used in a sparse fashion rather than as a contiguous set of virtual addresses. 3. Another approach to reducing the page table size is to apply a hashing function to the virtual address so that the page table need be only the size of the number of physical pages in main memory. Such a structure is called an inverted page table. Of course, the lookup process is slightly more complex with an inverted page table, because we can no longer just index the page table. 4. Multiple levels of page tables can also be used to reduce the total amount of page table storage. The first level maps large fixedsize blocks of virtual address space, perhaps 64 to 256 pages in total. These large blocks are sometimes called segments, and this first-level mapping table is sometimes called a segment table, though the segments are again invisible to the user. Each entry in the segment table indicates whether any pages in that segment are allocated and, if so, points to a page table for that segment. Address translation happens by first looking in the segment table, using the highest-order bits of the address. If the segment address is valid, the next set of high-order bits is used to index the page table indicated by the segment table entry. This scheme allows the address space to be used in a sparse fashion (multiple noncontiguous segments can be active) without having to allocate the entire page table. Such schemes are particularly useful with very large address spaces and in software systems that require noncontiguous allocation. The primary disadvantage of this two-level mapping is the more complex process for address translation. 5. To reduce the actual main memory tied up in page tables, most modern systems also allow the page tables to be paged. Although this sounds tricky, it works by using the same basic ideas of virtual memory and simply allowing the page tables to reside in the virtual address space. In addition, there are some small but critical problems, such as a never-ending series of page faults, which must be avoided. How these problems are overcome is both very detailed and typically highly processor specific. In brief, these problems are avoided by placing all the page tables in the address space of the operating system and placing at least some of the page tables for the operating system in a portion of main memory that is physically addressed and is always present and thus never on disk.

What about Writes?
The difference between the access time to the cache and main memory is tens to hundreds of cycles, and write-through schemes can be used, although we need a write buffer to hide the latency of the write from the processor. In a virtual memory system, writes to the next level of the hierarchy (disk) can take millions of processor clock cycles; therefore, building a write buffer to allow the system to write-through to disk would be completely impractical. Instead, virtual memory systems must use write-back, performing the individual writes into the page in memory, and copying the page back to disk when it is replaced in the memory.

H a r d w a r e / S o f t w a r e

I n t e r f a c e

A write-back scheme has another major advantage in a virtual memory system. Because the disk transfer time is small compared with its access time, copying back an entire page is much more efficient than writing individual words back to the disk. A write-back operation, although more efficient than transferring individual words, is still costly. Thus, we would like to know whether a page needs to be copied back when we choose to replace it. To track whether a page has been wri en since it was read into the memory, a dirty bit is added to the page table. The dirty bit is set when any word in a page is wri en. If the operating system chooses to replace the page, the dirty bit indicates whether the page needs to be wri en out before its location in memory can be given to another page. Hence, a modified page is often called a dirty page.

Making Address Translation Fast: the TLB
S ince the page tables are stored in main memory, every memory access by a program can take at least twice as long: one memory access to obtain the physical address and a second access to get the data. The key to improving access performance is to rely on locality of reference to the page table. When a translation for a virtual page number is used, it will probably be needed again in the near future, because the references to the words on that page have both temporal and spatial locality. Accordingly, modern processors include a special cache that keeps track of recently used translations. This special address translation cache is traditionally referred to as a translation-lookaside buffer (TLB), although it would be more accurate to call it a translation cache. The TLB corresponds to that little piece of paper we typically use to record the location of a set of books we look up in the card catalog; rather than continually searching the entire catalog, we record the location of several books and use the scrap of paper as a cache of Library of Congress call numbers.

t r a n s l a t i o n - l o o k a s i d e

b u ff e r

(

A cache that keeps track of recently used address mappings to try to avoid an access to the page table.

Figure 5.29 shows that each tag entry in the TLB holds a portion of the virtual page number, and each data entry of the TLB holds a physical page number. Because we access the TLB instead of the page table on every reference, the TLB will need to include other status bits, such as the dirty and the reference bits.

FIGURE 5.29 The TLB acts as a cache of the page table for the entries that map to physical pages only. The TLB contains a subset of the virtual-to-physical page mappings that are in the page table. The TLB mappings are shown in color. Because the TLB is a cache, it must have a tag field. If there is no matching entry in the TLB for a page, the page table must be examined. The page table either supplies a physical page number for the page (which can then be used to build a TLB entry) or indicates that the page resides on disk, in which case a page fault occurs. Since the page table has an entry for every virtual page, no tag field is needed; in other words, unlike a TLB, a page table is not a cache.

O n every reference, we look up the virtual page number in the TLB. If we get a hit, the physical page number is used to form the address, and the corresponding reference bit is turned on. If the processor is performing a write, the dirty bit is also turned on. If a miss in the TLB occurs, we must determine whether it is a page fault or merely a TLB miss. If the page exists in memory, then the TLB miss indicates only that the translation is missing. In such cases, the processor can handle the TLB miss by loading the translation from the page table into the TLB and then trying the reference again. If the page is not present in memory, then the TLB miss indicates a true page fault. In this case, the processor invokes the operating system using an exception. Because the TLB has many fewer entries than the number of pages in main memory, TLB misses will be much more frequent than true page faults. TLB misses can be handled either in hardware or in software. In practice, with care there can be li le performance difference between the two approaches, because the basic operations are the same in either case. After a TLB miss occurs and the missing translation has been retrieved from the page table, we will need to select a TLB entry to replace. Because the reference and dirty bits are contained in the TLB entry, we need to copy these bits back to the page table entry when we replace an entry. These bits are the only portion of the TLB entry that can be changed. U sing write-back—that is, copying these entries back at miss time rather than when they are wri en—is very efficient, since we expect the TLB miss rate to be small. S ome systems use other techniques to approximate the reference and dirty bits, eliminating the need to write into the TLB except to load a new table entry on a miss. Some typical values for a TLB might be ■ TLB size: 16–512 entries ■ Block size: 1–2 page table entries (typically 4–8 bytes each) ■ Hit time: 0.5–1 clock cycle ■ Miss penalty: 10–100 clock cycles ■ Miss rate: 0.01%–1% Designers have used a wide variety of associativities in TLBs. S ome systems use small, fully associative TLBs because a fully associative mapping has a lower miss rate; furthermore, since the TLB is small, the cost of a fully associative mapping is not too high. O ther systems use large TLBs, often with small associativity. With a fully associative mapping, choosing the entry to replace becomes tricky since implementing a hardware LRU scheme is too expensive. Furthermore, since TLB misses are much more frequent than page faults and thus must be handled more cheaply, we cannot afford an expensive software algorithm, as we can for page faults. As a result, many systems provide some support for randomly choosing an entry to replace. We’ll examine replacement schemes in a little more detail in Section 5.8.

The Intrinsity FastMATH TLB

To see these ideas in a real processor, let’s take a closer look at the TLB of the Intrinsity FastMATH. The memory system uses 4 KiB pages and a 32 bit address space; thus, the virtual page number is 20 bits long, as in the top of Figure 5.30. The physical address is the same size as the virtual address. The TLB contains 16 entries, it is fully associative, and it is shared between the instruction and data references. Each entry is 64 bits wide and contains a 20-bit tag (which is the virtual page number for that TLB entry), the corresponding physical page number (also 20 bits), a valid bit, a dirty bit, and other bookkeeping bits. Like most MIPS systems, it uses software to handle TLB misses.

FIGURE 5.30 The TLB and cache implement the process of going from a virtual address to a data item in the Intrinsity FastMATH. This figure shows the organization of the TLB and the data cache, assuming a 4 KiB page size. This diagram focuses on a read; Figure 5.31 describes how to handle writes. Note that unlike Figure 5.12, the tag and data RAMs are split. By addressing the long but narrow data RAM with the cache index concatenated with the block offset, we select the desired word in the block without a 16:1 multiplexor. While the cache is direct mapped, the TLB is fully associative. Implementing a fully associative TLB requires that every TLB tag be compared against the virtual page number, since the entry of interest can be anywhere in the TLB. (See content addressable memories in the Elaboration on page 408.) If the valid bit of the matching entry is on, the access is a TLB hit, and bits from the physical page number together with bits from the page offset form the index that is used to access the cache.

Figure 5.30 shows the TLB and one of the caches, while Figure 5.31 shows the steps in processing a read or write request. When a TLB miss occurs, the MIP S hardware saves the page number of the reference in a special register and generates an exception. The exception invokes the operating system, which handles the miss in software. To find the physical address for the missing page, the TLB miss routine indexes the page table using the page number of the virtual address and the page table register, which indicates the starting address of the active process page table. U sing a special set of system instructions that can update the TLB, the operating system places the physical address from the page table into the TLB. A TLB miss takes about 13 clock cycles, assuming the code and the page table entry are in the instruction cache and data cache, respectively. (We will see the MIP S TLB code on page 449.) A true page fault occurs if the page table entry does not have a valid physical address. The hardware maintains an index that indicates the recommended entry to replace; the recommended entry is chosen randomly.

FIGURE 5.31 Processing a read or a write-through in the Intrinsity FastMATH TLB and cache. If the TLB generates a hit, the cache can be accessed with the resulting physical address. For a read, the cache generates a hit or miss and supplies the data or causes a stall while the data is brought from memory. If the operation is a write, a portion of the cache entry is overwritten for a hit and the data is sent to the write buffer if we assume write-through. A write miss is just like a read miss except that the block is modified after it is read from memory. Write-back requires writes to set a dirty bit for the cache block, and a write buffer is loaded with the whole block only on a read miss or write miss if the block to be replaced is dirty. Notice that a TLB hit and a cache hit are independent events, but a cache hit can only occur after a TLB hit occurs, which means that the data must be present in memory. The relationship between TLB misses and cache misses is examined further in the following example and the exercises at the end of this chapter.

There is an extra complication for write requests: namely, the write access bit in the TLB must be checked. This bit prevents the program from writing into pages for which it has only read access. If the program a empts a write and the write access bit is off, an exception is generated. The write access bit forms part of the protection mechanism, which we will discuss shortly.

Integrating Virtual Memory, TLBs, and Caches
O ur virtual memory and cache systems work together as a hierarchy, so that data cannot be in the cache unless it is present in main memory. The operating system helps maintain this hierarchy by flushing the contents of any page from the cache when it decides to migrate that page to disk. At the same time, the OS modifies the page tables and TLB, so that an attempt to access any data on the migrated page will generate a page fault. U nder the best of circumstances, a virtual address is translated by the TLB and sent to the cache where the appropriate data is found, retrieved, and sent back to the processor. In the worst case, a reference can miss in all three components of the memory hierarchy: the TLB, the page table, and the cache. The following example illustrates these interactions in more detail.

O v e r a l l Example

O p e r a t i o n

o f

a

M e m o r y

In a memory hierarchy like that of Figure 5.30, which includes a TLB and a cache organized as shown, a memory reference can encounter three different types of misses: a TLB miss, a page fault, and a cache miss. Consider all the combinations of these three events with one or more occurring (seven possibilities). For each possibility, state whether this event can actually occur and under what circumstances.

Answer
Figure 5.32 shows all combinations and whether each is possible in practice.

FIGURE 5.32 The possible combinations of events in the TLB, virtual memory system, and cache. Three of these combinations are impossible, and one is possible (TLB hit, virtual memory hit, cache miss) but never detected.

E l a b o r a t i o n
Figure 5.32 assumes that all memory addresses are translated to physical addresses before the cache is accessed. In this organization, the cache is physically indexed and physically tagged (both the cache index and tag are physical, rather than virtual, addresses). In such a system, the amount of time to access memory, assuming a cache hit, must accommodate both a TLB access and a cache access; of course, these accesses can be pipelined.

Alternatively, the processor can index the cache with an address that is completely or partially virtual. This is called avirtually addressed cache, and it uses tags that are virtual addresses; hence, such a cache is virtually indexed and virtually tagged. In such caches, the address translation hardware (TLB) is unused during the normal cache access, since the cache is accessed with a virtual address that has not been translated to a physical address. This takes the TLB out of the critical path, reducing cache latency. When a cache miss occurs, however, the processor needs to translate the address to a physical address so that it can fetch the cache block from main memory.

v i r t u a l l y

a d d r e s s e d

c a c h e

A cache that is accessed with a virtual address rather than a physical address.

When the cache is accessed with a virtual address and pages are shared between processes (which may access them with different virtual addresses), there is the possibility of aliasing. Aliasing occurs when the same object has two names—in this case, two virtual addresses for the same page. This ambiguity creates a problem, because a word on such a page may be cached in two different locations, each corresponding to different virtual addresses. This ambiguity would allow one program to write the data without the other program being aware that the data had changed. Completely virtually addressed caches either introduce design limitations on the cache and TLB to reduce aliases or require the operating system, and possibly the user, to take steps to ensure that aliases do not occur.

a l i a s i n g
A situation in which two addresses access the same object; it can occur in virtual memory when there are two virtual addresses for the same physical page.

A common compromise between these two design points is caches that are virtually indexed—sometimes using just the page-offset portion of the address, which is really a physical address since it is not translated—but use physical tags. These designs, which are virtually indexed but physically tagged, a empt to achieve the performance advantages of virtually indexed caches with the architecturally simpler advantages of a physically addressed cache. For example, there is no alias problem in this case. Figure 5.30 assumed a 4 KiB page size, but it’s really 16 KiB, so the Intrinsity FastMATH can use this trick. To pull it off, there must be careful coordination between the minimum page size, the cache size, and associativity.

p h y s i c a l l y
A cache that is addressed by a physical address.

a d d r e s s e d

c a c h e

Implementing Protection with Virtual Memory
Perhaps the most important function of virtual memory today is to allow sharing of a single main memory by multiple processes, while providing memory protection among these processes and the operating system. The protection mechanism must ensure that although multiple processes are sharing the same main memory, one renegade process cannot write into the address space of another user process or into the operating system either intentionally or unintentionally. The write access bit in the TLB can protect a page from being wri en. Without this level of protection, computer viruses would be even more widespread.

H a r d w a r e / S o f t w a r e

I n t e r f a c e

To enable the operating system to implement protection in the virtual memory system, the hardware must provide at least the three basic capabilities summarized below. Note that the first two are the same requirements as needed for virtual machines (Section 5.6). 1. Support at least two modes that indicate whether the running process is a user process or an operating system process, variously called a supervisor process, a kernel process, or an executive process.

s u p e r v i s o r

m o d e

Also called kernel mode. A mode indicating that a running process is an operating system process.

2. Provide a portion of the processor state that a user process can read but not write. This includes the user/supervisor mode bit, which dictates whether the processor is in user or supervisor mode, the page table pointer, and the TLB. To write these elements, the operating system uses special instructions that are only available in supervisor mode. 3. Provide mechanisms whereby the processor can go from user mode to supervisor mode and vice versa. The first direction is typically accomplished by a system call exception, implemented as a special instruction (syscall in the MIPS instruction set) that transfers control to a dedicated location in supervisor code space. As with any other exception, the program counter from the point of the system call is saved in the exception PC (EPC), and the processor is placed in supervisor mode. To return to user mode from the exception, use the return from exception (ERET) instruction, which resets to user mode and jumps to the address in EPC.

s y s t e m

c a l l

A special instruction that transfers control from user mode to a dedicated location in supervisor code space, invoking the exception mechanism in the process.

By using these mechanisms and storing the page tables in the operating system’s address space, the operating system can change the page tables while preventing a user process from changing them, ensuring that a user process can access only the storage provided to it by the operating system.

We also want to prevent a process from reading the data of another process. For example, we wouldn’t want a student program to read the grades while they were in the processor’s memory. O nce we begin sharing main memory, we must provide the ability for a process to protect its data from both reading and writing by another process; otherwise, sharing the main memory will be a mixed blessing! Remember that each process has its own virtual address space. Thus, if the operating system keeps the page tables organized so that the independent virtual pages map to disjoint physical pages, one process will not be able to access another’s data. O f course, this also requires that a user process be unable to change the page table mapping. The operating system can assure safety if it prevents the user process from modifying its own page tables. However, the operating system must be able to modify the page tables. Placing the page tables in the protected address space of the operating system satisfies both requirements. When processes want to share information in a limited way, the operating system must assist them, since accessing the information of another process requires changing the page table of the accessing process. The write access bit can be used to restrict the sharing to just read sharing, and, like the rest of the page table, this bit can be changed only by the operating system. To allow another process, say, P 1, to read a page owned by process P 2, P 2 would ask the operating system to create a page table entry for a virtual page in P 1’s address space that points to the same physical page that P 2 wants to share. The operating system could use the write protection bit to prevent P 1 from writing the data, if that was P 2’s wish. Any bits that determine the access rights for a page must be included in both the page table and the TLB, because the page table is accessed only on a TLB miss.

E l a b o r a t i o n
When the operating system decides to change from running process P 1 to running process P 2 (called acontext switch or process switch), it must ensure that P 2 cannot get access to the page tables of P 1 because that would compromise protection. If there is no TLB, it suffices to change the page table register to point to P 2’s page table (rather than to P 1’s); with a TLB, we must clear the TLB entries that belong to P 1—both to protect the data of P 1 and to force the TLB to load the entries for P 2. If the process switch rate were high, this could be quite inefficient. For example, P 2 might load only a few TLB entries before the operating system switched back to P 1. U nfortunately, P 1 would then find that all its TLB entries were gone and would have to pay TLB misses to reload them. This problem arises because the virtual addresses used by P 1 and P 2 are the same, and we must clear out the TLB to avoid confusing these addresses.

c o n t e x t

s w i t c h

A changing of the internal state of the processor to allow a different process to use the processor that includes saving the state needed to return to the currently executing process.

A common alternative is to extend the virtual address space by adding a process identifier or task identifier . The Intrinsity FastMATH has an 8-bit address space ID (AS ID) field for this purpose. This small field identifies the currently running process; it is kept in a register loaded by the operating system when it switches processes. The process identifier is concatenated to the tag portion of the TLB, so that a TLB hit occurs only if both the page number the process identifier match. This combination eliminates the and need to clear the TLB, except on rare occasions. S imilar problems can occur for a cache, since on a process switch the cache will contain data from the running process. These problems arise in different ways for physically addressed and virtually addressed caches, and a variety of different solutions, such as process identifiers, are used to ensure that a process gets its own data.

Handling TLB Misses and Page Faults

Although the translation of virtual to physical addresses with a TLB is straightforward when we get a TLB hit, as we saw earlier, handling TLB misses and page faults is more complex. A TLB miss occurs when no entry in the TLB matches a virtual address. Recall that a TLB miss can indicate one of two possibilities: 1. The page is present in memory, and we need only create the missing TLB entry. 2. The page is not present in memory, and we need to transfer control to the operating system to deal with a page fault. MIP S traditionally handles a TLB miss in software. It brings in the page table entry from memory and then re-executes the instruction that caused the TLB miss. U pon re-executing, it will get a TLB hit. If the page table entry indicates the page is not in memory, this time it will get a page fault exception. Handling a TLB miss or a page fault requires using the exception mechanism to interrupt the active process, transferring control to the operating system, and later resuming execution of the interrupted process. A page fault will be recognized sometime during the clock cycle used to access memory. To restart the instruction after the page fault is handled, the program counter of the instruction that caused the page fault must be saved. Just as in Chapter 4, the exception program counter (EPC) is used to hold this value. In addition, a TLB miss or page fault exception must be asserted by the end of the same clock cycle that the memory access occurs, so that the next clock cycle will begin exception processing rather than continue normal instruction execution. If the page fault was not recognized in this clock cycle, a load instruction could overwrite a register, and this could be disastrous when we try to restart the instruction. For example, consider the instruction lw $1,0($1): the computer must be able to prevent the write pipeline stage from occurring; otherwise, it could not properly restart the instruction, since the contents of $1 would have been destroyed. A similar complication arises on stores. We must prevent the write into memory from actually completing when there is a page fault; this is usually done by deasserting the write control line to the memory.

H a r d w a r e / S o f t w a r e

I n t e r f a c e

Between the time we begin executing the exception handler in the operating system and the time that the operating system has saved all the state of the process, the operating system is particularly vulnerable. For example, if another exception occurred when we were processing the first exception in the operating system, the control unit would overwrite the exception program counter, making it impossible to return to the instruction that caused the page fault! We can avoid this disaster by providing the ability to disable and enable exceptions. When an exception first occurs, the processor sets a bit that disables all other exceptions; this could happen at the same time the processor sets the supervisor mode bit. The operating system will then save just enough state to allow it to recover if another exception occurs—namely, the exception program counter (EP C) and Cause registers. EP C and Cause are two of the special control registers that help with exceptions, TLB misses, and page faults;Figure 5.33 shows the rest. The operating system can then reenable exceptions. These steps make sure that exceptions will not cause the processor to lose any state and thereby be unable to restart execution of the interrupting instruction.

e x c e p t i o n

e n a b l e

Also called interrupt enable. A signal or action that controls whether the process responds to an exception or not; necessary for preventing the occurrence of exceptions during intervals before the processor has safely saved the state needed to restart.

FIGURE 5.33 MIPS control registers. These are considered to be in coprocessor 0, and hence are read using mfc0 and written using mtc0.

Once the operating system knows the virtual address that caused the page fault, it must complete three steps: 1. Look up the page table entry using the virtual address and find the location of the referenced page on disk. 2. Choose a physical page to replace; if the chosen page is dirty, it must be written out to disk before we can bring a new virtual page into this physical page. 3. Start a read to bring the referenced page from disk into the chosen physical page. O f course, this last step will take millions of processor clock cycles (so will the second if the replaced page is dirty); accordingly, the operating system will usually select another process to execute in the processor until the disk access completes. Because the operating system has saved the state of the process, it can freely give control of the processor to another process. When the read of the page from disk is complete, the operating system can restore the state of the process that originally caused the page fault and execute the instruction that returns from the exception. This instruction will reset the processor from kernel to user mode, as well as restore the program counter. The user process then re-executes the instruction that faulted, accesses the requested page successfully, and continues execution. Page fault exceptions for data accesses are difficult to implement properly in a processor because of a combination of three characteristics: 1. They occur in the middle of instructions, unlike instruction page faults. 2. The instruction cannot be completed before handling the exception. 3. After handling the exception, the instruction must be restarted as if nothing had occurred. Making instructions restartable, so that the exception can be handled and the instruction later continued, is relatively easy in an architecture like the MIP S . Because each instruction writes only one data item and this write occurs at the end of the instruction cycle, we can simply prevent the instruction from completing (by not writing) and restart the instruction at the beginning.

r e s t a r t a b l e

i n s t r u c t i o n

An instruction that can resume execution after an exception is resolved without the exception’s affecting the result of the instruction.

Let’s look in more detail at MIP S . When a TLB miss occurs, the MIP S hardware saves the page number of the reference in a special register calle BadVAddr and generates an exception. The exception invokes the operating system, which handles the miss in software. Control is transferred to address 8000 0000hex, the location of the TLB misshandler. To find the physical address for the missing page, the TLB miss routine indexes the page table using the page number of the virtual address and the page table register, which indicates the starting address of the active process page table. To make this indexing fast, MIP S hardware places everything you need in the special Context register: the upper 12 bits have the address of the base of the page table, and the next 18 bits have the virtual address of the missing page. Each page table entry is one word, so the last 2 bits are 0. Thus, the first two instructions copy the Context register into the kernel temporary register $k1 and then load the page table entry from that address into $k1. Recall that $k0 and $k1 are reserved for the operating system to use without saving; a major reason for this convention is to make the TLB miss handler fast. Below is the MIP S code for a typical TLB miss handler:

h a n d l e r
Name of a software routine invoked to “handle” an exception or interrupt.

As shown above, MIP S has a special set of system instructions to update the TLB. The instruction copies from control register EntryLo into tlbwr

As shown above, MIP S has a special set of system instructions to update the TLB. The instruction copies from control register EntryLo into tlbwr the TLB entry selected by the control registerRandom. Random implements random replacement, so it is basically a free-running counter. A TLB miss takes about a dozen clock cycles. Note that the TLB miss handler does not check to see if the page table entry is valid. Because the exception for TLB entry missing is much more frequent than a page fault, the operating system loads the TLB from the page table without examining the entry and restarts the instruction. If the entry is invalid, another and different exception occurs, and the operating system recognizes the page fault. This method makes the frequent case of a TLB miss fast, at a slight performance penalty for the infrequent case of a page fault. O nce the process that generated the page fault has been interrupted, it transfers control to 8000 0180hex, a different address than the TLB miss handler. This is the general address for exception; TLB miss has a special entry point to lower the penalty for a TLB miss. The operating system uses the exception Cause register to diagnose the cause of the exception. Because the exception is a page fault, the operating system knows that extensive processing will be required. Thus, unlike a TLB miss, it saves the entire state of the active process. This state includes all the general-purpose and floating-point registers, the page table address register, the EP C, and the exception Cause register. S ince exception handlers do not usually use the floating-point registers, the general entry point does not save them, leaving that to the few handlers that need them. Figure 5.34 sketches the MIP S code of an exception handler. Note that we save and restore the state in MIP S code, taking care when we enable and disable exceptions, but we invoke C code to handle the particular exception.

FIGURE 5.34 MIPS code to save and restore state on an exception.

The virtual address that caused the fault depends on whether the fault was an instruction or data fault. The address of the instruction that generated the fault is in the EP C. If it was an instruction page fault, the EP C contains the virtual address of the faulting page; otherwise, the faulting virtual address can be computed by examining the instruction (whose address is in the EPC) to find the base register and offset field.

E l a b o r a t i o n
This simplified version assumes that the stack pointer (sp) is valid. To avoid the problem of a page fault during this low-level exception code, MIP S sets aside a portion of its address space that cannot have page faults, called unmapped. The operating system places the exception entry point code and the exception stack in unmapped memory. MIP S hardware translates virtual addresses 8000 0000ex h to BFFF FFFFhex to physical addresses simply by ignoring the upper bits of the virtual address, thereby placing these addresses in the low part of physical memory. Thus, the operating system places exception entry points and exception stacks in unmapped memory.

u n m a p p e d
A portion of the address space that cannot have page faults.

E l a b o r a t i o n

E l a b o r a t i o n
The code in Figure 5.34 shows the MIPS-32 exception return sequence. The older MIPS-I architecture uses rfe and jr instead of eret.

E l a b o r a t i o n
For processors with more complex instructions that can touch many memory locations and write many data items, making instructions restartable is much harder. P rocessing one instruction may generate a number of page faults in the middle of the instruction. For example, x86 processors have block move instructions that touch thousands of data words. In such processors, instructions often cannot be restarted from the beginning, as we do for MIP S instructions. Instead, the instruction must be interrupted and later continued midstream in its execution. Resuming an instruction in the middle of its execution usually requires saving some special state, processing the exception, and restoring that special state. Making this work properly requires careful and detailed coordination between the exception-handling code in the operating system and the hardware.

E l a b o r a t i o n
Rather than pay an extra level of indirection on every memory access, the VMM maintains ashadow page table that maps directly from the guest virtual address space to the physical address space of the hardware. By detecting all modifications to the guest’s page table, the VMM can ensure the shadow page table entries being used by the hardware for translations correspond to those of the guest O S environment, with the exception of the correct physical pages substituted for the real pages in the guest tables. Hence, the VMM must trap any a empt by the guest O S to change its page table or to access the page table pointer. This is commonly done by write protecting the guest page tables and trapping any access to the page table pointer by a guest O S . As noted above, the la er happens naturally if accessing the page table pointer is a privileged operation.

E l a b o r a t i o n
The final portion of the architecture to virtualize is I/O . This is by far the most difficult part of system virtualization because of the increasing number of I/O devices a ached to the computerand the increasing diversity of I/O device types. Another difficulty is the sharing of a real device among multiple VMs, and yet another comes from supporting the myriad of device drivers that are required, especially if different guest O S es are supported on the same VM system. The VM illusion can be maintained by giving each VM generic versions of each type of I/O device driver, and then leaving it to the VMM to handle real I/O.

E l a b o r a t i o n
In addition to virtualizing the instruction set for a virtual machine, another challenge is virtualization of virtual memory, as each guest O S in every virtual machine manages its own set of page tables. To make this work, the VMM separates the notions of and real physical memory (which are often treated synonymously), and makes real memory a separate, intermediate level between virtual memory and physical memory. (S ome use the terms virtual memory, physical memory, and machine memory to name the same three levels.) The guest O S maps virtual memory to real memory via its page tables, and the VMM page tables map the guest’s real memory to physical memory. The virtual memory architecture is specified either via page tables, as in IBM VM/370 and the x86, or via the TLB structure, as in MIPS.

Summary
Virtual memory is the name for the level of memory hierarchy that manages caching between the main memory and secondary memory. Virtual memory allows a single program to expand its address space beyond the limits of main memory. More importantly, virtual memory supports sharing of the main memory among multiple, simultaneously active processes, in a protected manner. Managing the memory hierarchy between main memory and disk is challenging because of the high cost of page faults. S everal techniques are used to reduce the miss rate: 1. Pages are made large to take advantage of spatial locality and to reduce the miss rate. 2. The mapping between virtual addresses and physical addresses, which is implemented with a page table, is made fully associative so that a virtual page can be placed anywhere in main memory. 3. The operating system uses techniques, such as LRU and a reference bit, to choose which pages to replace. Writes to secondary memory are expensive, so virtual memory uses a write-back scheme and also tracks whether a page is unchanged (using a dirty bit) to avoid writing unchanged pages. The virtual memory mechanism provides address translation from a virtual address used by the program to the physical address space used for accessing memory. This address translation allows protected sharing of the main memory and provides several additional benefits, such as simplifying memory allocation. Ensuring that processes are protected from each other requires that only the operating system can change the address translations, which is implemented by preventing user programs from changing the page tables. Controlled sharing of pages among processes can be implemented with the help of the operating system and access bits in the page table that indicate whether the user program has read or write access to a page. If a processor had to access a page table resident in memory to translate every access, virtual memory would be too expensive, as caches would be pointless! Instead, a TLB acts as a cache for translations from the page table. Addresses are then translated from virtual to physical using the translations in the TLB.

translations in the TLB. Caches, virtual memory, and TLBs all rely on a common set of principles and policies. The next section discusses this common framework.

U n d e r s t a n d i n g

P r o g r a m

P e r f o r

Although virtual memory was invented to enable a small memory to act as a large one, the performance difference between secondary memory and main memory means that if a program routinely accesses more virtual memory than it has physical memory, it will run very slowly. Such a program would be continuously swapping pages between memory and disk, called thrashing. Thrashing is a disaster if it occurs, but it is rare. If your program thrashes, the easiest solution is to run it on a computer with more memory or buy more memory for your computer. A more complex choice is to re-examine your algorithm and data structures to see if you can change the locality and thereby reduce the number of pages that your program uses simultaneously. This set of popular pages is informally called the working set. A more common performance problem is TLB misses. S ince a TLB might handle only 32–64 page entries at a time, a program could easily see a high TLB miss rate, as the processor may access less than a quarter mebibyte directly: 64×4 KiB=0.25 MiB. For example, TLB misses are often a challenge for Radix S ort. To try to alleviate this problem, most computer architectures now support variable page sizes. For example, in addition to the standard 4 KiB page, MIP S hardware supports 16 KiB, 64 KiB, 256 KiB, 1 MiB, 4 MiB, 16 MiB, 64 MiB, and 256 MiB pages. Hence, if a program uses large page sizes, it can access more memory directly without TLB misses. The practical challenge is ge ing the operating system to allow programs to select these larger page sizes. O nce again, the more complex solution to reducing TLB misses is to re-examine the algorithm and data structures to reduce the working set of pages; given the importance of memory accesses to performance and the frequency of TLB misses, some programs with large working sets have been redesigned with that goal.

C h e c k Image

Y o u r s e l f

Match the definitions in the right column to the terms in the left column.

5.8 A Common Framework for Memory Hierarchy
By now, you’ve recognized that the different types of memory hierarchies have a great deal in common. Although many of the aspects of memory hierarchies differ quantitatively, many of the policies and features that determine how a hierarchy functions are similar qualitatively. Figure 5.35 shows how some of the quantitative characteristics of memory hierarchies can differ. In the rest of this section, we will discuss the common operational alternatives for memory hierarchies, and how these determine their behavior. We will examine these policies as a series of four questions that apply between any two levels of a memory hierarchy, although for simplicity we will primarily use terminology for caches.

FIGURE 5.35 The key quantitative design parameters that characterize the major elements of memory hierarchy in a computer. These are typical values for these levels as of 2012. Although the range of values is wide, this is partially because many of the values that have shifted over time are related; for example, as caches become larger to overcome larger miss penalties, block sizes also grow. While not shown, server microprocessors today also have L3 caches, which can be 2 to 8 MiB and contain many more blocks than L2 caches. L3 caches lower the L2 miss penalty to 30 to 40 clock cycles.

Question 1: Where Can a Block Be Placed?
We have seen that block placement in the upper level of the hierarchy can use a range of schemes, from direct mapped to set associative to fully associative. As mentioned above, this entire range of schemes can be thought of as variations on a set-associative scheme where the number of sets and the number of blocks per set varies:

The advantage of increasing the degree of associativity is that it usually decreases the miss rate. The improvement in miss rate comes from reducing misses that compete for the same location. We will examine these in more detail shortly. First, let’s look at how much improvement is gained. Figure 5.36 shows the miss rates for several cache sizes as associativity varies from direct mapped to eight-way set associative. The largest gains are obtained in going from direct mapped to two-way set associative, which yields between a 20% and 30% reduction in the miss rate. As cache sizes grow, the relative improvement from associativity increases only slightly; since the overall miss rate of a larger cache is lower, the opportunity for improving the miss rate decreases and the absolute improvement in the miss rate from associativity shrinks significantly. The potential disadvantages of associativity, as we mentioned earlier, are increased cost and slower access time.

FIGURE 5.36 The data cache miss rates for each of eight cache sizes improve as the associativity increases. While the benefit of going from one-way (direct mapped) to two-way set associative is significant, the benefits of further associativity are smaller (e.g., 1%–10% improvement going from two-way to four-way versus 20%–30% improvement going from one-way to two-way). There is even less improvement in going from four-way to eight-way set associative, which, in turn, comes very close to the miss rates of a fully associative cache. Smaller caches obtain a significantly larger absolute benefit from associativity because the base miss rate of a small cache is larger. Figure 5.16 explains how this data was collected.

Question 2: How is a Block Found?
The choice of how we locate a block depends on the block placement scheme, since that dictates the number of possible locations. We can summarize the schemes as follows:

The choice among direct-mapped, set-associative, or fully associative mapping in any memory hierarchy will depend on the cost of a miss versus the cost of implementing associativity, both in time and in extra hardware. Including the L2 cache on the chip enables much higher associativity, because the hit times are not as critical and the designer does not have to rely on standard S RAM chips as the building blocks. Fully associative caches are prohibitive except for small sizes, where the cost of the comparators is not overwhelming and where the absolute miss rate improvements are greatest. In virtual memory systems, a separate mapping table—the page table—is kept to index the memory. In addition to the storage required for the table, using an index table requires an extra memory access. The choice of full associativity for page placement and the extra table is motivated by these facts: 1. Full associativity is beneficial, since misses are very expensive.

1. Full associativity is beneficial, since misses are very expensive. 2. Full associativity allows software to use sophisticated replacement schemes that are designed to reduce the miss rate. 3. The full map can be easily indexed with no extra hardware and no searching required. Therefore, virtual memory systems almost always use fully associative placement. S et-associative placement is often used for caches and TLBs, where the access combines indexing and the search of a small set. A few systems have used direct-mapped caches because of their advantage in access time and simplicity. The advantage in access time occurs because finding the requested block does not depend on a comparison. S uch design choices depend on many details of the implementation, such as whether the cache is on-chip, the technology used for implementing the cache, and the critical role of cache access time in determining the processor cycle time.

Question 3: Which Block Should Be Replaced on a Cache Miss?
When a miss occurs in an associative cache, we must decide which block to replace. In a fully associative cache, all blocks are candidates for replacement. If the cache is set associative, we must choose among the blocks in the set. O f course, replacement is easy in a direct-mapped cache because there is only one candidate. There are the two primary strategies for replacement in set-associative or fully associative caches: ■ Random: Candidate blocks are randomly selected, possibly using some hardware assistance. For example, MIPS supports random replacement for TLB misses. ■ Least recently used (LRU): The block replaced is the one that has been unused for the longest time. In practice, LRU is too costly to implement for hierarchies with more than a small degree of associativity (two to four, typically), since tracking the usage information is costly. Even for four-way set associativity, LRU is often approximated—for example, by keeping track of which pair of blocks is LRU (which requires 1 bit), and then tracking which block in each pair is LRU (which requires 1 bit per pair). For larger associativity, either LRU is approximated or random replacement is used. In caches, the replacement algorithm is in hardware, which means that the scheme should be easy to implement. Random replacement is simple to build in hardware, and for a two-way set-associative cache, random replacement has a miss rate about 1.1 times higher than LRU replacement. As the caches become larger, the miss rate for both replacement strategies falls, and the absolute difference becomes small. In fact, random replacement can sometimes be be er than the simple LRU approximations that are easily implemented in hardware. In virtual memory, some form of LRU is always approximated, since even a tiny reduction in the miss rate can be important when the cost of a miss is enormous. Reference bits or equivalent functionality are often provided to make it easier for the operating system to track a set of less recently used pages. Because misses are so expensive and relatively infrequent, approximating this information primarily in software is acceptable.

Question 4: What Happens on a Write?
A key characteristic of any memory hierarchy is how it deals with writes. We have already seen the two basic options: ■ Write-through: The information is written to both the block in the cache and the block in the lower level of the memory hierarchy (main memory for a cache). The caches in Section 5.3 used this scheme. ■ Write-back: The information is written only to the block in the cache. The modified block is written to the lower level of the hierarchy only when it is replaced. Virtual memory systems always use write-back, for the reasons discussed in Section 5.7. Both write-back and write-through have their advantages. The key advantages of write-back are the following: ■ Individual words can be written by the processor at the rate that the cache, rather than the memory, can accept them. ■ Multiple writes within a block require only one write to the lower level in the hierarchy. ■ When blocks are written back, the system can make effective use of a high-bandwidth transfer, since the entire block is written. Write-through has these advantages: ■ Misses are simpler and cheaper because they never require a block to be written back to the lower level. ■ Write-through is easier to implement than write-back, although to be practical, a write-through cache will still need to use a write buffer.

T h e
Question 1: Answer: Question 2: Answer:

B I G

P i c t u r e

Caches, TLBs, and virtual memory may initially look very different, but they rely on the same two principles of locality, and they can be understood by their answers to four questions:
Where can a block be placed? One place (direct mapped), a few places (set associative), or any place (fully associative). How is a block found? There are four methods: indexing (as in a direct-mapped cache), limited search (as in a set-associative cache), full search (as in a fully associative cache), and a separate lookup table (as in a page table). Question 3: What block is replaced on a miss? Answer: Typically, either the least recently used or a random block. Question 4: How are writes handled? Answer: Each level in the hierarchy can use either write-through or write-back.

In virtual memory systems, only a write-back policy is practical because of the long latency of a write to the lower level of the hierarchy. The rate at which writes are generated by a processor generally exceeds the rate at which the memory system can process them, even allowing for physically and logically wider memories and burst modes for DRAM. Consequently, today lowest-level caches typically use write-back.

The Three Cs: An Intuitive Model for Understanding the Behavior of Memory Hierarchies
In this subsection, we look at a model that provides insight into the sources of misses in a memory hierarchy and how the misses will be affected by changes in the hierarchy. We will explain the ideas in terms of caches, although the ideas carry over directly to any other level in the hierarchy. In this model, all misses are classified into one of three categories (the three Cs):

t h r e e

C s

m o d e l

A cache model in which all cache misses are classified into one of three categories: compulsory misses, capacity misses, and conflict

A cache model in which all cache misses are classified into one of three categories: compulsory misses, capacity misses, and conflict misses.

■ Compulsory misses: These are cache misses caused by the first access to a block that has never been in the cache. These are also called cold-

start misses.

c o m p u l s o r y

m i s s

Also called cold-start miss. A cache miss caused by the first access to a block that has never been in the cache.

■ Capacity misses: These are cache misses caused when the cache cannot contain all the blocks needed during execution of a program. Capacity

misses occur when blocks are replaced and then later retrieved.

c a p a c i t y

m i s s

A cache miss that occurs because the cache, even with full associativity, cannot contain all the blocks needed to satisfy the request.

■ Conflict misses: These are cache misses that occur in set-associative or direct-mapped caches when multiple blocks compete for the same set.

Conflict misses are those misses in a direct-mapped or set-associative cache that are eliminated in a fully associative cache of the same size. These cache misses are also called collision misses.

c o n fl i c t

m i s s

Also called collision miss. A cache miss that occurs in a set-associative or direct-mapped cache when multiple blocks compete for the same set and that are eliminated in a fully associative cache of the same size.

Figure 5.37 shows how the miss rate divides into the three sources. These sources of misses can be directly attacked by changing some aspect of the cache design. S ince conflict misses arise directly from contention for the same cache block, increasing associativity reduces conflict misses. Associativity, however, may slow access time, leading to lower overall performance.

FIGURE 5.37 The miss rate can be broken into three sources of misses. This graph shows the total miss rate and its components for a range of cache sizes. This data is for the SPEC CPU2000 integer and floating-point benchmarks and is from the same source as the data in Figure 5.36 The compulsory miss component is 0.006% and cannot be seen in this graph. The next component is the capacity miss rate, which depends on cache size. The conflict portion, which depends both on associativity and on cache size, is shown for a range of associativities from one-way to eight-way. In each case, the labeled section corresponds to the increase in the miss rate that occurs when the associativity is changed from the next higher degree to the labeled degree of associativity. For example, the section labeled two-way indicates the additional misses arising when the cache has associativity of two rather than four. Thus, the difference in the miss rate incurred by a direct-mapped cache versus a fully associative cache of the same size is given by the sum of the sections marked four-way, two-way, and one-way. The difference between eight-way and eight-way is so small that it is difficult to see on this graph.

Capacity misses can easily be reduced by enlarging the cache; indeed, second-level caches have been growing steadily larger for many years. O f course, when we make the cache larger, we must also be careful about increasing the access time, which could lead to lower overall performance. Thus, first-level caches have been growing slowly, if at all.

Thus, first-level caches have been growing slowly, if at all. Because compulsory misses are generated by the first reference to a block, the primary way for the cache system to reduce the number of compulsory misses is to increase the block size. This will reduce the number of references required to touch each block of the program once, because the program will consist of fewer cache blocks. As mentioned above, increasing the block size too much can have a negative effect on performance because of the increase in the miss penalty.

T h e

B I G

P i c t u r e

The challenge in designing memory hierarchies is that every change that potentially improves the miss rate can also negatively affect overall performance, as Figure 5.38 summarizes. This combination of positive and negative effects is what makes the design of a memory hierarchy interesting.

FIGURE 5.38 Memory hierarchy design challenges.

The decomposition of misses into the three Cs is a useful qualitative model. In real cache designs, many of the design choices interact, and changing one cache characteristic will often affect several components of the miss rate. Despite such shortcomings, this model is a useful way to gain insight into the performance of cache designs.

C h e c k

Y o u r s e l f

Which of the following statements (if any) are generally true? 1. There is no way to reduce compulsory misses. 2. Fully associative caches have no conflict misses. 3. In reducing misses, associativity is more important than capacity.

5.9 Using a Finite-State Machine to Control a Simple Cache
We can now implement control for a cache, just as we implemented control for the single-cycle and pipelined datapaths in Chapter 4. This section starts with a definition of a simple cache and then a description of finite-state machines (FS Ms). It finishes with the FS M of a controller for this simple cache. Section 5.12 goes into more depth, showing the cache and controller in a new hardware description language.

A Simple Cache
We’re going to design a controller for a simple cache. Here are the key characteristics of the cache: ■ Direct-mapped cache ■ Write-back using write allocate ■ Block size is 4 words (16 bytes or 128 bits) ■ Cache size is 16 KiB, so it holds 1024 blocks ■ 32-byte addresses ■ The cache includes a valid bit and dirty bit per block From Section 5.3, we can now calculate the fields of an address for the cache: ■ Cache index is 10 bits ■ Block offset is 4 bits ■ Tag size is 32−(10+4) or 18 bits The signals between the processor to the cache are ■ 1-bit Read or Write signal ■ 1-bit Valid signal, saying whether there is a cache operation or not ■ 32-bit address ■ 32-bit data from processor to cache ■ 32-bit data from cache to processor ■ 1-bit Ready signal, saying the cache operation is complete The interface between the memory and the cache has the same fields as between the processor and the cache, except that the data fields are now 128 bits wide. The extra memory width is generally found in microprocessors today, which deal with either 32-bit or 64-bit words in the processor while the DRAM controller is often 128 bits. Making the cache block match the width of the DRAM simplified the design. Here are the signals: ■ 1-bit Read or Write signal ■ 1-bit Valid signal, saying whether there is a memory operation or not ■ 32-bit address ■ 128-bit data from cache to memory ■ 128-bit data from memory to cache ■ 1-bit Ready signal, saying the memory operation is complete Note that the interface to memory is not a fixed number of cycles. We assume a memory controller that will notify the cache via the Ready signal when the memory read or write is finished. Before describing the cache controller, we need to review finite-state machines, which allow us to control an operation that can take multiple clock cycles.

Finite-State Machines
To design the control unit for the single-cycle datapath, we used a set of truth tables that specified the se ing of the control signals based on the instruction class. For a cache, the control is more complex because the operation can be a series of steps. The control for a cache must specify both the signals to be set in any step and the next step in the sequence. The most common multistep control method is based on finite-state machines, which are usually represented graphically. A finite-state machine consists of a set of states and directions on how to change states. The directions are defined by a next-state function, which maps the current state and the inputs to a new state. When we use a finite-state machine for control, each state also specifies a set of outputs that are asserted when the machine is in that state. The implementation of a finite-state machine usually assumes that all outputs that are not explicitly asserted are deasserted. S imilarly, the correct operation of the datapath depends on the fact that a signal that is not explicitly asserted is deasserted, rather than acting as a don’t care.

fi n i t e - s t a t e

m a c h i n e

A sequential logic function consisting of a set of inputs and outputs, a next-state function that maps the current state and the inputs to a new state, and an output function that maps the current state and possibly the inputs to a set of asserted outputs.

n e x t - s t a t e

f u n c t i o n

A combinational function that, given the inputs and the current state, determines the next state of a finite-state machine.

Multiplexor controls are slightly different, since they select one of the inputs whether they are 0 or 1. Thus, in the finite-state machine, we always specify the se ing of all the multiplexor controls that we care about. When we implement the finite-state machine with logic, se ing a control to 0 may be the default and thus may not require any gates. A simple example of a finite-state machine appears in Appendix B, and if you are unfamiliar with the concept of a finite-state machine, you may want to examine Appendix B before proceeding. A finite-state machine can be implemented with a temporary register that holds the current state and a block of combinational logic that determines both the datapath signals to be asserted and the next state. Figure 5.39 shows how such an implementation might look. Appendix D describes in detail how the finite-state machine is implemented using this structure. In S ection B.3, the combinational control logic for a finite-state

describes in detail how the finite-state machine is implemented using this structure. In S ection B.3, the combinational control logic for a finite-state machine is implemented both with either a ROM (read-only memory) and a PLA (programmable logic array). (Also see Appendix B for a description of these logic elements.)

E l a b o r a t i o n
Note that this simple design is called a blocking cache, in that the processor must wait until the cache has finished the request. Section 5.12 describes the alternative, which is called a nonblocking cache.

E l a b o r a t i o n
The style of finite-state machine in this book is called a Moore machine, after Edward Moore. Its identifying characteristic is that the output depends only on the current state. For a Moore machine, the box labeled combinational control logic can be split into two pieces. One piece has the control output and only the state input, while the other has only the next-state output. An alternative style of machine is a Mealy machine, named after George Mealy. The Mealy machine allows both the input and the current state to be used to determine the output. Moore machines have potential implementation advantages in speed and size of the control unit. The speed advantages arise because the control outputs, which are needed early in the clock cycle, do not depend on the inputs, but only on the current state. In Appendix B, when the implementation of this finite-state machine is taken down to logic gates, the size advantage can be clearly seen. The potential disadvantage of a Moore machine is that it may require additional states. For example, in situations where there is a one-state difference between two sequences of states, the Mealy machine may unify the states by making the outputs depend on the inputs.

FIGURE 5.39 Finite-state machine controllers are typically implemented using a block of combinational logic and a register to hold the current state. The outputs of the combinational logic are the next-state number and the control signals to be asserted for the current state. The inputs to the combinational logic are the current state and any inputs used to determine the next state. Notice that in the finite-state machine used in this chapter, the outputs depend only on the current state, not on the inputs. The Elaboration explains this in more detail.

FSM for a Simple Cache Controller
Figure 5.40 shows the four states of our simple cache controller: ■ Idle: This state waits for a valid read or write request from the processor, which moves the FSM to the Compare Tag state. ■ Compare Tag: As the name suggests, this state tests to see if the requested read or write is a hit or a miss. The index portion of the address selects the tag to be compared. If the data in the cache block referred to by the index portion of the address is valid, and the tag portion of the address matches the tag, then it is a hit. Either the data is read from the selected word if it is a load or written to the selected word if it is a store. The Cache Ready signal is then set. If it is a write, the dirty bit is set to 1. Note that a write hit also sets the valid bit and the tag field; while it seems unnecessary, it is included because the tag is a single memory, so to change the dirty bit we also need to change the valid and tag fields. If it is a hit and the block is valid, the FSM returns to the idle state. A miss first updates the cache tag and then goes either to the Write-Back state, if the block at this location has dirty bit value of 1, or to the Allocate state if it is 0. ■ Write-Back: This state writes the 128-bit block to memory using the address composed from the tag and cache index. We remain in this state waiting for the Ready signal from memory. When the memory write is complete, the FSM goes to the Allocate state. ■ Allocate: The new block is fetched from memory. We remain in this state waiting for the Ready signal from memory. When the memory read is complete, the FSM goes to the Compare Tag state. Although we could have gone to a new state to complete the operation instead of reusing the Compare Tag state, there is a good deal of overlap, including the update of the appropriate word in the block if the access was a write.

FIGURE 5.40 Four states of the simple controller.

This simple model could easily be extended with more states to try to improve performance. For example, the Compare Tag state does both the compare and the read or write of the cache data in a single clock cycle. O ften the compare and cache access are done in separate states to try to improve the clock cycle time. Another optimization would be to add a write buffer so that we could save the dirty block and then read the new block first so that the processor doesn’t have to wait for two memory accesses on a dirty miss. The cache would then write the dirty block from the write buffer while the processor is operating on the requested data. Section 5.12, goes into more detail about the FS M, showing the full controller in a hardware description language and a block diagram of this simple cache.

5.10 Parallelism and Memory Hierarchy: Cache Coherence
Given that a multicore multiprocessor means multiple processors on a single chip, these processors very likely share a common physical address space. Caching shared data introduces a new problem, because the view of memory held by two different processors is through their individual caches, which, without any additional precautions, could end up seeing two different values. Figure 5.41 illustrates the problem and shows how two different processors can have two different values for the same location. This difficulty is generally referred to as the cache coherence problem.

FIGURE 5.41 The cache coherence problem for a single memory location (X), read and written by two processors (A and B). We initially assume that neither cache contains the variable and that X has the value 0. We also assume a write-through cache; a write-back cache adds some additional but similar complications. After the value of X has been written by A, A’s cache and the memory both contain the new value, but B’s cache does not, and if B reads the value of X, it will receive 0!

Informally, we could say that a memory system is coherent if any read of a data item returns the most recently wri en value of that data item. This definition, although intuitively appealing, is vague and simplistic; the reality is much more complex. This simple definition contains two different aspects of memory system behavior, both of which are critical to writing correct shared memory programs. The first aspect, called coherence, defines what values can be returned by a read. The second aspect, called consistency, determines when a wri en value will be returned by a read. Let’s look at coherence first. A memory system is coherent if 1. A read by a processor P to a location X that follows a write by P to X, with no writes of X by another processor occurring between the write and the read by P, always returns the value written by P. Thus, in Figure 5.41, if CPU A were to read X after time step 3, it should see the value 1. 2. A read by a processor to location X that follows a write by another processor to X returns the written value if the read and write are sufficiently separated in time and no other writes to X occur between the two accesses. Thus, in Figure 5.41, we need a mechanism so that the value 0 in the cache of CPU B is replaced by the value 1 after CPU A stores 1 into memory at address X in time step 3. 3. Writes to the same location are serialized; that is, two writes to the same location by any two processors are seen in the same order by all processors. For example, if CPU B stores 2 into memory at address X after time step 3, processors can never read the value at location X as 2 and then later read it as 1. The first property simply preserves program order—we certainly expect this property to be true in uniprocessors, for example. The second property defines the notion of what it means to have a coherent view of memory: if a processor could continuously read an old data value, we would clearly say that memory was incoherent. The need for write serialization is more subtle, but equally important. S uppose we did not serialize writes, and processor P 1 writes location X followed by P 2 writing location X. S erializing the writes ensures that every processor will see the write done by P 2 at some point. If we did not serialize the writes, it might be the case that some processor could see the write of P 2 first and then see the write of P 1, maintaining the value wri en by P1 indefinitely. The simplest way to avoid such difficulties is to ensure that all writes to the same location are seen in the same order, which we call write serialization.

Basic Schemes for Enforcing Coherence In a cache coherent multiprocessor, the caches provide both migration and replication of shared data items: ■ Migration: A data item can be moved to a local cache and used there in a transparent fashion. Migration reduces both the latency to access a shared data item that is allocated remotely and the bandwidth demand on the shared memory.
■ Replication: When shared data are being simultaneously read, the caches make a copy of the data item in the local cache. Replication reduces

both latency of access and contention for a read shared data item. S upporting migration and replication is critical to performance in accessing shared data, so many multiprocessors introduce a hardware protocol to maintain coherent caches. The protocols to maintain coherence for multiple processors are called cache coherence protocols. Key to implementing a cache coherence protocol is tracking the state of any sharing of a data block. The most popular cache coherence protocol is snooping. Every cache that has a copy of the data from a block of physical memory also has a copy of the sharing status of the block, but no centralized state is kept. The caches are all accessible via some broadcast medium (a bus or network), and all cache controllers monitor or snoop on the medium to determine whether or not they have a copy of a block that is requested on a bus or switch access. In the following section we explain snooping-based cache coherence as implemented with a shared bus, but any communication medium that broadcasts cache misses to all processors can be used to implement a snooping-based coherence scheme. This broadcasting to all caches makes snooping protocols simple to implement but also limits their scalability.

Snooping Protocols
O ne method of enforcing coherence is to ensure that a processor has exclusive access to a data item before it writes that item. This style of protocol is called a write invalidate protocol because it invalidates copies in other caches on a write. Exclusive access ensures that no other readable or writable copies of an item exist when the write occurs: all other cached copies of the item are invalidated. Figure 5.42 shows an example of an invalidation protocol for a snooping bus with write-back caches in action. To see how this protocol ensures coherence, consider a write followed by a read by another processor: since the write requires exclusive access, any copy held by the reading processor must be invalidated (hence the protocol name). Thus, when the read occurs, it misses in the cache, and the cache is forced to fetch a new copy of the data. For a write, we require that the writing processor have exclusive access, preventing any other processor from being able to write simultaneously. If two processors do a empt to write the same data simultaneously, one of them wins the race, causing the other processor’s copy to be invalidated. For the other processor to complete its write, it must obtain a new copy of the data, which must now contain the updated value. Therefore, this protocol also enforces write serialization.

FIGURE 5.42 An example of an invalidation protocol working on a snooping bus for a single cache block (X) with write-back caches. We assume that neither cache initially holds X and that the value of X in memory is 0. The CPU and memory contents show the value after the processor and bus activity have both completed. A blank indicates no activity or no copy cached. When the second miss by B occurs, CPU A responds with the value canceling the response from memory. In addition, both the contents of B’s cache and the memory contents of X are updated. This update of memory, which occurs when a block becomes shared, simplifies the protocol, but it is possible to track the ownership and force the write-back only if the block is replaced. This requires the introduction of an additional state called “owner,” which indicates that a block may be shared, but the owning processor is responsible for updating any other processors and memory when it changes the block or replaces it.

H a r d w a r e / S o f t w a r e

I n t e r f a c e

O ne insight is that block size plays an important role in cache coherency. For example, take the case of snooping on a cache with a block size of eight words, with a single word alternatively wri en and read by two processors. Most protocols exchange full blocks between processors, thereby increasing coherency bandwidth demands. Large blocks can also cause what is called false sharing: when two unrelated shared variables are located in the same cache block, the full block is exchanged between processors even though the processors are accessing different variables. P rogrammers and compilers should lay out data carefully to avoid false sharing.

f a l s e

s h a r i n g

When two unrelated shared variables are located in the same cache block and the full block is exchanged between processors even though the processors are accessing different variables.

E l a b o r a t i o n
Although the three properties on pages 466 and 467 are sufficient to ensure coherence, the question of when a wri en value will be seen is also important. To see why, observe that we cannot require that a read of X inFigure 5.41 instantaneously sees the value wri en for X by some other processor. If, for example, a write of X on one processor precedes a read of X on another processor very shortly beforehand, it may be impossible to ensure that the read returns the value of the data wri en, since the wri en data may not even have left the processor at that point. The issue of exactly when a wri en value must be seen by a reader is defined by a memory consistency model. We make the following two assumptions. First, a write does not complete (and allow the next write to occur) until all processors have seen the effect of that write. S econd, the processor does not change the order of any write with respect to any other memory access. These two conditions mean that if a processor writes location X followed by location Y, any processor that sees the new value of Y must also see the new value of X. These restrictions allow the processor to reorder reads, but forces the processor to finish a write in program order.

E l a b o r a t i o n
S ince input can change memory behind the caches and since output could need the latest value in a write back cache, there is also a cache coherency problem for I/O with the caches of a single processor as well as just between caches of multiple processors. The cache coherence problem for multiprocessors and I/O (seeChapter 6), although similar in origin, has different characteristics that affect the appropriate solution. U nlike I/O , where multiple data copies are a rare event—one to be avoided whenever possible—a program running on multiple processors will normally have copies of the same data in several caches.

E l a b o r a t i o n
In addition to the snooping cache coherence protocol where the status of shared blocks is distributed, a directory-based cache coherence protocol keeps the sharing status of a block of physical memory in just one location, called the directory. Directory-based coherence has slightly higher implementation overhead than snooping, but it can reduce traffic between caches and thus scale to larger

has slightly higher implementation overhead than snooping, but it can reduce traffic between caches and thus scale to larger processor counts.

Parallelism and Memory Hierarchy: Redundant Arrays of Inexpensive Disks

This online section describes how using many disks in conjunction can offer much higher throughput, which was the orginal inspiration of Redundant Arrays of Inexpensive Disks(RAID). The real popularlity of RAID, however, was due more to the much greater dependability offered by including a modest number of redundant disks. The section explains the differences in performance, cost, and dependability between the different RAID levels.

Advanced Material: Implementing Cache Controllers
This online section shows how to implement control for a cache, just as we implemented control for the single-cycle and pipelined datapaths in Chapter 4. This section starts with a description of finite-state machines and the implemention of a cache controller for a simple data cache, including a description of the cache controller in a hardware description language. It then goes into details of an example cache coherence protocol and the difficulties in implementing such a protocol.

5.13 Real Stuff: The ARM Cortex-A8 and Intel Core i7 Memory Hierarchies
In this section, we will look at the memory hierarchy of the same two microprocessors described in Chapter 4: the ARM Cortex-A8 and Intel Core i7. This section is based on Section 2.6 of Computer Architecture: A Quantitative Approach, 5th edition. Figure 5.43 summarizes the address sizes and TLBs of the two processors. Note that the A8 has two TLBs with a 32-bit virtual address space and a 32-bit physical address space. The Core i7 has three TLBs with a 48-bit virtual address and a 44-bit physical address. Although the 64-bit registers of the Core i7 could hold a larger virtual address, there was no software need for such a large space and 48-bit virtual addresses shrinks both the page table memory footprint and the TLB hardware.

FIGURE 5.43 Address translation and TLB hardware for the ARM Cortex-A8 and Intel Core i7 920. Both processors provide support for large pages, which are used for things like the operating system or mapping a frame buffer. The large-page scheme avoids using a large number of entries to map a single object that is always present.

Figure 5.44 shows their caches. Keep in mind that the A8 has just one processor or core while the Core i7 has four. Both have identically organized 32 KiB, 4-way set associative, L1 instruction caches (per core) with 64 byte blocks. The A8 uses the same design for data cache, while the Core i7 keeps everything the same except the associativity, which it increases to 8-way. Both use an 8-way set associative unified L2 cache (per core) with 64 byte blocks, although the A8 varies in size from 128 KiB to 1 MiB while the Core i7 is fixed at 256 KiB. As the Core i7 is used for servers,also offers it an L3 cache shared by all the cores on the chip. Its size varies depending on the number of cores. With four cores, as in this case, the size is 8 MiB.

FIGURE 5.44 Caches in the ARM Cortex-A8 and Intel Core i7 920.

A significant challenge facing cache designers is to support processors like the A8 and the Core i7 that can execute more than one memory instruction per clock cycle. A popular technique is to break the cache into banks and allow multiple, independent, parallel accesses, provided the accesses are to different banks. The technique is similar to interleaved DRAM banks (see Section 5.2).

The Core i7 has additional optimizations that allow them to reduce the miss penalty. The first of these is the return of the requested word first on a miss. It also continues to execute instructions that access the data cache during a cache miss. Designers who are a empting to hide the cache miss latency commonly use this technique, called a nonblocking cache, when building out-of-order processors. They implement two flavors of nonblocking. Hit under miss allows additional cache hits during a miss, while miss under miss allows multiple outstanding cache misses. The aim of the first of these two is hiding some miss latency with other work, while the aim of the second is overlapping the latency of two different misses.

n o n b l o c k i n g

c a c h e

A cache that allows the processor to make references to the cache while the cache is handling an earlier miss.

O verlapping a large fraction of miss times for multiple outstanding misses requires a high-bandwidth memory system capable of handling multiple misses in parallel. In a personal mobile device, the memory may only be able to take limited advantage of this capability, but large servers and multiprocessors often have memory systems capable of handling more than one outstanding miss in parallel. The Core i7 has a prefetch mechanism for data accesses. It looks at a pa ern of data misses and use this information to try to predict the next address to start fetching the data before the miss occurs. Such techniques generally work best when accessing arrays in loops. The sophisticated memory hierarchies of these chips and the large fraction of the dies dedicated to caches and TLBs show the significant design effort expended to try to close the gap between processor cycle times and memory latency.

Performance of the A8 and Core i7 Memory Hierarchies

The memory hierarchy of the Cortex-A8 was simulated with a 1 MiB eight-way set associative L2 cache using the integer Minnespec benchmarks. As mentioned in Chapter 4, Minnespec is a set of benchmarks consisting of the S P EC2000 benchmarks but with different inputs that reduce the running times by several orders of magnitude. Although the use of smaller inputs does not change the instruction mix, it does affect the cache behavior. For example, on mcf, the most memory-intensive S P EC2000 integer benchmark, Minnespec has a miss rate for a 32 KiB cache that is only 65% of the miss rate for the full S P EC2000 version. For a 1 MiB cache the difference is a factor of six! For this reason, one cannot compare the Minnespec benchmarks against the S P EC2000 benchmarks, much less the even larger S P EC2006 benchmarks used for the Core i7 Figure 5.47. in Instead, the data are useful for looking at the relative impact of L1 and L2 misses and on overall CPI, which we used in Chapter 4. The A8 instruction cache miss rates for these benchmarks (and also for the full S P EC2000 versions on which Minnespec is based) are very small even for just the L1: close to zero for most and under 1% for all of them. This low rate probably results from the computationally intensive nature of the S P EC programs and the four-way set associative cache that eliminates most conflict misses.Figure 5.45 shows the data cache results for the A8, which have significant L1 and L2 miss rates. The L1 miss penalty for a 1 GHz Cortex-A8 is 11 clock cycles, while the L2 miss penalty is assumed to be 60 clock cycles. Using these miss penalties, Figure 5.46 shows the average miss penalty per data access.

FIGURE 5.45 Data cache miss rates for ARM Cortex-A8 when running Minnespec, a small version of SPEC2000. Applications with larger memory footprints tend to have higher miss rates in both L1 and L2. Note that the L2 rate is the global miss rate; that is, counting all references, including those that hit in L1. (See Elaboration in Section 5.4.) Mcf is known as a cache buster. Note that this figure is for the same systems and benchmarks as Figure 4.76 in Chapter 4.

FIGURE 5.46 The average memory access penalty in clock cycles per data memory reference coming from L1 and L2 is shown for the ARM processor when running Minnespec. Although the miss rates for L1 are significantly higher, the L2 miss penalty, which is more than five times higher, means that the L2 misses can contribute significantly.

Figure 5.47 shows the miss rates for the caches of the Core i7 using the S P EC2006 benchmarks. The L1 instruction cache miss rate varies from 0.1% to 1.8%, averaging just over 0.4%. This rate is in keeping with other studies of instruction cache behavior for the S P ECCP U 2006 benchmarks, which show low instruction cache miss rates. With L1 data cache miss rates running 5% to 10%, and sometimes higher, the importance of the L2 and L3 caches should be obvious. S ince the cost for a miss to memory is over 100 cycles and the average data miss rate in L2 is 4%, L3 is obviously critical. Assuming about half the instructions are loads or stores, without L3 the L2 cache misses could add two cycles per instruction to the CP I! In comparison, the average L3 data miss rate of 1% is still significant but four times lower than the L2 miss rate and six times less than the L1 miss rate.

E l a b o r a t i o n
Because speculation may sometimes be wrong (see Chapter 4), there are references to the L1 data cache that do not correspond to loads or stores that eventually complete execution. The data in Figure 5.45 is measured against all data requests including some that are cancelled. The miss rate when measured against only completed data accesses is 1.6 times higher (an average of 9.5% versus 5.9% for L1 Dcache misses)

FIGURE 5.47 The L1, L2, and L3 data cache miss rates for the Intel Core i7 920 running the full integer SPECCPU2006 benchmarks.

5.14 Going Faster: Cache Blocking and Matrix Multiply
O ur next step in the continuing saga of improving performance of DGEMM by tailoring it to the underlying hardware is to add cache blocking to the subword parallelism and instruction level parallelism optimizations of Chapters 3 and 4. Figure 5.48 shows the blocked version of DGEMM from Figure 4.80. The changes are the same as was made earlier in going from unoptimized DGEMM in Figure 3.21 to blocked DGEMM in Figure 5.21 above. This time we taking the unrolled version of DGEMM from Chapter 4 and invoke it many times on the submatrices of A, B, and C. Indeed, lines 28–34 and lines 7–8 in Figure 5.48 are identical to lines 14–20 and lines 5–6 in Figure 5.21, with the exception of incrementing the for loop in line 7 by the amount unrolled.

FIGURE 5.48 Optimized C version of DGEMM from Figure 4.80 using cache blocking. These changes are the same ones found in Figure 5.21. The assembly language produced by the compiler for the do_block function is nearly identical to Figure 4.81. Once again, there is no overhead to call the do_block because the compiler inlines the function call.

U nlike the earlier chapters, we do not show the resulting x86 code because the inner loop code is nearly identical to Figure 4.81, as the blocking does not affect the computation, just the order that it accesses data in memory. What does change is the bookkeeping integer instructions to implement the for loops. It expands from 14 instructions before the inner loop and 8 after the loop for Figure 4.80 to 40 and 28 instructions respectively for the bookkeeping code generated for Figure 5.48. Nevertheless, the extra instructions executed pale in comparison to the performance improvement of reducing cache misses. Figure 5.49 compares unoptimzed to optimizations for subword parallelism, instruction level parallelism, and caches. Blocking improves performance over unrolled AVX code by factors of 2 to 2.5 for the larger matrices. When we compare unoptimized code to the code with all three optimizations, the performance improvement is factors of 8 to 15, with the largest increase for the largest matrix.

E l a b o r a t i o n
As mentioned in the Elaboration inSection 3.8, these results are with Turbo mode turned off. As in Chapters 3 and 4, when we turn it on we improve all the results by the temporary increase in the clock rate of 3.3/2.6=1.27. Turbo mode works particularly well in this case because it is using only a single core of an eight-core chip. However, if we want to run fast we should use all cores, which we’ll see in Chapter 6.

FIGURE 5.49 Performance of four versions of DGEMM from matrix dimensions 32×32 to 960×960. The fully optimized code for largest matrix is almost 15 times as fast the unoptimized version in Figure 3.21 in Chapter 3.

5.15 Fallacies and Pitfalls
As one of the most naturally quantitative aspects of computer architecture, the memory hierarchy would seem to be less vulnerable to fallacies and pitfalls. Not only have there been many fallacies propagated and pitfalls encountered, but some have led to major negative outcomes. We start with a pitfall that often traps students in exercises and exams.

Pitfall: Ignoring memory system behavior when writing programs or when generating code in a compiler.
This could easily be rewri en as a fallacy: “P rogrammers can ignore memory hierarchies in writing code.” The evaluation of sort inFigure 5.19 and of cache blocking in S ection 5.14 demonstrate that programmers can easily double performance if they factor the behavior of the memory system into the design of their algorithms.

Pitfall: Forgetting to account for byte addressing or the cache block size in simulating a cache.
When simulating a cache (by hand or by computer), we need to make sure we account for the effect of byte addressing and multiword blocks in determining into which cache block a given address maps. For example, if we have a 32-byte direct-mapped cache with a block size of 4 bytes, the byte address 36 maps into block 1 of the cache, since byte address 36 is block address 9 and (9 modulo 8)=1. O n the other hand, if address 36 is a word address, then it maps into block (36 mod 8)=4. Make sure the problem clearly states the base of the address. In like fashion, we must account for the block size. S uppose we have a cache with 256 bytes and a block size of 32 bytes. Into which block does the byte address 300 fall? If we break the address 300 into fields, we can see the answer:

Byte address 300 is block address

The number of blocks in the cache is

Block number 9 falls into cache block number (9 modulo 8)=1. This mistake catches many people, including the authors (in earlier drafts) and instructors who forget whether they intended the addresses to be in words, bytes, or block numbers. Remember this pitfall when you tackle the exercises.

Pitfall: Having less set associativity for a shared cache than the number of cores or threads sharing that cache.
Without extra care, a parallel program running on 2n processors or threads can easily allocate data structures to addresses that would map to the same set of a shared L2 cache. If the cache is at least 2n-way associative, then these accidental conflicts are hidden by the hardware from the program. If not, programmers could face apparently mysterious performance bugs—actually due to L2 conflict misses—when migrating from, say, a 16-core design to 32-core design if both use 16-way associative L2 caches.

Pitfall: Using average memory access time to evaluate the memory hierarchy of an out-of-order processor.
If a processor stalls during a cache miss, then you can separately calculate the memory-stall time and the processor execution time, and hence evaluate the memory hierarchy independently using average memory access time (see page 399). If the processor continues to execute instructions, and may even sustain more cache misses during a cache miss, then the only accurate assessment of the memory hierarchy is to simulate the out-of-order processor along with the memory hierarchy.

Pitfall: Extending an address space by adding segments on top of an unsegmented address space.
During the 1970s, many programs grew so large that not all the code and data could be addressed with just a 16-bit address. Computers were then revised to offer 32-bit addresses, either through an unsegmented 32-bit address space (also called a flat address space ) or by adding 16 bits of segment to the existing 16-bit address. From a marketing point of view, adding segments that were programmer-visible and that forced the programmer and compiler to decompose programs into segments could solve the addressing problem. U nfortunately, there is trouble any time a programming language wants an address that is larger than one segment, such as indices for large arrays, unrestricted pointers, or reference parameters. Moreover, adding segments can turn every address into two words—one for the segment number and one for the segment offset—

parameters. Moreover, adding segments can turn every address into two words—one for the segment number and one for the segment offset— causing problems in the use of addresses in registers.

Fallacy: Disk failure rates in the field match their specifications.

Two recent studies evaluated large collections of disks to check the relationship between results in the field compared to specifications. O ne study was of almost 100,000 disks that had quoted MTTF of 1,000,000 to 1,500,000 hours, or AFR of 0.6% to 0.8%. They found AFRs of 2% to 4% to be common, often three to five times higher than the specified rates [S chroeder and Gibson, 2007]. A second study of more than 100,000 disks at Google, which had a quoted AFR of about 1.5%, saw failure rates of 1.7% for drives in their first year rise to 8.6% for drives in their third year, or about five to six times the specified rate [Pinheiro, Weber, and Barroso, 2007].

Fallacy: Operating systems are the best place to schedule disk accesses.
As mentioned in S ection 5.2, higher-level disk interfaces offer logical block addresses to the host operating system. Given this high-level abstraction, the best an O S can do to try to help performance is to sort the logical block addresses into increasing order. However, since the disk knows the actual mapping of the logical addresses onto the physical geometry of sectors, tracks, and surfaces, it can reduce the rotational and seek latencies by rescheduling. For example, suppose the workload is four reads [Anderson, 2003]:
Operation Starting LBA Length

Read Read Read Read

724 100 9987 26

8 16 1 128

The host might reorder the four reads into logical block order:
Operation Starting LBA Length

Read Read Read Read

26 100 724 9987

128 16 8 1

Depending on the relative location of the data on the disk, reordering could make it worse, as Figure 5.50 shows. The disk-scheduled reads complete in three-quarters of a disk revolution, but the OS-scheduled reads take three revolutions.

Pitfall: Implementing a virtual machine monitor on an instruction set architecture that wasn’t designed to be virtualizable.

FIGURE 5.50 Example showing OS versus disk schedule accesses, labeled host-ordered versus drive-ordered. The former takes three revolutions to complete the four reads, while the latter completes them in just three-fourths of a revolution (from Anderson [2003]).

Many architects in the 1970s and 1980s weren’t careful to make sure that all instructions reading or writing information related to hardware resource information were privileged. This laissez-faire a itude causes problems for VMMs for all of these architectures, including the x86, which we use here as an example. Figure 5.51 describes the 18 instructions that cause problems for virtualization [Robin and Irvine, 2000]. The two broad classes are instructions that ■ Read control registers in user mode that reveals that the guest operating system is running in a virtual machine (such as POPF, mentioned earlier) ■ Check protection as required by the segmented architecture but assume that the operating system is running at the highest privilege level

FIGURE 5.51 Summary of 18 x86 instructions that cause problems for virtualization [Robin and Irvine, 2000]. The first five instructions in the top group allow a program in user mode to read a control register, such as descriptor table registers, without causing a trap. The pop flags instruction modifies a control register with sensitive information but fails silently when in user mode. The protection checking of the segmented architecture of the x86 is the downfall of the bottom group, as each of these instructions checks the privilege level implicitly as part of instruction execution when reading a control register. The checking assumes that the OS must be at the highest privilege level, which is not the case for guest VMs. Only the Move to segment register tries to modify control state, and protection checking foils it as well.

To simplify implementations of VMMs on the x86, both AMD and Intel have proposed extensions to the architecture via a new mode. Intel’s VT-x provides a new execution mode for running VMs, an architected definition of the VMstate, instructions to swap VMs rapidly, and a large set of parameters to select the circumstances where a VMM must be invoked. Altogether, VT-x adds 11 new instructions for the x86. AMD’s Pacifica makes similar proposals. An alternative to modifying the hardware is to make small modifications to the operating system to avoid using the troublesome pieces of the architecture. This technique is called paravirtualization, and the open source Xen VMM is a good example. The Xen VMM provides a guest O S with a virtual machine abstraction that uses only the easy-to-virtualize parts of the physical x86 hardware on which the VMM runs.

5.16 Concluding Remarks
The difficulty of building a memory system to keep pace with faster processors is underscored by the fact that the raw material for main memory, DRAMs, is essentially the same in the fastest computers as it is in the slowest and cheapest. It is the principle of locality that gives us a chance to overcome the long latency of memory access—and the soundness of this strategy is demonstrated at all levels of the memory hierarchy. Although these levels of the hierarchy look quite different in quantitative terms, they follow similar strategies in their operation and exploit the same properties of locality.

Multilevel caches make it possible to use more cache optimizations more easily for two reasons. First, the design parameters of a lower-level cache are different from a first-level cache. For example, because a lower-level cache will be much larger, it is possible to use larger block sizes. S econd, a lower-level cache is not constantly being used by the processor, as a first-level cache is. This allows us to consider having the lower-level cache do something when it is idle that may be useful in preventing future misses. Another trend is to seek software help. Efficiently managing the memory hierarchy using a variety of program transformations and hardware facilities is a major focus of compiler enhancements. Two different ideas are being explored. O ne idea is to reorganize the program to enhance its spatial and temporal locality. This approach focuses on loop-oriented programs that use large arrays as the major data structure; large linear algebra problems are a typical example, such as DGEMM. By restructuring the loops that access the arrays, substantially improved locality—and, therefore, cache performance—can be obtained. Another approach is prefetching. In prefetching, a block of data is brought into the cache before it is actually referenced. Many microprocessors use hardware prefetching to try to predict accesses that may be difficult for software to notice.

p r e f e t c h i n g
A technique in which data blocks needed in the future are brought into the cache early by the use of special instructions that specify the address of the block.

A third approach is special cache-aware instructions that optimize memory transfer. For example, the microprocessors in S ection 6.10 in Chapter 6 use an optimization that does not fetch the contents of a block from memory on a write miss because the program is going to write the full block. This optimization significantly reduces memory traffic for one kernel. As we will see in Chapter 6, memory systems are a central design issue for parallel processors. The growing importance of the memory hierarchy in determining system performance means that this important area will continue to be a focus for both designers and researchers for some years to come.

Historical Perspective and Further Reading
This section, which appears online, gives an overview of memory technologies, from mercury delay lines to DRAM, the invention of the memory hierarchy, protection mechanisms, and virtual machines, and concludes with a brief history of operating systems, including CTS S , MU LTICS UNIX, BSD UNIX, MS-DOS, Windows, and Linux.

5.18 Exercises
5.1 In this exercise we look at memory locality properties of matrix computation. The following code is written in C, where elements within the same row are stored contiguously. Assume each word is a 32-bit integer. for (I=0; I