nsities growing at 60%/year, resulting in 35%-50%/year decreases in the cost per byte. In recent years, the amount of storage sold almost doubled each year and is expected to sustain annual growth of at least 60%. Secondary storage has a healthy place in future computer systems.
While many storage products are directly attached to personal computers, most disk array products (65% and rising) are deployed in local area network file servers. This centralization of storage resources enables effective sharing, better administrative control and less redundancy. However, it also increases dependence on network and file server performance.
With the emergence of high-performance cluster systems based on commodity personal computers and scalable network switching, rapidly increasing demands on storage performance are anticipated. Specifically, storage performance must cost-effectively scale with customer investments in client processors, network links and storage capacity. Unfortunately, current distributed file system architectures severely limit scalable storage. In current distributed file systems, all storage bytes are copied through file server machines between peripheral buses (typically SCSI) and client
LANs. In essence, these file server machines act as application-level inter-network routers, converting namespaces
(disk block versus file range) and protocol layers
(SCSI versus RPC/UDP/IP). This is a critical limitation for cost-effective scalable storage. Specifically, the sustained bandwidth of storage devices is rapidly outstripping installed interconnection technologies and rendering storeand-forward servers impractical. With disk data rates growing at 40% per year, 25-40 MB/s sustained disk bandwidths will be a reality by the end of the decade. With this much bandwidth from each commodity drive, conventional server architectures can not be cost-effective.
Storage devices, however, are already effective network data transfer engines. For example, Seagate’s Fibre
Channel Baracuda drives burst packetized SCSI at 1 Gbps.
Moreover, through careful hardware support for interlayer processing, the marginal cost of these network-attached disk drives is expected to be similar to that of high-end drive interfaces, such as differential SCSI [Anderson95]. It is our contention that cost-effective scalable storage performance depends on eliminating the file server’s role as an inter-network router. Instead, we advocate exploiting the drive’s ability to inject packets directly into the clients’ network at high-bandwidth. With effective network-attached storage, striping of data over multiple devices can effectively scale storage bandwidth [Patterson88, Hartman93].
The other dimension to delivering scalable storage bandwidth is efficient networking. Networking technologies, such as Fibre Channel, are specifically designed for storage systems. However, traditional client-server protocol stacks and operating system software layers often copy data several times in delivering it to applications, significantly reducing network-attached storage performance. Newer technologies, such as user-level networking, avoid this problem by allowing applications to directly access the network.
Hence, user-level networking could be an ideal substrate for network-attached storage.
This papers examines networking requirements for storage and the integration of user-level networking with network-attached storage (NAS). To provide context for the networking demands of NAS, we begin by describing alternative network-attached storage architectures and CMU’s network-attached storage system. Next, we survey storage’s networking requirements and describe how one user-level networking architecture, the VI Architecture (VIA), can be effectively mapped onto our network-attached storage pro for integrity and privacy. The principal limitation of
NetSCSI is that the file manager is still involved in each storage access; it translates namespaces and sets up the third-party transfer on each request.
The second architecture, Network-Attached Secure
Disks (NASD, see Figure 1), relaxes the constraint of minimal change from the existing SCSI interface. The NASD architecture provides a command interface that reduces the number of client-storage interactions that must be relayed through the file manager, thus avoiding a file manager bottleneck without integrating file system policy into the disk.
In NASD, data-intensive operations (e.g., reads and writes) go straight to the disk, while less-common policy making operations (e.g., namespace and access control manipulations) go to the file manager.
1
2
Backplane Bus
Local Area Network
5 3 4
Figure 1: Network-attached secure disks (NASD) are designed to offload more of the file system’s simple and performance-critical operations. For example, in one potential protocol, a client, prior to reading a file, requests access to that file from the file manager (1), which delivers a capability to the authorized client (2). So equipped, the client may make repeated accesses to different regions of the file (3, 4) without contacting the file manager again unless the file manager chooses to force reauthorization by revoking the capability (5).
5
NASD File Manager
Access Control
Network Interface
Network Protocol
Network Driver
Object Storage
Controller
Network Security
Security
Because clients directly request their data, a NASD drive must have sufficient metadata to map and authorize a request for disk sectors. Authorization, in the form of a time-limited capability applicable to a given file’s map and contents, is provided by the file manager to protect the manager’s control over storage access policy. The storage mapping metadata is maintained by the drive, allowing smart drives to better exploit detailed knowledge of their own resources to optimize data layout, read-ahead, and cache management [Cao94, Patterson95, Golding95]. This is precisely the type of value-add opportunity that nimble storage vendors can exploit for market and customer advantage.
With mapping metadata at the drive controlling the layout of files, a NASD drive exports a “namespace” of filelike objects. Because control of naming is more appropriate to the higher-level file system, pathnames are not understood at the drive, and pathname resolution is split between the file manager and client. While a single drive object will suffice to represent a simple client file, multiple objects may be logically linked by the file system into one client file. Such an interface provides support for banks of striped files [Hartman93], Macintosh-style resource forks, or logically-contiguous chunks of complex files [deJong93].
3 NASD Implementation
To experiment with the performance and scalability of
NASD, we designed and implemented a prototype NASD storage interface, ported two popular distributed file systems
(AFS and NFS) to use this interface, and implemented a striped version of NFS on top of this interface
[Gibson97b]. The NASD interface offers logical partitions containing a flat name space of variable length objects with size, time, security, clustering, cloning, and uninterpreted attributes. Access control is enforced by cryptographic capabilities authenticating the arguments of each request to a file manager/drive secret through the use of a digest.
In the NASD/AFS and NASD/NFS filesystems, frequent data-moving operations and attribute read operations occur directly between client and NASD drive, while lessfrequent requests are handled by the file manager. NFS’s simple distributed filesystem model of a stateless server, weak cache consistency, and few mechanisms for filesystem management made it easy to port to a NASD environment; based on a client’s RPC request opcode, RPC destination addresses are modified to deliver requests to the
NASD drive. Our AFS port was more interesting, specifically in maintaining the sequential consistency guarantees of AFS, and in implementing volume quotas. In both cases, we exploited the ability of NASD capabilities to be revoked based on expired time or object attributes (e.g., size).
Using our implementations1
, we compared NASD/AFS and NASD/NFS performance against the traditional ServerAttached
Disk (SAD) implementations of AFS and N