In 1976, the microcomputer was just beginning to make personal computers a reality for a handful of individuals, the first “hackers”.  The premier systems of the day were built around an S-100 or STD bus, and had Intel 8080 or Motorola 6800 processors running with clock frequencies below 5 Mhz. The primary interactive I/O devices were converted teletypes or alphanumeric terminals from mini and mainframe computers. Primary secondary storage was via converted audio cassette recorders. A fully populated memory board could have as little as 4K of Random Access Memory and the processors could only directly address 64K of RAM. And, even if they could use more memory, the cost and power requirements of the existing static RAM made a system with more than 16K of RAM unusual. Each hacker or vendor had a unique control program, usually embedded in a hardware front panel and every application was responsible for all of its own low level I/O.

In short order, system capabilities and subsystems were standardized and expanded in capability. Interactive I/O was extended to graphics, embedded keyboards, and pointing devices. Secondary storage expanded to include floppy disk drives and then hard disks. And processor capabilities have increased many fold.  The standardization of peripheral and bus interfaces and operating systems led to a commonality that allowed the development of highly sophisticated applications and the use by large numbers of individuals.

In the area of disk subsystems, the major technical changes have dealt with increased capacities, single component reliability, data bandwidth, and decreased access times. None of these changes has altered the fundamental architecture of the disk subsystem. Two recent trends in computers have required the consideration of some fundamental changes in this architecture.

Networks and Virtualization – Two Drivers for Fast, Reliable, and Resilient Storage

Until mid-1990s, the main characteristic of most systems has been their use by individuals. A system failure was isolated to a single system and the individual or function that it supported. The impact was isolated and could be remedied by correcting the fault and rebuilding from the latest backups of the data system. In the worst case, work since the last backup had to be redone. Now, the vast majority of networks are incorporating file servers. The loss of the file server complicates matters considerably. First, the loss of the server incapacitates all of the personal systems using the data from that server, not just a single individual. As more and more companies move databases from their mini and mainframe systems onto network based transaction systems, the loss of the file server can mean a shutdown of corporate functions until the system (and its databases) are restored. Also, since database updates are being received from multiple sources, not just the local machine, it may be difficult or impossible to roll forward from the last backup. Most formal database systems will have an independent transaction log, but it is naive to assume that the developers of custom applications (particularly single machine applications that have been adapted for networks) have incorporated these sorts of sophisticated protection mechanisms. An even if the databases are secure, what about other files, such as a compendium of outgoing correspondence and undelivered and/or saved email? This is further exacerbated by server virtualization – a situation where multiple virtual servers are running on the same physical server while sharing storage. Obviously, what is needed is some way to provide data protection and/or recovery even if the disk subsystem fails.

The general computer user and the popular computer magazines tend to generalize the capabilities of a computer system based upon the type of processor and the clock speed. We talk about an “Intel system” or an “AMD system” or “a Quad Core Xeon machine”. To some extent, these generalizations are valid with regard to memory and instruction architectures and capabilities. But, increasingly, we find test results showing that the performance differences between an Intel-based file server and an AMD-based file server may be considerably less than the differences between two Xeon-based file servers. In examining this apparent paradox, we find that the raw computing power of the processors has little to do with the overall system performance. The processor capabilities have so far exceeded the capabilities of the other subsystems that processor throughput is a negligible component in most cases. Ninety nine percent of the time, the processor is waiting for the disk subsystem or the network adapter to complete some task. The conclusion is obvious. In future systems, high performance will be obtained by integrating a set of balanced subsystems. In the area of disk subsystems, we need higher throughput and faster access times.

Is RAID the answer?

The Redundant Array of Inexpensive Disks, or RAID, provides the ability to deliver both increased reliability and increased performance at moderate cost and using proven existing technology. In the posts that will follow, we’ll attempt to address and explore the following issues: