Previous Table of Contents Next


These requirement statements are unacceptable since they suffer from one or more of the following problems:

1.  Nonspecific: No clear numbers are specified. Qualitative words such as low, high, rare, and extremely small are used instead.
2.  Nonmeasurable: There is no way to measure a system and verify that it meets the requirement.
3.  Nonacceptable: Numerical values of requirements, if specified, are set based upon what can be achieved or what looks good. If an attempt is made to set the requirements realistically, they turn out to be so low that they become unacceptable.
4.  Nonrealizable: Often, requirements are set high so that they look good. However, such requirements may not be realizable.
5.  Nonthorough: No attempt is made to specify a possible outcomes.

What all these problems lack can be summarized in one word: SMART. That is, the requirements must be Specific, Measurable, Acceptable, Realizable, and Thorough. Specificity precludes the use of words like “low probability” and “rare.” Measurability requires verification that a given system meets the requirements. Acceptability and realizability demand new configuration limits or architectural decisions so that the requirements are high enough to be acceptable and low enough to be achievable. Thoroughness includes all possible outcomes and failure modes. As discussed in Section 3.2, every system provides a set of services. For every request for a service, there are three possible outcomes: successful performance, incorrect performance, and nonperformance. Thoroughness dictates that the requirements be set on all possible outcomes.

For the requirements to be meaningful, specify bounds, if any, on the configurations, workloads, and environments.
These ideas are illustrated in the following case study.
Case Study 3.2 Consider the problem of specifying the performance requirements for a high-speed LAN system. A LAN basically provides the service of transporting frames (or packets) to the specified destination station. Given a user request to send a frame to destination station D, there are three categories of outcomes: the frame is correctly delivered to D, incorrectly delivered (delivered to a wrong destination or with an error indication to D), or not delivered at all. The performance requirements for these three categories of outcomes were specified as follows:
1.  Speed: If the packet is correctly delivered, the time taken to deliver it and the rate at which it is delivered are important. This leads to the following two requirements:
(a)  The access delay at any station should be less than 1 second.
(b)  Sustained throughput must be at least 80 Mbits/sec.
2.  Reliability: Five different error modes were considered important. Each of these error modes causes a different amount of damage and, hence, has a different level of acceptability. The probability requirements for each of these error modes and their combined effect are specified as follows:
(a)  The probability of any bit being in error must be less than 10-7.
(b)  The probability of any frame being in error (with error indication set) must be less than 1%.
(c)  The probability of a frame in error being delivered without error indication must be less than 10-15.
(d)  The probability of a frame being misdelivered due to an undetected error in the destination address must be less than 10-18.
(e)  The probability of a frame being delivered more than once (duplicate) must be less than 10-5.
(f)  The probability of losing a frame on the LAN (due to all sorts of errors) must be less than 1%.
3.  Availability: Two fault modes were considered significant. The first was the time lost due to the network reinitializations, and the second was time lost due to permanent failures requiring field service calls. The requirements for frequency and duration of these fault modes were specified as follows:
(a)  The mean time to initialize the LAN must be less than 15 milliseconds.
(b)  The mean time between LAN initializations must be at least 1 minute.
(c)  The mean time to repair a LAN must be less than 1 hour. (LAN partitions may be operational during this period.)
(d)  The mean time between LAN partitioning must be at least half a week.

All of the numerical values specified above were checked for realizability by analytical modeling, which showed that LAN systems satisfying these requirements were feasible.

EXERCISES

3.1  What methodology would you choose?
a.  To select a personal computer for yourself
b.  To select 1000 workstations for your company
c.  To compare two spread sheet packages
d.  To compare two data-flow architectures, if the answer was required:
i.  Yesterday
ii.  Next quarter
iii.  Next year
3.2  Make a complete list of metrics to compare
a.  Two personal computers
b.  Two database systems
c.  Two disk drives
d.  Two window systems

FURTHER READING FOR PART I

There are a number of books on computer systems performance evaluation. However, most of these books emphasize only one of the three evaluation techniques.

The book by Lazowska et al. (1984) has an excellent treatment of queueing models. Lavenberg (1983) provides a good review of queueing models and simulation. Ferrari (1978), Ferrari, Serazzi, and Zeigner (1983), and Howard (1983) have good discussions of measurement techniques and their applications to a wide variety of performance problems, such as system tuning, workload characterization, program tuning, and others.

Other books on performance analysis and modeling are Gelenbe and Mitrani (1980), Kobayashi (1978), Leung (1987), McKerrow (1987), Molloy (1989), and Sauer and Chandy (1981).

Majone and Quade (1980) and Koopman (1956) have interesting discussions of common pitfalls of analysis.

The fairness definition presented in Case Study 3.1 is due to Jain, Chiu, and Hawe (1984). The complete case study is described in Jain (1985).


Previous Table of Contents Next

Copyright © John Wiley & Sons, Inc.