TABLE 1.1 Packets Lost on Two Links
|
|
File Size
| Link A
| Link B
|
|
1000
| 5
| 10
|
1200
| 7
| 3
|
1300
| 3
| 0
|
50
| 0
| 1
|
|
Which link is better?
- 4. Design measurement and simulation experiments to provide the most information with the least effort. Given a number of factors that affect the system performance, it is useful to separate out the effects of individual factors. In Part IV on experimental design, techniques to organize experiments to obtain maximum information with a minimum number of experiments are presented. The following is an example of an experimental design question that you should be able to answer after reading that part.
Example 1.4 The performance of a system depends on the following three factors:
- (a) Garbage collection technique used: G1, G2, or none.
- (b) Type of workload: editing, computing, or artificial intelligence (AI).
- (c) Type of CPU: C1, C2, or C3.
How many experiments are needed? How does one estimate the performance impact of each factor?
- 5. Perform simulations correctly. In designing a simulation model, one has to select a language for simulation, select seeds and algorithms for random-number generation, decide the length of simulation run, and analyze the simulation results. These issues are discussed in Part V on simulation. After reading that part, you should be able to answer the following simulation-related question.
Example 1.5 In order to compare the performance of two cache replacement algorithms:
- (a) What type of simulation model should be used?
- (b) How long should the simulation be run?
- (c) What can be done to get the same accuracy with a shorter run?
- (d) How can one decide if the random-number generator in the simulation is a good generator?
- 6. Use simple queueing models to analyze the performance of systems. Queueing models are commonly used for analytical modeling of computer systems. In Part VI, different types of queues and networks of queues are discussed and and their use to answer commonly asked questions about system performance is described. The following is an exercise that you will be able to solve after reading that part.
Example 1.6 The average response time of a database system is 3 seconds. During a 1-minute observation interval, the idle time on the system was 10 seconds. Using a queueing model for the system, determine the following:
- (a) System utilization
- (b) Average service time per query
- (c) Number of queries completed during the observation interval
- (d) Average number of jobs in the system
- (e) Probability of number of jobs in the system being greater than 10
- (f) 90-percentile response time
- (g) 90-percentile waiting time
The preceding six examples correspond to the six parts of this book.
The remainder of Part I explains the steps common to all performance studies beginning with an example of the performance games people play to show that their system is better. Some common mistakes that beginners make are described. In addition, the key components of performance studies, in particular, the selection of performance metrics and evaluation techniques, are discussed.
1.2 THE ART OF PERFORMANCE EVALUATION
Contrary to common belief, performance evaluation is an art. Like a work of art, successful evaluation cannot be produced mechanically. Every evaluation requires an intimate knowledge of the system being modeled and a careful selection of the methodology, workload, and tools. When first presented to an analyst, most performance problems are expressed as an abstract feeling, like a rough sketch, by the end user. Defining the real problem and converting it to a form in which established tools and techniques can be used and where time and other constraints can be met is a major part of the analysts art.
Like an artist, each analyst has a unique style. Given the same problem, two analysts may choose different performance metrics and evaluation methodologies. In fact, given the same data, two analysts may interpret them differently. The following example shows a typical case for which, given the same measurements, two system designers can each prove that his/her system is better than that of the other.
- Example 1.7 The throughputs of two systems A and B were measured in transactions per second. The results are shown in Table 1.2.
TABLE 1.2 Throughput in Transactions per Second
|
|
System
| Workload 1
| Workload 2
|
|
A
| 20
| 10
|
B
| 10
| 20
|
|
- There are three ways to compare the performance of the two systems. The first way is to take the average of the performances on the two workloads. This leads to the analysis shown in Table 1.3. The conclusion in this case is that the two systems are equally good. The second way is to consider the ratio of the performances with system B as the base, as shown in Table 1.4. The conclusion in this case is that system A is better than B. The third way is to consider the performance ratio with system A as the base, as shown in Table 1.5. The conclusion in this case is that system B is better than A.