Preface
Acknowledgments

PART I—AN OVERVIEW OF PERFORMANCE EVALUATION
CHAPTER 1—INTRODUCTION
1.1 OUTLINE OF TOPICS
1.2 THE ART OF PERFORMANCE EVALUATION
1.3 PROFESSIONAL ORGANIZATIONS, JOURNALS, AND CONFERENCES
1.4 PERFORMANCE PROJECTS
EXERCISE
CHAPTER 2—COMMON MISTAKES AND HOW TO AVOID THEM
2.1 COMMON MISTAKES IN PERFORMANCE EVALUATION
2.2 A SYSTEMATIC APPROACH TO PERFORMANCE EVALUATION
EXERCISES
CHAPTER 3—SELECTION OF TECHNIQUES AND METRICS
3.1 SELECTING AN EVALUATION TECHNIQUE
3.2 SELECTING PERFORMANCE METRICS
3.3 COMMONLY USED PERFORMANCE METRICS
3.4 UTILITY CLASSIFICATION OF PERFORMANCE METRICS
3.5 SETTING PERFORMANCE REQUIREMENTS
EXERCISES
FURTHER READING FOR PART I

PART II—MEASUREMENT TECHNIQUES AND TOOLS
CHAPTER 4—TYPES OF WORKLOADS
4.1 ADDITION INSTRUCTION
4.2 INSTRUCTION MIXES
4.3 KERNELS
4.4 SYNTHETIC PROGRAMS
4.5 APPLICATION BENCHMARKS
4.6 POPULAR BENCHMARKS
4.6.1 Sieve
4.6.2 Ackermann’s Function
4.6.3 Whetstone
4.6.4 LINPACK
4.6.5 Dhrystone
4.6.6 Lawrence Livermore Loops
4.6.7 Debit-Credit Benchmark
4.6.8 SPEC Benchmark Suite
EXERCISES
CHAPTER 5—THE ART OF WORKLOAD SELECTION
5.1 SERVICES EXERCISED
5.2 LEVEL OF DETAIL
5.3 REPRESENTATIVENESS
5.4 TIMELINESS
5.5 OTHER CONSIDERATIONS IN WORKLOAD SELECTION
EXERCISES
CHAPTER 6—WORKLOAD CHARACTERIZATION TECHNIQUES
6.1 TERMINOLOGY
6.2 AVERAGING
6.3 SPECIFYING DISPERSION
6.4 SINGLE-PARAMETER HISTOGRAMS
6.5 MULTIPARAMETER HISTOGRAMS
6.6 PRINCIPAL-COMPONENT ANALYSIS
6.7 MARKOV MODELS
6.8 CLUSTERING
6.8.1 Sampling
6.8.2 Parameter Selection
6.8.3 Transformation
6.8.4 Outliers
6.8.5 Data Scaling
6.8.6 Distance Metric
6.8.7 Clustering Techniques
6.8.8 Minimum Spanning Tree Method
6.8.9 Cluster Interpretation
6.8.10 Problems with Clustering
EXERCISES
CHAPTER 7—MONITORS
7.1 MONITOR TERMINOLOGY
7.2 MONITOR CLASSIFICATION
7.3 SOFTWARE MONITORS
7.3.1 Issues In Software Monitor Design
7.4 HARDWARE MONITORS
7.5 SOFTWARE VERSUS HARDWARE MONITORS
7.6 FIRMWARE AND HYBRID MONITORS
7.7 DISTRIBUTED-SYSTEM MONITORS
7.7.1 Observation
7.7.2 Collection
7.7.3 Analysis
7.7.4 Presentation
7.7.5 Interpretation
7.7.6 Console Functions
EXERCISES
CHAPTER 8—PROGRAM EXECUTION MONITORS AND ACCOUNTING LOGS
8.1 PROGRAM EXECUTION MONITORS
8.1.1 Issues In Designing a Program Execution Monitor
8.2 TECHNIQUES FOR IMPROVING PROGRAM PERFORMANCE
8.3 ACCOUNTING LOGS
8.4 ANALYSIS AND INTERPRETATION OF ACCOUNTING LOG DATA
8.5 USING ACCOUNTING LOGS TO ANSWER COMMONLY ASKED QUESTIONS
EXERCISE
CHAPTER 9—CAPACITY PLANNING AND BENCHMARKING
9.1 STEPS IN CAPACITY PLANNING AND MANAGEMENT
9.2 PROBLEMS IN CAPACITY PLANNING
9.3 COMMON MISTAKES IN BENCHMARKING
9.4 BENCHMARKING GAMES
9.5 LOAD DRIVERS
9.6 REMOTE-TERMINAL EMULATION
9.7 COMPONENTS OF AN RTE
9.8 LIMITATIONS OF CURRENT RTES
EXERCISES
CHAPTER 10—THE ART OF DATA PRESENTATION
10.1 TYPES OF VARIABLES
10.2 GUIDELINES FOR PREPARING GOOD GRAPHIC CHARTS
10.3 COMMON MISTAKES IN PREPARING CHARTS
10.4 PICTORIAL GAMES
10.5 GANTT CHARTS
10.6 KIVIAT GRAPHS
10.6.1 Shapes of Kiviat Graphs
10.6.2 Application of Kiviat Graphs to Other Systems
10.7 SCHUMACHER CHARTS
10.8 DECISION MAKER’S GAMES
CHAPTER 11—RATIO GAMES
11.1 CHOOSING AN APPROPRIATE BASE SYSTEM
11.2 USING AN APPROPRIATE RATIO METRIC
11.3 USING RELATIVE PERFORMANCE ENHANCEMENT
11.4 RATIO GAMES WITH PERCENTAGES
11.5 STRATEGIES FOR WINNING A RATIO GAME
11.6 CORRECT ANALYSIS
EXERCISES
FURTHER READING FOR PART II

PART III—PROBABILITY THEORY AND STATISTICS
CHAPTER 12—SUMMARIZING MEASURED DATA
12.1 BASIC PROBABILITY AND STATISTICS CONCEPTS
12.2 SUMMARIZING DATA BY A SINGLE NUMBER
12.3 SELECTING AMONG THE MEAN, MEDIAN, AND MODE
12.4 COMMON MISUSES OF MEANS
12.5 GEOMETRIC MEAN
12.6 HARMONIC MEAN
12.7 MEAN OF A RATIO
12.8 SUMMARIZING VARIABILITY
12.9 SELECTING THE INDEX OF DISPERSION
12.10 DETERMINING DISTRIBUTION OF DATA
EXERCISES
CHAPTER 13—COMPARING SYSTEMS USING SAMPLE DATA
13.1 SAMPLE VERSUS POPULATION
13.2 CONFIDENCE INTERVAL FOR THE MEAN
13.3 TESTING FOR A ZERO MEAN
13.4 COMPARING TWO ALTERNATIVES
13.4.1 Paired Observations
13.4.2 Unpaired Observations
13.4.3 Approximate Visual Test
13.5 WHAT CONFIDENCE LEVEL TO USE
13.6 HYPOTHESIS TESTING VERSUS CONFIDENCE INTERVALS
13.7 ONE-SIDED CONFIDENCE INTERVALS
13.8 CONFIDENCE INTERVALS FOR PROPORTIONS
13.9 DETERMINING SAMPLE SIZE
13.9.1 Sample Size for Determining Mean
13.9.2 Sample Size for Determining Proportions
13.9.3 Sample Size for Comparing Two Alternatives
CHAPTER 14—SIMPLE LINEAR REGRESSION MODELS
14.1 DEFINITION OF A GOOD MODEL
14.2 ESTIMATION OF MODEL PARAMETERS
14.5 CONFIDENCE INTERVALS FOR REGRESSION PARAMETERS
14.6 CONFIDENCE INTERVALS FOR PREDICTIONS
EXERCISES
CHAPTER 15—OTHER REGRESSION MODELS
15.1 MULTIPLE LINEAR REGRESSION MODELS
15.1.1 Analysis of Variance
15.1.2 Problem of Multicollinearity
15.2 REGRESSION WITH CATEGORICAL PREDICTORS
15.3 CURVILINEAR REGRESSION
15.4 TRANSFORMATIONS
15.5 OUTLIERS
15.6 COMMON MISTAKES IN REGRESSION
EXERCISES
FURTHER READING FOR PART III

PART IV—EXPERIMENTAL DESIGN AND ANALYSIS
CHAPTER 16—INTRODUCTION TO EXPERIMENTAL DESIGN
16.1 TERMINOLOGY
16.2 COMMON MISTAKES IN EXPERIMENTATION
16.3 TYPES OF EXPERIMENTAL DESIGNS
16-3.1 Simple Designs
16.3.2 Full Factorial Design
16.3.3 Fractional Factorial Designs
EXERCISE
CHAPTER 17—2k FACTORIAL DESIGNS
17.1 22 FACTORIAL DESIGNS
17.2 COMPUTATION OF EFFECTS
17.3 SIGN TABLE METHOD FOR CALCULATING EFFECTS
17.4 ALLOCATION OF VARIATION
17.5 GENERAL 2k FACTORIAL DESIGNS
EXERCISE
CHAPTER 18—2kr FACTORIAL DESIGNS WITH REPLICATIONS
18.1 22r FACTORIAL DESIGNS
18.2 COMPUTATION OF EFFECTS
18.3 ESTIMATION OF EXPERIMENTAL ERRORS
18.4 ALLOCATION OF VARIATION
18.5 CONFIDENCE INTERVALS FOR EFFECTS
18.6 CONFIDENCE INTERVALS FOR PREDICTED RESPONSES
18.7 VISUAL TESTS FOR VERIFYING THE ASSUMPTIONS
18.9 GENERAL 2kr FACTORIAL DESIGN
EXERCISE
CHAPTER 19—2k–p FRACTIONAL FACTORIAL DESIGNS
19.1 PREPARING THE SIGN TABLE FOR A 2k–p DESIGN
19.2 CONFOUNDING
19.3 ALGEBRA OF CONFOUNDING
19.4 DESIGN RESOLUTION
EXERCISES
CHAPTER 20—ONE-FACTOR EXPERIMENTS
20.1 MODEL
20.2 COMPUTATION OF EFFECTS
20.3 ESTIMATING EXPERIMENTAL ERRORS
20.4 ALLOCATION OF VARIATION
20.5 ANALYSIS OF VARIANCE
20.6 VISUAL DIAGNOSTIC TESTS
20.7 CONFIDENCE INTERVALS FOR EFFECTS
20.8 UNEQUAL SAMPLE SIZES
EXERCISE
CHAPTER 21—TWO-FACTOR FULL FACTORIAL DESIGN WITHOUT REPLICATIONS
21.1 MODEL
21.2 COMPUTATION OF EFFECTS
21.3 ESTIMATING EXPERIMENTAL ERRORS
21.4 ALLOCATION OF VARIATION
21.5 ANALYSIS OF VARIANCE
21.6 CONFIDENCE INTERVALS FOR EFFECTS
21.7 MULTIPLICATIVE MODELS FOR TWO-FACTOR EXPERIMENTS
21.8 MISSING OBSERVATIONS
EXERCISES
CHAPTER 22—TWO-FACTOR FULL FACTORIAL DESIGN WITH REPLICATIONS
22.1 MODEL
22.2 COMPUTATION OF EFFECTS
22.3 COMPUTATION OF ERRORS
22.4 ALLOCATION OF VARIATION
22.5 ANALYSIS OF VARIANCE
22.6 CONFIDENCE INTERVALS FOR EFFECTS
CHAPTER 23—GENERAL FULL FACTORIAL DESIGNS WITH k FACTORS
23.1 MODEL
23.2 ANALYSIS OF A GENERAL DESIGN
23.3 INFORMAL METHODS
23.3.1 Observation Method
23.3.2 Ranking Method
23.3.3 Range Method
EXERCISES
FURTHER READING FOR PART IV

PART V—SIMULATION
CHAPTER 24—INTRODUCTION TO SIMULATION
24.1 COMMON MISTAKES IN SIMULATION
24.2 OTHER CAUSES OF SIMULATION ANALYSIS FAILURE
24.3 TERMINOLOGY
24.4 SELECTING A LANGUAGE FOR SIMULATION
24.5 TYPES OF SIMULATIONS
24.5.1 Monte Carlo Simulation
24.5.2 Trace-Driven Simulation
24.5.3 Discrete-Event Simulations
24.6 EVENT-SET ALGORITHMS
EXERCISES
CHAPTER 25—ANALYSIS OF SIMULATION RESULTS
25.1 MODEL VERIFICATION TECHNIQUES
25.1.1 Top-Down Modular Design
25.1.2 Antibugging
25.1.3 Structured Walk-Through
25.1.4 Deterministic Models
25.1.5 Run Simplified Cases
25.1.6 Trace
25.1.7 On-Line Graphic Displays
25.1.8 Continuity Test
25.1.9 Degeneracy Tests
25.1.10 Consistency Tests
25.1.11 Seed Independence
25.2 MODEL VALIDATION TECHNIQUES
25.2.1 Expert Intuition
25.2.2 Real-System Measurements
25.2.3 Theoretical Results
25.3 TRANSIENT REMOVAL
25.3.1 Long Runs
25.3.2 Proper Initialization
25.3.3 Truncation
25.3.4 Initial Date Deletion
25.3.5 Moving Average of Independent Replications
25.3.6 Batch Means
25.4 TERMINATING SIMULATIONS
25.5 STOPPING CRITERIA: VARIANCE ESTIMATION
25.5.1 Independent Replications
25.5.2 Batch Means
25.5.3 Method of Regeneration
25.6 VARIANCE REDUCTION
EXERCISES
CHAPTER 26—RANDOM-NUMBER GENERATION
26.1 DESIRED PROPERTIES OF A GOOD GENERATOR
26.2 LINEAR-CONGRUENTIAL GENERATORS
26.2.1 Multiplicative LCG
26.2.2 Multiplicative LCG with m = 2k
26.2.3 Multiplicative LCG with m ≠ 2k
26.3 TAUSWORTHE GENERATORS
26.4 EXTENDED FIBONACCI GENERATORS
26.5 COMBINED GENERATORS
26.6 A SURVEY OF RANDOM-NUMBER GENERATORS
26.7 SEED SELECTION
26.8 MYTHS ABOUT RANDOM-NUMBER GENERATION
EXERCISES
CHAPTER 27—TESTING RANDOM-NUMBER GENERATORS
27.1 CHI-SQUARE TEST
27.2 KOLMOGOROV-SMIRNOV TEST
27.3 SERIAL-CORRELATION TEST
27.4 TWO-LEVEL TESTS
27.5 k-DIMENSIONAL UNIFORMITY OR k-DISTRIBUTIVITY
27.6 SERIAL TEST
27.7 SPECTRAL TEST
EXERCISES
CHAPTER 28—RANDOM-VARIATE GENERATION
28.1 INVERSE TRANSFORMATION
28.2 REJECTION
28.3 COMPOSITION
28.4 CONVOLUTION
28.5 CHARACTERIZATION
EXERCISE
CHAPTER 29—COMMONLY USED DISTRIBUTIONS
29.1 BERNOULLI DISTRIBUTION
29.2 BETA DISTRIBUTION
29.3 BINOMIAL DISTRIBUTION
29.4 CHI-SQUARE DISTRIBUTION
29.5 ERLANG DISTRIBUTION
29.6 EXPONENTIAL DISTRIBUTION
29.7 F DISTRIBUTION
29.8 GAMMA DISTRIBUTION
29.9 GEOMETRIC DISTRIBUTION
29.10 LOGNORMAL DISTRIBUTION
29.11 NEGATIVE BINOMIAL DISTRIBUTION
29.12 NORMAL DISTRIBUTION
29.13 PARETO DISTRIBUTION
29.14 PASCAL DISTRIBUTION
29.15 POISSON DISTRIBUTION
29.16 STUDENT’S t DISTRIBUTION
29.17 UNIFORM DISTRIBUTION (CONTINUOUS)
29.18 UNIFORM DISTRIBUTION (DISCRETE)
29.19 WEIBULL DISTRIBUTION
29.20 RELATIONSHIPS AMONG DISTRIBUTIONS
EXERCISES
FURTHER READING FOR PART V
CURRENT AREAS OF RESEARCH IN SIMULATION

PART VI—QUEUEING MODELS
CHAPTER 30—INTRODUCTION TO QUEUEING THEORY
30.1 QUEUEING NOTATION
30.2 RULES FOR ALL QUEUES
30.3 LITTLE’S LAW
30.4 TYPES OF STOCHASTIC PROCESSES
EXERCISES
CHAPTER 31—ANALYSIS OF A SINGLE QUEUE
31.1 BIRTH-DEATH PROCESSES
31.2 M/M/1 QUEUE
31.3 M/M/m QUEUE
31.4 M/M/m/B OUEUE WITH FINITE BUFFERS
31.5 RESULTS FOR OTHER QUEUEING SYSTEMS
EXERCISES
CHAPTER 32—QUEUEING NETWORKS
32.1 OPEN AND CLOSED QUEUEING NETWORKS
32.2 PRODUCT FORM NETWORKS
32.3 QUEUEING NETWORK MODELS OF COMPUTER SYSTEMS
EXERCISE
CHAPTER 33—OPERATIONAL LAWS
33.1 UTILIZATION LAW
33.2 FORCED FLOW LAW
33.3 LITTLE'S LAW
33.4 GENERAL RESPONSE TIME LAW
33.5 INTERACTIVE RESPONSE TIME LAW
33.6 BOTTLENECK ANALYSIS
EXERCISES
CHAPTER 34—MEAN-VALUE ANALYSIS AND RELATED TECHNIQUES
34.1 ANALYSIS OF OPEN QUEUEING NETWORKS
34.2 MEAN-VALUE ANALYSIS
34.3 APPROXIMATE MVA
34.4 BALANCED JOB BOUNDS
EXERCISES
CHAPTER 35—CONVOLUTION ALGORITHM
35.1 DISTRIBUTION OF JOBS IN A SYSTEM
35.2 CONVOLUTION ALGORITHM FOR COMPUTING G(N)
35.3 COMPUTING PERFORMANCE USING G(N)
35.4 TIMESHARING SYSTEMS
EXERCISES
CHAPTER 36—HIERARCHICAL DECOMPOSITION OF LARGE QUEUEING NETWORKS
36.1 LOAD-DEPENDENT SERVICE CENTERS
36.2 HIERARCHICAL DECOMPOSITION
36.3 LIMITATIONS OF QUEUEING THEORY
EXERCISES
FURTHER READING FOR PART VI
SYMBOLS FREQUENTLY USED IN QUEUEING ANALYSIS
References
Appendix A
Appendix B
Appendix C
Subject Index