Previous Table of Contents Next


The above discussion on common mistakes is summarized in Box 2.1, which presents a checklist of questions concerning performance analysis. All questions should be answered affirmatively. The list can also be used by the decision makers to review performance analyses presented to them.

Box 2.1 Checklist for Avoiding Common Mistakes in Performance Evaluation

1.  Is the system correctly defined and the goals clearly stated?
2.  Are the goals stated in an unbiased manner?
3.  Have all the steps of the analysis followed systematically?
4.  Is the problem clearly understood before analyzing it?
5.  Are the performance metrics relevant for this problem?
6.  Is the workload correct for this problem?
7.  Is the evaluation technique appropriate?
8.  Is the list of parameters that affect performance complete?
9.  Have all parameters that affect performance been chosen as factors to be varied?
10.  Is the experimental design efficient in terms of time and results?
11.  Is the level of detail proper?
12.  Is the measured data presented with analysis and interpretation?
13.  Is the analysis statistically correct?
14.  Has the sensitivity analysis been done?
15.  Would errors in the input cause an insignificant change in the results?
16.  Have the outliers in the input or output been treated properly?
17.  Have the future changes in the system and workload been modeled?
18.  Has the variance of input been taken into account?
19.  Has the variance of the results been analyzed?
20.  Is the analysis easy to explain?
21.  Is the presentation style suitable for its audience?
22.  Have the results been presented graphically as much as possible?
23.  Are the assumptions and limitations of the analysis clearly documented?

2.2 A SYSTEMATIC APPROACH TO PERFORMANCE EVALUATION

Most performance problems are unique. The metrics, workload, and evaluation techniques used for one problem generally cannot be used for the next problem. Nevertheless, there are steps common to all performance evaluation projects that help you avoid the common mistakes listed in Section 2.1. These steps are as follows.

1.  State Goals and Define the System: The first step in any performance evaluation project is to state the goals of the study and define what constitutes the system by delineating system boundaries. Given the same set of hardware and software, the definition of the system may vary depending upon the goals of the study. Given two CPUs, for example, the goal may be to estimate their impact on the response time of interactive users. In this case, the system would consist of the timesharing system, and the conclusions of the study may depend significantly on components external to the CPU. On the other hand, if the two CPUs are basically similar except for their Arithmetic-Logic Units (ALUs) and the goal is to decide which ALU should be chosen, the CPUs may be considered the system’s and only the components inside the CPU may be considered part of the system.
The choice of system boundaries affects the performance metrics as well as workloads used to compare the systems. Therefore, understanding the system boundaries is important. Although the key consideration in setting the system boundaries is the objective of the study, other considerations, such as administrative control of the sponsors of the study, may also need to be taken into account. If the sponsors do not have a control over some components, they may want to keep those components outside the system boundaries.
2.  List Services and Outcomes: Each system provides a set of services. For example, a computer network allows its users to send packets to specified destinations. A database system responds to queries. A processor performs a number of different instructions. The next step in analyzing a system is to list these services. When a user requests any of these services, there are a number of possible outcomes. Some of these outcomes are desirable and some are not. For example, a database system may answer a query correctly, incorrectly (due to inconsistent updates), or not at all (due to deadlocks or some similar problems). A list of services and possible outcomes is useful later in selecting the right metrics and workloads.
3.  Select Metrics: The next step is to select criteria to compare the performance. These criteria are called metrics. In general, the metrics are related to the speed, accuracy, and availability of services. The performance of a network, for example, is measured by the speed (throughput and delay), accuracy (error rate), and availability of the packets sent. The performance of a processor is measured by the speed of (time taken to execute) various instructions. The selection of the correct metrics is discussed in Section 3.2.
4.  List Parameters: The next step in performance projects is to make a list of all the parameters that affect performance. The list can be divided into system parameters and workload parameters. System parameters include both hardware and software parameters, which generally do not vary among various installations of the system. Workload parameters are characteristics of users’ requests, which vary from one installation to the next.
The list of parameters may not be complete. That is, after the first pass of the analysis, you may discover that there are additional parameters that affect the performance. You can then add these parameters to the list, but at all times keep the list as comprehensive as possible. This allows the analyst and decision makers to discuss the impact of various parameters and determine what data needs to be collected before or during the analysis.
5.  Select Factors to Study: The list of parameters can be divided into two parts: those that will be varied during the evaluation and those that will not. The parameters to be varied are called factors and their values are called levels. In general, the list of factors, and their possible levels, is larger than what the available resources will allow. Otherwise, the list will keep growing until it becomes obvious that there are not enough resources to study the problem. It is better to start with a short list of factors and a small number of levels for each factor and to extend the list in the next phase of the project if the resources permit. For example, you may decide to have only two factors: quantum size and the number of users. For each of these two factors you may choose only two levels: small and large. The working set size and the type of workload may be fixed.
The parameters that are expected to have a high impact on the performance should be preferably selected as factors. Like metrics, a common mistake in selecting the factors is that the parameters that are easy to vary and measure are used as factors while other more influential parameters are ignored simply because of the difficulty involved.
In selecting factors, it is important to consider the economic, political, and technological constraints that exist as well as including the limitations imposed by the decision makers’ control and the time available for the decision. This increases the chances of finding a solution that is acceptable and implementable.


Previous Table of Contents Next

Copyright © John Wiley & Sons, Inc.