Previous Table of Contents Next


TABLE 16.1 Noninteracting Factors

A1 A2

B1 3 5
B2 6 8

TABLE 16.2 Interacting Factors

A1 A2

B1 3 5
B2 6 9


FIGURE 16.1  Graphical presentation of interacting and noninteracting factors.

16.2 COMMON MISTAKES IN EXPERIMENTATION

Novice analysts who are not aware of the experimental design and analysis techniques often get misleading conclusions due to the following mistakes:

1.  The variation due to experimental error is ignored. Every measured value is a random value. Each time it is repeated, the measured value would be slightly different even if all the controllable factors are kept at the same value. In making decisions based on measurements, it is important to isolate the effect of errors. The variation due to a factor must be compared with that due to errors before making a decision about its effect. Inexperienced analysts who are not aware of this assign all variation to the factors and completely ignore the errors.
2.  Important parameters are not controlled. Earlier, in Section 2.2, it was pointed out that the list of parameters should include all workload, environment, and system parameters that affect the performance. Only some of these parameters are selected as factors and are varied. For example, when comparing two workstations, the user of the workstation has a significant effect on the measured performance. However, if the effect of users is not correctly accounted for, the results may not be meaningful.
3.  Effects of different factors are not isolated. An analyst may vary several factors simultaneously and then may not be able to allocate the change in performance to any particular factor. To avoid this, some analysts use very simple experimental designs that lead to the problem discussed next.
4.  Simple one-factor-at-a-time designs are used. Such a design is wasteful of the resources. It requires too many experiments to get the same information. With proper experimental design, it is possible to get narrower confidence intervals for the effects with the same number of experiments.
5.  Interactions are ignored. Often the effect of one factor depends on the level of other factors. For example, the effect of adding 1 kbyte of cache, may depend on the size of the program. Such interactions cannot be estimated with one-factor-at-a-time designs.
6.  Too many experiments are conducted. The number of experiments is a function of the number of factors and their levels. It is better to break up the project into several steps each using a small design rather than use one enormous design with too many factors and levels. In the first step, the number of factors and levels should be small. Such a design will help debug the experimental process and also help find out the factors that are not significant and need not be included in further designs. The first design will also tell whether the assumptions of the analysis are satisfied and whether any transformations of data are required. More factors and levels can then be added in the second and subsequent steps.

The experimental design and analysis techniques presented in this part help avoid these problems.

16.3 TYPES OF EXPERIMENTAL DESIGNS

There are numerous varieties of experimental designs. The three most frequently used designs are simple designs, full factorial designs, and fractional factorial designs. Explanations of these designs and their advantanges and disadvantages follow.

16-3.1 Simple Designs

In a simple design, we start with a typical configuration and vary one factor at a time to see how that factor affects performance.

For example, in the workstation design study discussed earlier in Section 16.1, a typical configuration might consist of a Z80 CPU with two disk drives running a managerial task by a college graduate. The performance of this configuration is measured first. Next, we vary the first factor—the CPU—and then performance is compared with other CPUs in the same configuration and workload. This will help us decide which CPU is the best. We then change the number of disk drives to one, three, and four, comparing performance so as to find the optimal number.

Given k factors, with the ith factor having ni levels, a simple design requires only n experiments, where

However, this design does not make the best use of the effort spent. It is not statistically efficient. Also, if the factors have interaction, this design may lead to wrong conclusions. For example, if the effect of the CPU depends upon the size of the memory, the optimal combination cannot be determined until all possibilities are tried. This design, therefore, is not recommended.

16.3.2 Full Factorial Design

A full factorial design utilizes every possible combination at all levels of all factors. A performance study with k factors, with the ith factor having ni levels, requires n experiments, where

In the workstation design study, the number of experiments would be

n = (3 CPUs)(3 memory levels)(4 disk drives)
      × (3 workloads)(3 educational levels)
   = 324 experiments

The advantage of a full factorial design is that every possible combination of configuration and workload is examined. We can find the effect of every factor including the secondary factors and their interactions. The main problem is the cost of the study. It would take too much time and money to conduct these marry experiments, especially when taking into account the possibility that each of these experiments may have to be repeated several times. There are three ways to reduce the number of experiments:

  Reduce the number of levels for each factor.
  Reduce the number of factors.
  Use fractional factorial designs.

The first alternative is specially recommended. In some cases, one can try just two levels of each factor and determine the relative importance of each factor. A full factorial design in which each of the k factors is used at two levels requires 2k experiments. This is a very popular design and is called 2k design. After the list of factors has been reduced substantially, one can try more levels per factor. The third alternative of fractional factorial design is described in the next section.


Previous Table of Contents Next

Copyright © John Wiley & Sons, Inc.