PART V
SIMULATION
This part presents the key issues in simulation modeling. Following is a list of types of questions that you should be able to answer after reading this part:
- What are the common mistakes in simulation and why do most simulations fail?
- What language should you use for developing a simulation model?
- What are the different types of simulations?
- How should you schedule events in a simulation?
- How should you verify and validate a model?
- How should you determine that the simulation has reached a steady state?
- How long should you run a simulation?
- How should you generate uniform random numbers?
- How should you verify that a given random-number generator is good?
- How should you select seeds for a random-number generators?
- How should you generate random variables with a given distribution?
- What distributions should you use and when?
A knowledge of these issues will help you perform simulation correctly and successfully.
CHAPTER 24
INTRODUCTION TO SIMULATION
The best advice to those about to embark on a very large simulation is often the same as Punchs famous advice to those about to marry. Dont!
Bratley, Fox, and Schrage (1986)
Simulation is a useful technique for computer systems performance analysis. If the system to be characterized is not available, as is often the case during the design or procurement stage, a simulation model provides an easy way to predict the performance or compare several alternatives. Further, even if a system is available for measurement, a simulation model may be preferred over measurements because it allows the alternatives to be compared under a wider variety of workloads and environments. Criteria for selecting among simulation, analytical modeling, and measurement have already been discussed in Section 3.1.
However, simulation models often fail; that is, they produce no useful results or misleading results. This is either because the model developers are proficient in software development but lack statistical background or because they are proficient in statistical techniques but are not aware of good software development techniques. More simulation efforts are terminated prematurely than completed. This is because simulation models take a long time to developmuch longer than initially anticipated.
This chapter therefore begins with a list of common mistakes in simulation. Techniques to overcome these mistakes are presented subsequently.
24.1 COMMON MISTAKES IN SIMULATION
- 1. Inappropriate Level of Detail: Simulation allows a system to be studied in more detail than analytical modeling. Analysis requires several simplifications and assumptions. In other words, an analytical model is less detailed. In a simulation model, the level of detail is limited only by the time available for simulation development. A more detailed simulation requires more time to develop. The likelihood of bugs increases and it becomes harder to spot them. The debugging time increases. A more detailed simulation also requires a greater computer time to run. This is particularly important for large simulations where the execution time may be of the order of several hours or days.
It is generally assumed that a more detailed model is a better model, since it makes fewer assumptions. This is not always true. A detailed model may require more detailed knowledge of input parameters, which, if not available, may make the model inaccurate. For example, in simulating a timesharing system, suppose time spent in servicing disk requests needs to be simulated. One alternative is to generate it using an exponential distribution. A more detailed alternative is to simulate the head movement and disk rotation. If the second alternative is chosen, one would get more accurate results only if sector and track references are known. In an actual case study, an analyst chose the second alternative. However, since the sector reference information was not available for inputting into the model, the analyst then decided to generate exponentially distributed random sector numbers. The net result was that the service time came out exponentially distributedan effect that could have been less expensively obtained with the first alternative.
Another problem with detailed models is that they take too much time to develop. It is better to start with a less detailed model, get some results, study sensitivities, and introduce details in the areas that have the highest impact on the results.
- 2. Improper Language: The choice of the programming language has a significant impact on the timely development of the model. Special-purpose simulation languages require less time for model development and ease several common tasks such as verification (using traces) and statistical analysis. General-purpose languages, on the other hand, are more portable and provide more control over the efficiency and run time of the simulation. This issue is discussed further in Section 24.4.
- 3. Unverified Models: Simulation models are generally large computer programs, and unless special precautions are taken, it is possible to have several bugs or programming errors, which would make the conclusions meaningless. A number of techniques to verify simulation models are described in Section 25.1.
- 4. Invalid Models: Even if a simulation program has no errors, it may not represent the real system correctly because of incorrect assumptions about the systems behavior. It is essential that the models be validated to ensure that the conclusion reached would be the same as those obtained from the real system. All simulation model results should be suspected until confirmed by analytical models, measurements, or intuition.
Techniques for model validation are described in Section 25.2.
- 5. Improperly Handled Initial Conditions: The initial part of a simulation trajectory is generally not representative of the system behavior in a steady state. The initial part should therefore be discarded. Several techniques for identifying the initial part are described in Section 25.3.
- 6. Too Short Simulations: Analysts often try to save their own time as well as the computers time by running simulations that are too short. The results in such cases are heavily dependent on initial conditions and may not be representative of the real system. The correct length for simulations depends upon the accuracy (width of the confidence intervals) desired and the variance of the observed quantities.
Straightforward computation of variance from several values of a random variable does not produce a correct estimate of the variance in simulation models because of the correlation among the values. Several techniques for variance estimation are described in Section 25.5.
- 7. Poor Random-Number Generators: Simulation models require random quantities, and the procedures to generate the random numbers are called random-number generators. It is safer to use a well-known generator that has been extensively analyzed than to develop ones own. Even well-known generators have been known to have problems. This is discussed further in Chapter 26.
- 8. Improper Selection of Seeds: Random-number generators are computer procedures that given one random number generate another one. The first random number in the sequence is called the seed and has to be supplied by the analyst. The seed for different random-number streams should be carefully chosen to maintain independence among the streams. Often, analysts either share one stream among several different processes or use the same seed (generally initialized to zeros) for all streams. This introduces a correlation among various processes in the system and leads to conclusions that may not be representative of the real system. Guidelines for proper selection of seeds are presented in Section 26.7.
24.2 OTHER CAUSES OF SIMULATION ANALYSIS FAILURE
Any given program, when running, is obsolete.
If a program is useful, it will have to be changed.
Program complexity grows until it exceeds the capacity
of the programmer who must maintain it.
Datamation 1968
Adding manpower to a late software project makes it later.
Fred Brooks