Previous Table of Contents Next


These quotations apply to all software development projects including simulation programs. Most simulation efforts become large projects that, unless managed properly, can fail. The following list of common causes of simulation failures is based largely on those pointed out by Annino and Russell (1979):

1.  Inadequate Time Estimate: The first and foremost cause of failures of simulation models is that the model developers underestimate the time and effort required to develop a simulation model. It is common for simulation projects to start off as a “one-week” or “one-month” project and then continue for years. If a simulation is successful and provides useful information, its users want more features, parameters, and details to be added. On the other hand, if a simulation does not provide useful information, it is often expected that adding more features, parameters, and details will probably make it useful. In either case, the project continues far beyond initial projections.
Among the three performance analysis techniques—modeling, measurement, and simulation—the simulation generally takes the longest time, particularly if a complete new model has to be developed from scratch. Time should also be allocated for model verification. Analysts new to the field underestimate the complexity of simulation development. For long simulation projects, provision should be made for incorporating changes in the system, which are inevitable over a long period.
2.  No Achievable Goal: Simulation is a large complex project. Like any other project it should have a clearly specified set of goals that are specific, measurable, achievable, repeatable, and thorough (SMART) (see Section 3.5 for a discussion of SMART goals). The goals should be clearly written down and agreed to by the analysts and the end users of the results before beginning the model development.
A common example of a goal that is not measurable is “to model X.” It is possible to model several different characteristics of X at several different levels of detail. Without proper specification, it is impossible to tell whether the goal has been achieved. The projects without goals continue forever and are eventually terminated when the funding runs out.
3.  Incomplete Mix of Essential Skills: A simulation project requires at least the following four areas of skills:
(a)  Project Leadership: The ability to motivate, lead, and manage the members of the simulation team.
(b)  Modeling and Statistics: The ability to identify the key characteristics of the system and model them at the required level of detail.
(c)  Programming: The ability to write a readable and verifiable computer program that implements the model correctly.
(d)  Knowledge of the Modeled System: The ability to understand the system, explain it to the modeling team, and interpret the modeling results in terms of their impact on the system design.

A simulation team should have members with these skills and be ideally led by a member who has some knowledge of all of the skills.
4.  Inadequate Level of User Participation: It is essential that the modeling team and the user organizations meet periodically and discuss the progress, problems, and changes, if any, in the system. Most systems evolve or change with time, and a model developed without end user participation is rarely successful. The periodic meetings help point out the “modeling bugs” at an early stage and help keep the model in sync with changes in the system.
5.  Obsolete or Nonexistent Documentation: Most simulation models evolve over a long period of time and are continuously modified as the system is modified or better understood. Documentation of these models often lags behind the development and, unless special care is taken to keep it up to date, it soon becomes obsolete. The best strategy is to include the documentation in the program itself and to use computer languages that are easier to read.
6.  Inability to Manage the Development of a Large Complex Computer Program: A number of software engineering tools are available for management of large software projects. These tools help keep track of design objectives, functional requirements, data structures, and progress estimates. Also, a number of design principles, such as top-down design and structured programming, have been developed to help orderly development of large computer programs. Without the use of these tools and techniques, it is impossible to successfully develop a large simulation model.
7.  Mysterious Results: Most mysterious results are due to bugs in the simulation program, invalid model assumptions, or lack of understanding of the real system. The model developers should therefore try to verify the model and, if the mysterious result still persists, bring it to the attention of the end users. It may provide a valuable insight into the system behavior or may point to the system features that need to be modeled in more detail.

The list of mistakes presented in Section 24.1 and the causes of simulation failures presented in this section lead to the checklist presented in Box 24.1. The questions in the list should all be anwered in the affirmative. The list consists of three sublists corresponding to the planning, development, and usage phases of a simulation project.

Box 24.1 Checklist for Simulations

1.  Checks before developing a simulation:
(a)  Is the goal of the simulation properly specified?
(b)  Is the level of detail in the model appropriate for the goal?
(c)  Does the simulation team include personnel with project leadership, modeling, programming, and computer systems backgrounds?
(d)  Has sufficient time been planned for the project?
2.  Checks during development:
(a)  Has the random-number generator used in the simulation been tested for uniformity and independence?
(b)  Is the model reviewed regularly with the end user?
(c)  Is the model documented?
3.  Checks after the simulation is running:
(a)  Is the simulation length appropriate?
(b)  Are the initial transients removed before computation?
(c)  Has the model been verified thoroughly?
(d)  Has the model been validated before using its results?
(e)  If there are any surprising results, have they been validated?
(f)  Are all seeds such that the random-number streams will not overlap?

See also Box 2.1 for a checklist applicable to all performance evaluation projects.


Previous Table of Contents Next

Copyright © John Wiley & Sons, Inc.