|
|
 |
| |
|
| |
What are the most common mistakes made when building a simulation?
|
As in any software development, there is a lot of room for mistakes when building a simulation. This Fact Sheet focuses on the unique "opportunities" to make mistakes when developing a simulation, touching on some of the more common software errors when appropriate. This Fact Sheet references the development processes described in Fact Sheet #4.
|
|
1) Representation Requirements - The first chance to make a mistake in any development endeavor is in the process of gathering user requirements, identifying derived user requirements and turning these into system requirements. Along with all the typical types of mistakes in this phase of a standard system development, simulation developers may forget or ignore the need to clearly identify the specific uses of the simulation. It is not enough to say, "We need to do aircraft trade-off studies." The intended use must be specific enough to dictate the level of fidelity (i.e. detail) that is needed in a model or simulation. If the aircraft studies include maneuverability in combat, then the simulation must have enough detail to identify how changes in key design parameters impact maneuverability and how those translate into advantages in different combat situations. Similarly, a user requirement cannot be as simple as, "We need to train soldiers in urban warfare." More specific training goals would dictate derived user requirements from this generalization. For example, a derived user requirement should specify the types of decisions the soldier would need to make during the training and what situational factors are deemed, by the experts, to impact those decisions. Derived user requirements should be identified through an iterative process between the user and the developer, after the system developer is chosen; simulation procurement organizations very seldom put this level of detail into a request for proposal. Representation requirements can only be developed for specific uses based on this detail. The failure to identify appropriate representation requirements before moving to the modeling phase is the single largest cause of simulation failure leading to excessive cost and schedule overruns.
|
|
2) Inappropriate modeling methods - After the pitfalls in the requirements phase have been avoided, it is essential that developers avoid major mistakes when choosing the appropriate modeling methods for the level of detail needed.
|
|
2a) Data availability - Quite often a modeling expert will craft a model for which there is no data or for which the cost to get the data is prohibitive. This usually happens when the modeling experts "make up" parameters which appear to make sense, but for which there are no measures in the real world. For example, a behavioral scientist may try to model a decision process based on the decision maker's perception about an opponent's level of risk aversion. The scientist could believe that perception accuracy can be measured on a scale of 0 to 10 (0 - perceives the opponent is a very low risk taker, 10 - perceives the opponent is a very high risk taker). The theory behind the model may be valid, but there is probably no realistic way to collect this type of data in the volume needed to support the model. Gathering it may be either impossible to prohibitively costly.
|
|
2b) Oversimplifying the model - Ensuring that all the key parameters are included in a model (i.e. those that are critical for the level of fidelity needed) sometimes makes finding a modeling method difficult, if not impossible. In one solution the modeler oversimplifies the model, but ignoring some of the key parameters can make a model or simulation invalid and in some cases "dangerous" because it can produce erroneous results.
|
|
2c) - Computational Speed - Another common mistake is selecting a modeling technique that is so computationally intense that a simulation runs too slowly to be useful. One example the author encountered was modeling Early Warning Radars in a theater level combat simulation. The equations needed to calculate when a radar can see an aircraft are complex; they must account for the radar power, beam width, level of radar jamming, weather conditions, number of aircraft in the radar beam, etc. In a theater level simulation there are hundreds of aircraft and hundreds of Early Warning Radars, so the simulation must check for many radars looking for many aircraft. Since aircraft speeds make the positions of the aircraft change rapidly, the equations needed to be reevaluated every 10-15 seconds of real time. In the early 1980s, computer speeds made this modeling approach impractical. Even with today's processing speeds, this could still be a major factor in selecting the modeling method.
|
|
2d) Robustness - Some modeling methods are not robust; they are only valid for a narrow range of some key parameters; ranges that are so restrictive that the simulation becomes useless in many situations. For example, a dispersion model of a chemical agent spreading across a region may be valid for low to modest wind conditions, but not be valid when winds are above 20 miles an hour. This model would not be useful for emergency personnel in a real situation predicting where evacuations should occur unless, of course, they could be sure that there would never be a chemical spill or attack when the winds where above 20 miles per hour.
|
|
3) Not having the right people at the right time working on the development - A big financial mistake that can be made is bringing too many software engineers onto a program too early, before the modeling decisions have been made and the functioning prototype models of the critical processes and entities have been developed and tested. There is much for the software engineers to do, but they must first have a road map, clear specifications on what is needed, and a framework, or architecture, for the object oriented analysis to be useful. If the majority of the software engineers are added to a simulation development effort during or just after the user requirements are gathered, they would not have enough constructive work to do because the modelers would still be making essential modeling decisions. What they do will either be of little or no value; or it might actually be counter productive because it starts part of the development process too early and it is hard to turn it around if it is off track. Without the models that must be coded and integrated with the other simulation system modules, the software engineers don't have the key ingredient for their object oriented analysis and design. The software engineers don't know the data structure or how the models interact, making anything they produce (if they actually produce something) almost useless.
|
|
4) Failing to build in flexibility - Nothing ever stays the same. The first day a simulation is used in a production or operational mode, the users will start to identify how it needs to be changed. Standard software engineering practices have improved over the years and this is less of a problem than it was 10-20 years ago. However, the decisions on how to model the processes and entities are essential to the flexibility and extensibility of a simulation. If the simulation architecture is not built to allow developers to easily substitute models with different levels of fidelity, the simulation will become "static" and too expensive to modify.
|
|
In summary, this Fact Sheet touches on the most common problems encountered in simulation development programs. The more naïve the management team, the more likely major mistakes will be made. Developing a simulation is not just another software project; system engineers must do more than just gather user needs, develop user requirements, and hand them over to the software team. The time and energy it takes to get the right representation requirements and develop the right models is often considered "too much to do". The program management team may believe that they can pass on these steps and still succeed; hundreds of millions of dollars have been wasted with this approach.
|
| |
| Download as Word Document |
| Download as PDF |
|
|
.gif){kind=spacer}
