Why one run can mislead
One simulation run is one possible day. It may be a good day or a bad one, and it won't tell you which. Before you put a number in front of a client, check how much it moves.
A real example
Take a small triage desk: one nurse, patients arriving every 6 minutes on average, and triage taking 5 minutes on average. Both times are Exponential. We ran an 8-hour day in Quodsi at the default seed.
| What we ran | Average time in system | 95% confidence interval |
|---|---|---|
| 1 replication | 14.2 minutes | None: one run gives no range |
| 10 replications | 23.8 minutes | 16.1 to 31.5 minutes |
Across those 10 replications, the average for a single day ran from 14.2 minutes to 47.9 minutes. All of them are the same process with the same nurse and the same average arrival rate. Only the luck of the day differs.
The single run happened to be the best day of the ten. Had you stopped there, you'd have told your client that patients are through in about 14 minutes, and the model would have backed you up.
Ten replications aren't magic either
We ran the same 10 replications with a different seed. The average came out at 16.6 minutes, with a range of 12.5 to 20.8. That's a different set of possible days, and it's lower than the first result. The two ranges still overlap, which is what you'd hope for. The lesson is to read the range, not just the mean.
Utilization shows the same thing more quietly. Across the first 10 replications, the mean was 81%, with single days from 72% to 92%.
How to work with this
- Keep the default of 10 replications or more. Don't drop to 1 to save time on anything you plan to report.
- Read the confidence interval. If the range is too wide for your decision, add replications. The limit is 100 per run.
- Compare ranges, not means. If option A averages 20 minutes and option B averages 22, but their ranges overlap heavily, you can't yet say A is better.
- Try a second seed before a big call. A conclusion you can defend holds up under a different set of random draws.
- Report the range. "Between 16 and 32 minutes, most likely around 24" is more honest than "24 minutes", and it holds up better when someone pushes on it.