Selecting the optimal policy using simulation is subject to input model risk when input models that mimic real-world randomness in the simulation have estimation error due to finite sample sizes. Instead of trying to find the optimal solution under unknown real-world input distributions by taking a conservative stance or with low statistical guarantee, the input–output uncertainty comparisons (IOU-C) procedure finds a set of solutions that cannot be separated from the best given the resolution decided by the finite sample sizes. The common-input-data (CID) effects measure how differently solutions are affected by the common estimated input models. When CID effects of two systems are positively correlated, the comparison becomes easier than estimating the performance measures of two systems precisely under input model risk; the IOU-C procedure takes advantage of the CID effects to develop a sharp comparison and thereby provides a small subset even in the presence of input model risk.
When input distributions to a simulation model are estimated from real-world data, they naturally have estimation error causing
input uncertainty
in the simulation output. If an optimization via simulation (OvS) method is applied that treats the input distributions as “correct,” then there is a risk of making a suboptimal decision for the real world, which we call
input model risk
. This paper addresses a discrete OvS (DOvS) problem of selecting the real-world optimal from among a finite number of systems when all of them share the same input distributions estimated from common input data. Because input uncertainty cannot be reduced without collecting additional real-world data—which may be expensive or impossible—a DOvS procedure should reflect the limited resolution provided by the simulation model in distinguishing the real-world optimal solution from the others. In light of this, our input–output uncertainty comparisons (IOU-C) procedure focuses on
comparisons rather than selection
: it provides simultaneous confidence intervals for the difference between each system’s real-world mean and the best mean of the rest with any desired probability, while accounting for both stochastic and input uncertainty. To make the resolution as high as possible (intervals as short as possible) we exploit the common input data effect to reduce uncertainty in the estimated differences. Under mild conditions we prove that the IOU-C procedure provides the desired statistical guarantee asymptotically as the real-world sample size and simulation effort increase, but it is designed to be effective in finite samples.
The electronic companion of this paper is available at
https://doi.org/10.1287/opre.2018.1796
....

Input–Output Uncertainty Comparisons for Discrete Optimization via Simulation

10.1287/opre.2018.1796