In this paper, we investigate the importance of different loss functions when estimating and evaluating option pricing models. Our analysis shows that it is important to take into account parameter uncertainty, because this leads to uncertainty in the predicted option price. We illustrate the effect on the out-of-sample pricing errors in an application of the ad hoc Black-Scholes model to DAX index options. We confirm the empirical results of Christoffersen and Jacobs (Christoffersen, P., K. Jacobs. 2004. The importance of the loss function in option valuation. J. Financial Econom. 72 291-318) and find strong evidence for their conjecture that the squared pricing error criterion may serve as a general-purpose loss function in option valuation applications. At the same time, we provide a first yardstick to evaluate the adequacy of the loss function. This is accomplished through a data-driven method to deliver not just a point estimate of the root mean squared pricing error, but a distribution.