# Can the performance of a deterministic model be evaluated without an estimate of model uncertainty?

note: This question was inspired by Assessing the accuracy of a deterministic mathematical model. I have tried to give a more explicit example and specific question.

A meteorological model that predicts the weather is deterministic, so for any set of inputs it will give the same output. Commonly, a weather forecast will use today's observed meteorological conditions to predict tomorrow's high temperature. We will call the current conditions $x$, the model $g$, and the estimate of tomorrow's high temperature $\hat{T}$:

$\text{T}_\text{model}=g(x)$

Tomorrow, I observe that the high temperature was $\text{T}_\text{obs}$, with an uncertainty due to an observation error of $\epsilon_\text{obs}\sim \text{N}(0,\sigma)$.

There is no estimate of model uncertainty - theoretically, it could be as low as $\pm0.0001$ or as high as $\pm\infty$, but given the way that the model is used with fixed inputs, the model can only make a discrete estimate of a continuous variable.

Is it possible to say that the forecast is correct? Specifically, can I test the hypothesis that $\text{T}_\text{model}=\textrm{T}_\text{obs}$?

Perhaps $\text{T}_\text{model}$ falls inside the 95%CI for $\text{T}_\text{obs}$, but since the 95%CI for $\text{T}_\text{model}$ could be >> the 95%CI for $\text{T}_\text{obs}$, then it isn't clear that the hypothesis can be tested. So, can model performance be evaluated without an estimate of model uncertainty, or is an estimate of model uncertainty required?

Model accuracy can be defined as the difference between the model prediction and truth expressed in terms of squared error. So model accuracy is $E([T_{model}-T]^2)$ However you don't know the true $T$. But you say the you have $T_{obs}$ and you know its error distribution. So based on your assumption $E([T_{obs}-T]^2)=\sigma^2$.

Now $E([T_{model}-T_{obs}]^2)$ is unknown but can be estimated from the average squared difference between the model prediction and the observed value. What you are interested in is $E([T_{model}-T]^2)$. Add and subtract $T_{obs}$ inside the brackets and expand.

After a few algebra steps you get $$E([T_{model}-T]^2)= E([T_{model}-T_{obs}]^2)+ E([T_{obs}-T]^2 + E([T_{model}-T_{obs}] [T_{obs}-T]).$$

Note that the term $E([T_{obs}-T]^2) = \sigma^2$ and, since $T_{model}$ is independent of $T_{obs}$, $$E([T_{model}-T_{obs}] [T_{obs}-T])= E(T_{model} -T_{obs}) E(T_{obs}-T)$$ and by assuming the error in $T_{obs}$ is $\rm N(0,\sigma^2)$, $E(T_{obs}-T)=0.$

So we have the variance decomposition $E([T_{model}-T]^2)= E([T_{model}-T_{obs}]^2)+\sigma^2$. So we can estimate the model error by taking the estimate for $E([T_{model}-T_{obs}]^2)$ and adding it to the known $\sigma^2$ .

However if you want to assess the uncertainty in the estimate of $E([T_{model}-T_{obs}]^2)$ you still need to get a sampling distribution for it under the null hypothesis which amounts to still doing what I recommended in my answer to Steven's previous question.

• Can the person who downvoted this answer explain why? i explained how you can get an estimate of the model accuracy based on the assumptions and the data. That answered the title question. As for test the hypothesis that the model is correct, that doesn't make much sense. Certainly you can inspect the output to see if it give exactly the same value as the observed value without testing anything. In all likelihood there will be some level of discrepancy. – Michael R. Chernick Jun 19 '12 at 22:59
• Perhaps it was downvoted because of formatting? I $\TeX$ed the math in an attempt to improve the readability! – MånsT Jun 20 '12 at 11:53
• (it wasn't me, but the downvote appears to have been removed) – David LeBauer Jun 20 '12 at 14:07
• +1 This answer also sheds some light on the question at stats.stackexchange.com/questions/30643/…, too (which can be construed as asking the same question about model accuracy in a specific situation, that of predicting the value of a random field at an unsampled location). – whuber Jun 20 '12 at 14:13

Given that there is $N(0, \sigma)$ error in your observation, then the likelihood of the observation $T_{obs}$ given measurements $x$ is $L(T_{obs}|x) = N(T_{obs}; g(x), \sigma)$. One would need multiple measurements and temperature observations to have an estimateof $\sigma$, e.g. maximum likelihood. This is the gist of regression as noted in another answer. An alternative to the parametric form for the error surrounding model estimates is called semi-parametric regression. For example, one could fit the model to measurements and then bootstrap the residuals. Another, more sophisticated approach involves Gaussian processes. Generally, semi-parametric regression is useful when assumptions such as homoscedastic errors are unrealistic. For example, the model $g(x)$ might be more consistent in predicting small temperatures and noisier in estimating large values.

The state of the art in Meteorological forecasting is Ensemble Forecasting. This has only become possible in the last few years because of advances in computing power and the corresponding reduction of the cost of computing.

Ensemble forecasting tries to address the problem of how to get realistic probabilities from deterministic models. The basic idea is that the state a model is initialised with (all the pressures, temperatures, densities etc at every grid cell) is not known with certainty. We might know the state very well at locations where we can measure it, but the whole vertical profile in principle needs to be known everywhere. Modern deterministic models can actually do very well if all of this is known, but it is impossible to do in practice as we only have measurements in certain places at certain times.

With this in mind, the initial conditions are randomly perturbed based on the best understanding of the probable distribution of initial conditions, based on available measurements. For each randomly perturbed set of initial conditions the deterministic model is run to observe the likely range of final states given the uncertainty in the initial states.

In practice there is a fine art to this because the models aren't perfect, so if the above process is followed the final distribution is too restrictive compared with reality and extra model uncertainty is injected using a variety of different approaches. In general it takes a fair bit of tweaking to accurately calibrate an ensemble forecasting system with respect to historical data. This is a whole field of study in itself which encompasses physics, numerical methods and statistics.

• This is a very helpful answer, but it doesn't help with my current situation: I don't have an ensemble forecast! Is this like MCMC? – Abe Jun 20 '12 at 14:06
• I don't think comparing ensemble forecasting to MCMC is an apt analogy. Basically if you have access to the model (i.e. you are able to run it) then you can do some rudimentary ensemble forecasting by perturbing the initial conditions based on the measurement errors of the initial state. If you don't have access to the model, for instance if you want to compare someone else's published model predictions, then Michael Chernick's answer explains what you need to do. – Bogdanovist Jun 21 '12 at 1:39

Typically, statistical models (i.e., models of data) have a random component (also sometimes called a 'stochastic component'). For example, a model might be:
$$Y=X\beta+\epsilon \\ \text{where }\epsilon\sim\mathcal{N}(0,\sigma^2)$$
This example is a basic regression model. The $X\beta$ is called the structural component, and the $\epsilon$ is the random component. However, models are often written and discussed in terms of the predicted value or the expected value. The same model could be put:
$$\hat{Y}=X\beta \\ \text{or} \\ E(Y)=X\beta$$
The random component still exists, but is implicit.

You have described a deterministic model. (Note that I don't know much about meteorology or weather forecasting, so I can't say anything about how normal or appropriate that might be in the field.) At any rate, the model makes a simple prediction--it should be fairly simple to assess: the prediction either matches the observation, or it doesn't.

One oddity is that there seems to be a separate model of the intrinsic measurement error of the observations. I would think that the measurement of temperature has advanced to the point where this is inconsequential, but you could certainly assess the performance of the model over repeated observations, and see if the predictions fall within a $(1-\alpha)\% CI$, $(1-\alpha)\%$ of the time. My first guess is that the error in weather prediction will swamp the measurement error in the observation of temperature, and so I would have expected that people would not spend time on a deterministic model, but would include a random component directly into the primary model, which would mean they could be evaluated just like any normal statistical model would.

• Plenty of models are deterministic - also known as "process models" or "simulation models" (though these can be stochastic): many models of physical and chemical processes do not account for model uncertainty. I used weather as an example, and for the purposes of the example, I imagined using a standard outdoor thermometer. Of course more accurate measurements can be made, but the error will still exist, unlike in a deterministic model. – Abe Jun 19 '12 at 22:54
• Abe, I'll take your word for it; it's just not what I'm used to. – gung - Reinstate Monica Jun 19 '12 at 23:00
• @gung Any model that using classical laws of physics is deterministic. You seem to be very quick to dismiss an entire, somewhat important(!), field of study as 'weird'... – Bogdanovist Jun 20 '12 at 0:27
• @Bogdanovist, I certainly do not dismiss classical physics. Yes, it's true that many classical models of physical phenomena are deterministic, starting with levers and pulleys and such. However, those models are not statistical models or data models. I state in the first sentence (the 2nd word, actually) that that is what I'm talking about. I also gave examples for clarity, & acknowledged that I don't know about meteorology & that deterministic models may be normal there. However, I will delete the part about weirdness, since that seems to be causing trouble. – gung - Reinstate Monica Jun 20 '12 at 2:02
• It seems logical to (to me) to have separate models for measurement error and process error - why is this odd? – Abe Jun 20 '12 at 14:13