Can the performance of a deterministic model be evaluated without an estimate of model uncertainty?

Question

note: This question was inspired by Assessing the accuracy of a deterministic mathematical model. I have tried to give a more explicit example and specific question.

---

A meteorological model that predicts the weather is deterministic, so for any set of inputs it will give the same output. Commonly, a weather forecast will use today's observed meteorological conditions to predict tomorrow's high temperature. We will call the current conditions $x$, the model $g$, and the estimate of tomorrow's high temperature $\hat{T}$:

$\text{T}_\text{model}=g(x)$

Tomorrow, I observe that the high temperature was $\text{T}_\text{obs}$, with an uncertainty due to an observation error of $\epsilon_\text{obs}\sim \text{N}(0,\sigma)$.

There is no estimate of model uncertainty - theoretically, it could be as low as $\pm0.0001$ or as high as $\pm\infty$, but given the way that the model is used with fixed inputs, the model can only make a discrete estimate of a continuous variable.

Is it possible to say that the forecast is correct? Specifically, can I test the hypothesis that $\text{T}_\text{model}=\textrm{T}_\text{obs}$?

Perhaps $\text{T}_\text{model}$ falls inside the 95%CI for $\text{T}_\text{obs}$, but since the 95%CI for $\text{T}_\text{model}$ could be >> the 95%CI for $\text{T}_\text{obs}$, then it isn't clear that the hypothesis can be tested. So, can model performance be evaluated without an estimate of model uncertainty, or is an estimate of model uncertainty required?

Answer 1

Model accuracy can be defined as the difference between the model prediction and truth expressed in terms of squared error. So model accuracy is $E([T_{model}-T]^2)$ However you don't know the true $T$. But you say the you have $T_{obs}$ and you know its error distribution. So based on your assumption $E([T_{obs}-T]^2)=\sigma^2$.

Now $E([T_{model}-T_{obs}]^2)$ is unknown but can be estimated from the average squared difference between the model prediction and the observed value. What you are interested in is $E([T_{model}-T]^2)$. Add and subtract $T_{obs}$ inside the brackets and expand.

After a few algebra steps you get $$E([T_{model}-T]^2)= E([T_{model}-T_{obs}]^2)+ E([T_{obs}-T]^2 + E([T_{model}-T_{obs}] [T_{obs}-T]).$$

Note that the term $E([T_{obs}-T]^2) = \sigma^2$ and, since $T_{model}$ is independent of $T_{obs}$, $$E([T_{model}-T_{obs}] [T_{obs}-T])= E(T_{model} -T_{obs}) E(T_{obs}-T)$$ and by assuming the error in $T_{obs}$ is $\rm N(0,\sigma^2)$, $E(T_{obs}-T)=0.$

So we have the variance decomposition $E([T_{model}-T]^2)= E([T_{model}-T_{obs}]^2)+\sigma^2$. So we can estimate the model error by taking the estimate for $E([T_{model}-T_{obs}]^2)$ and adding it to the known $\sigma^2$ .

However if you want to assess the uncertainty in the estimate of $E([T_{model}-T_{obs}]^2)$ you still need to get a sampling distribution for it under the null hypothesis which amounts to still doing what I recommended in my answer to Steven's previous question.

Answer 2

Given that there is $N(0, \sigma)$ error in your observation, then the likelihood of the observation $T_{obs}$ given measurements $x$ is $L(T_{obs}|x) = N(T_{obs}; g(x), \sigma)$. One would need multiple measurements and temperature observations to have an estimateof $\sigma$, e.g. maximum likelihood. This is the gist of regression as noted in another answer. An alternative to the parametric form for the error surrounding model estimates is called semi-parametric regression. For example, one could fit the model to measurements and then bootstrap the residuals. Another, more sophisticated approach involves Gaussian processes. Generally, semi-parametric regression is useful when assumptions such as homoscedastic errors are unrealistic. For example, the model $g(x)$ might be more consistent in predicting small temperatures and noisier in estimating large values.

Answer 3

The state of the art in Meteorological forecasting is Ensemble Forecasting. This has only become possible in the last few years because of advances in computing power and the corresponding reduction of the cost of computing.

Ensemble forecasting tries to address the problem of how to get realistic probabilities from deterministic models. The basic idea is that the state a model is initialised with (all the pressures, temperatures, densities etc at every grid cell) is not known with certainty. We might know the state very well at locations where we can measure it, but the whole vertical profile in principle needs to be known everywhere. Modern deterministic models can actually do very well if all of this is known, but it is impossible to do in practice as we only have measurements in certain places at certain times.

With this in mind, the initial conditions are randomly perturbed based on the best understanding of the probable distribution of initial conditions, based on available measurements. For each randomly perturbed set of initial conditions the deterministic model is run to observe the likely range of final states given the uncertainty in the initial states.

In practice there is a fine art to this because the models aren't perfect, so if the above process is followed the final distribution is too restrictive compared with reality and extra model uncertainty is injected using a variety of different approaches. In general it takes a fair bit of tweaking to accurately calibrate an ensemble forecasting system with respect to historical data. This is a whole field of study in itself which encompasses physics, numerical methods and statistics.

Answer 4

Typically, statistical models (i.e., models of data) have a random component (also sometimes called a 'stochastic component'). For example, a model might be:
$$ Y=X\beta+\epsilon \\ \text{where }\epsilon\sim\mathcal{N}(0,\sigma^2) $$
This example is a basic regression model. The $X\beta$ is called the structural component, and the $\epsilon$ is the random component. However, models are often written and discussed in terms of the predicted value or the expected value. The same model could be put:
$$ \hat{Y}=X\beta \\ \text{or} \\ E(Y)=X\beta $$
The random component still exists, but is implicit.

You have described a deterministic model. (Note that I don't know much about meteorology or weather forecasting, so I can't say anything about how normal or appropriate that might be in the field.) At any rate, the model makes a simple prediction--it should be fairly simple to assess: the prediction either matches the observation, or it doesn't.

One oddity is that there seems to be a separate model of the intrinsic measurement error of the observations. I would think that the measurement of temperature has advanced to the point where this is inconsequential, but you could certainly assess the performance of the model over repeated observations, and see if the predictions fall within a $(1-\alpha)\% CI$, $(1-\alpha)\%$ of the time. My first guess is that the error in weather prediction will swamp the measurement error in the observation of temperature, and so I would have expected that people would not spend time on a deterministic model, but would include a random component directly into the primary model, which would mean they could be evaluated just like any normal statistical model would.