Paired t-test with à priori known variance for Model validation

Question

We have a multiple linear regression model $y = f(x1, x2)$, which has been fitted with measurements of a designed experiment of the input variables $x1$ and $x2$. Based on the design, the complexity of the model function (in our case a 2nd order polynomial) and the inherent measurement uncertainty, we can build a confidence interval on the model predictions. (see Understanding the shape of confidence interval for polynomial regression (MLR))

After our model has been fitted, our goal is to compare the predictions of the model with some supplementary experimental data (model verification).

In a given point $x_1, x_2$, we perform 4 measurements on 4 different days and compare them to the model predictions. Note that:

• the input variable $x1$ cannot directly be controlled with the measurement setup and
• the response variable cannot be measured directly, but is calculated with the measured value of $x_1$.

Due to stochastic variations, the 4 values of the $x_1$ variable differ from each other in the 4 measurements. This, on the one hand has an influence on the measured value and on the other hand on the predicted value of the model. As is depicted in the figure, the measured values and the predicted values are therefore linked (in the first measurement the value of $x_1$ was lower than in the second measurement, which leads to a lower value of the response variable in the first measurement as well as in the model prediction for the first measurement in comparison to the second measurement).

In order to assess if there is a significant difference between the model and the measurements we perform a paired t-test. In this case the problem is, however, that the t-test does not take into consideration the model uncertainty (calculated with the à priori known confidence interval of the MLR, depicted with error bars in the figure).

In the depicted example, the result of the paired t-test would suggest a significant difference, although the difference is clearly always included in the uncertainty of the model.

The figure suggests a positive bias of the model (systematic overestimation of the response variable, but the error is included in the confidence interval).

How could I solve this ?

Suggestion: Perform a supplementary two sample t-test with (i) the mean of the difference with the standard deviation of the difference and (ii) the value 0 with the standard deviation of the model.

Answer 1

You have to be more clear by what you mean with 'the model predictions will also differ' and with 'a certain uncertainty, which is known'. What is exactly the basis, the pre-existing knowledge, and what is the data/observation?

Eventually some sort of paired differences could be compared with some measure of extreme variations, but from your text it is unclear whether this should be a t-test (variance based on estimate from sample) a z-test (variance/uncertainty which is known) or something more exotic (more complex dependencies, e.g. uncertainty in the variance which is not linked to the variance of the observations) or complex (assuming correlation between the error terms of the measurements).

---

Also, you should maybe describe the underlying problem (this could be an example of the xy communication problem, you ask for x but you want to solve y).

I imagine that you have some sort of measurement or settings of parameters that define/specify the function $y=f(x)$ and you want to test whether the observation of $y$ correspond to it.

The way that I would tackle this is in the opposite direction and use some regression to predict those parameters based on the observations (which can be expressed with a confidence region) and see whether your set parameters correspond with it.

The problem with your approach is that you need to be careful about potential correlation between the error terms. You probably do not get independent error terms. So errors that are all in the same direction might be more likely than errors in different directions.

And you need to be careful with the interpretation of a discrepancy. If you observe a significant difference, with large errors in different directions, then you could ask yourselve whether your estimate of the noise level is correct or not, or whether your deterministic model f(x) is wrong.