We have a multiple linear regression model $y = f(x1, x2)$, which has been fitted with measurements of a designed experiment of the input variables $x1$ and $x2$. Based on the design, the complexity of the model function (in our case a 2nd order polynomial) and the inherent measurement uncertainty, we can build a confidence interval on the model predictions. (see Understanding the shape of confidence interval for polynomial regression (MLR))
After our model has been fitted, our goal is to compare the predictions of the model with some supplementary experimental data (model verification).
In a given point $x_1, x_2$, we perform 4 measurements on 4 different days and compare them to the model predictions. Note that:
- the input variable $x1$ cannot directly be controlled with the measurement setup and
- the response variable cannot be measured directly, but is calculated with the measured value of $x_1$.
Due to stochastic variations, the 4 values of the $x_1$ variable differ from each other in the 4 measurements. This, on the one hand has an influence on the measured value and on the other hand on the predicted value of the model. As is depicted in the figure, the measured values and the predicted values are therefore linked (in the first measurement the value of $x_1$ was lower than in the second measurement, which leads to a lower value of the response variable in the first measurement as well as in the model prediction for the first measurement in comparison to the second measurement).
In order to assess if there is a significant difference between the model and the measurements we perform a paired t-test. In this case the problem is, however, that the t-test does not take into consideration the model uncertainty (calculated with the à priori known confidence interval of the MLR, depicted with error bars in the figure).
In the depicted example, the result of the paired t-test would suggest a significant difference, although the difference is clearly always included in the uncertainty of the model.
The figure suggests a positive bias of the model (systematic overestimation of the response variable, but the error is included in the confidence interval).
How could I solve this ?
Suggestion: Perform a supplementary two sample t-test with (i) the mean of the difference with the standard deviation of the difference and (ii) the value 0 with the standard deviation of the model.