Paired t-test with à priori known variance for Model validation

We have a multiple linear regression model $$y = f(x1, x2)$$, which has been fitted with measurements of a designed experiment of the input variables $$x1$$ and $$x2$$. Based on the design, the complexity of the model function (in our case a 2nd order polynomial) and the inherent measurement uncertainty, we can build a confidence interval on the model predictions. (see Understanding the shape of confidence interval for polynomial regression (MLR))

After our model has been fitted, our goal is to compare the predictions of the model with some supplementary experimental data (model verification).

In a given point $$x_1, x_2$$, we perform 4 measurements on 4 different days and compare them to the model predictions. Note that:

• the input variable $$x1$$ cannot directly be controlled with the measurement setup and
• the response variable cannot be measured directly, but is calculated with the measured value of $$x_1$$.

Due to stochastic variations, the 4 values of the $$x_1$$ variable differ from each other in the 4 measurements. This, on the one hand has an influence on the measured value and on the other hand on the predicted value of the model. As is depicted in the figure, the measured values and the predicted values are therefore linked (in the first measurement the value of $$x_1$$ was lower than in the second measurement, which leads to a lower value of the response variable in the first measurement as well as in the model prediction for the first measurement in comparison to the second measurement).

In order to assess if there is a significant difference between the model and the measurements we perform a paired t-test. In this case the problem is, however, that the t-test does not take into consideration the model uncertainty (calculated with the à priori known confidence interval of the MLR, depicted with error bars in the figure).

In the depicted example, the result of the paired t-test would suggest a significant difference, although the difference is clearly always included in the uncertainty of the model.

The figure suggests a positive bias of the model (systematic overestimation of the response variable, but the error is included in the confidence interval).

How could I solve this ?

Suggestion: Perform a supplementary two sample t-test with (i) the mean of the difference with the standard deviation of the difference and (ii) the value 0 with the standard deviation of the model.

You have to be more clear by what you mean with 'the model predictions will also differ' and with 'a certain uncertainty, which is known'. What is exactly the basis, the pre-existing knowledge, and what is the data/observation?

Eventually some sort of paired differences could be compared with some measure of extreme variations, but from your text it is unclear whether this should be a t-test (variance based on estimate from sample) a z-test (variance/uncertainty which is known) or something more exotic (more complex dependencies, e.g. uncertainty in the variance which is not linked to the variance of the observations) or complex (assuming correlation between the error terms of the measurements).

Also, you should maybe describe the underlying problem (this could be an example of the xy communication problem, you ask for x but you want to solve y).

I imagine that you have some sort of measurement or settings of parameters that define/specify the function $$y=f(x)$$ and you want to test whether the observation of $$y$$ correspond to it.

The way that I would tackle this is in the opposite direction and use some regression to predict those parameters based on the observations (which can be expressed with a confidence region) and see whether your set parameters correspond with it.

The problem with your approach is that you need to be careful about potential correlation between the error terms. You probably do not get independent error terms. So errors that are all in the same direction might be more likely than errors in different directions.

And you need to be careful with the interpretation of a discrepancy. If you observe a significant difference, with large errors in different directions, then you could ask yourselve whether your estimate of the noise level is correct or not, or whether your deterministic model f(x) is wrong.

• Thanks for the input. I have detailed my question, hope this helps. Let me know if I need to add anything else. Jul 7, 2020 at 13:31
• I have to re-read your new post several times. I see many good additions but I am still a bit puzzled about it. In the end, this will come down to describing accurately the model how data is supposed to be generated. After that the mathematics is straightforward. The problem here is that step before the mathematics (although I imagine that this is also not a typical 'simple' case and has not necessarily a solution with simple mathemetics, but it might be still solved 'easily' with a computational method. Anyway the problem is uilding a reasonable/correct model.) Jul 7, 2020 at 14:40
• What do you understand by "describing accurately the model how data is supposed to be generated" ? Jul 7, 2020 at 15:08
• Accurately enough such that I can model it. Or otherwise I should be able to approximate it, but at least understand discrepancy between model and your data. Jul 7, 2020 at 17:35
• The differences between the model and the data can arise through two different reasons: uncertainty of the measurements during the verification process (which is tricky because it affects both the measured value (blue points) and the model prediction (red point)) or a mispecification of the regression model (for example the model can be biased and thus is not able to accuractely represent the physical quantity). Jul 7, 2020 at 20:31