A probabilistic comparison of the models, e.g. involving some likelihood computed from the $\epsilon$ with some data (and derived from this AIC or ratio test), makes little sense.
This is because
- You already know for certain that the model is gonna be wrong.
- The residuals that you end up with have no relation with the hypothesised distribution of errors that you use to test different hypotheses. (you do not have a statistical/probabilisitc model)
- Your goal is not to test a hypothesis (basic/pure science), but to characterize the prediction performance of a simplified model (applied science).
Most often people describe models in terms of the percent of error for predictions.
Examples:
It is shown that these correlations can be used to predict pressure drop to within ±20% for a given sludge concentration and operating condition.
The present model suits with the 501 viscosity values with mean deviations lower than 5% and 75% of them are within the correlation coefficient 0.78–1.
Figure 2 presents a comparison between measured viscosity ($\rho$) and the viscosity calculated by Einstein model. A difference between calculated and measured values confirms that there is an elevated physical interaction between asphalt base and rubber particles.
A correlation coefficient (r2) of 0.94 was determined for the relationship between known LWAPCs (log water‐to‐air partition coefficients) and bond estimated LWAPCs for the 345 compound data set.
Basically you can google any model that is a simplification of reality and you will find people describing their discrepancy with reality in terms of correlation coefficients, or percent of variation.
I want to test the hypothesis that "phenomenon A" involving
x_3,i
contributes measurably to the production ofy
. Modelf
incorporates phenomenon A whileg
andh
do not, so if my hypothesis were true, I would predict that modelf
performs significantly better than eitherg
orh
.
For such comparison you could consider the measured performance as a sample, a sample taken out of a larger (hypothetical) population of performance.
So you sort of wish to describe the parameters of the population distribution of the errors $\epsilon$ and compare those. This you might consider as probabilistic. For instance, you could phrase it as 'the average error of the model is $y \pm x$'. Your hypothesis is about those parameters that describe the distribution of the errors.
However this view is a bit problematic, since often the "sample" that is used to measure performance, is not really a random selection (e.g. it are measurements along a predifined range or among a selected practical set of items). Then any quantification of the error in the estimate of general peformance should not be based on a model for random selection (e.g. using variance in the sample to describe te error of the estimate). So it still makes little sense to use a probabilistic model to describe the comparisons. It might be sufficient to just state descriptive data, and make your "estimate" about generalization based on logical arguments.