How can I compare models without fitting?

Question

Regression and machine learning are used in the natural sciences to test hypotheses, estimate parameters, and make predictions by fitting models to data. However, when I have an a priori model, I don't want to do any fitting---for example, a model of a deterministic physical system calculated from first principles. I simply want to know how well my model matches the data, and then understand which parts of the model contribute significantly to the match. Could someone point me towards a statistically rigorous way of doing this?

In more specific terms, suppose I have a physical system for which I measured a dependent variable $y_i$ ($i$ ranges from 1 to $n$, the sample size) under varying conditions described by three independent variables $x_{1,i}$, $x_{2,i}$, and $x_{3,i}$. Although the real system that generated the data is complicated, I made some simplifying assumptions to derive a theoretical model $f$ for the system, such that

$y_i = f(x_{1,i}, x_{2,i}, x_{3,i}) + \epsilon_i$,

where $f$ is a non-linear (and not linearizable) function of the independent variables and $\epsilon_i$ is the difference between the model-predicted and measured values. $f$ is completely pre-specified; no fitting is done and no parameters are estimated. My first goal is to determine if $f$ is a reasonable model for the process that produced the measured values $y_i$.

I also developed simplified models $g(x_{1,i}, x_{2,i})$ and $h(x_{1,i})$, which are nested in $f$ (if that matters in this case). My second goal is to determine if $f$ matches the data significantly better than $g$ or $h$, suggesting that the features that differentiate model $f$ from models $g$ and $h$ play an important role in the process that generates $y_i$.

Ideas so far

Perhaps if there were some way to determine the number of parameters or number of degrees of freedom for my mathematical model, it would be possible to use existing procedures like a likelihood ratio test or AIC comparison. However, given the nonlinear form of $f$ and the absence of any obvious parameters, I'm not sure if it's reasonable to assign parameters or to assume what constitutes a degree of freedom.

I've read that measures of goodness-of-fit, such as the coefficient of determination ($R^2$), can be used to compare model performance. However, it's not clear to me what the threshold for a meaningful difference between $R^2$ values might be. Further, because I don't fit the model to the data, the mean of the residuals is not zero and may be different for each model. Thus, a well-matching model that tends to underpredict the data might yield as poor a value of $R^2$ as a model that was unbiased but poorly matched to the data.

I've also read a bit about goodness-of-fit tests (e.g., Anderson-Darling), but as statistics is not my field, I'm not sure how well this type of test suits my purpose. Any guidance would be appreciated.

Answer 1

In this situation you are essentially comparing the distributions of the $\epsilon_i$ among the 3 models. So you need to examine issues like:

1. Are the mean values of the $\epsilon_i$ different among the 3 models, and is any of these mean values different from 0? (That is, is there a bias in any of the models and do the 3 models differ in bias?)
2. Is there any systematic relation of the $\epsilon_i$ to the values predicted from the corresponding model, or to the values of the independent variables $x_{1,i},x_{2,i}, x_{3,1}$? You should consider all three independent variables here even if the particular model only used 1 or 2 of them.
3. Are there significant differences in the variances of the $\epsilon_i$ among the 3 models?

The details of how best to approach these questions will depend on the nature of your data. For example, if values of $y_i$ are necessarily positive and have typical measurement errors proportional to their values (as often is the case in practice), it might make sense to do this analysis on differences between log-transformed $y_i$ and log-transformed predictions from each of your models.

Visual analysis of the distributions of the $\epsilon_i$ among the 3 models, for example with density plots, would be an important first step.

Depending on the nature of the data, standard parametric or non-parametric statistical tests for differences in mean values, applied to the $\epsilon_i$ for the 3 models, would address Issue 1.

Issue 2 is essentially what is done to examine the quality of any fitted model; in your case this analysis might show domains of the independent variables over which one or more of your pre-specified models does not work well. Plots of $\epsilon_i$ versus predicted values and independent-variable values, with loess curves to highlight trends, for each of your models would be useful.

If there is no bias in any models and analysis of Issue 2 shows no problems, then the remaining Issue 3 is whether any of the models is superior in terms of precision/variance. In the ideal case with normally distributed $\epsilon_i$ within each model, F-tests could test for equality of variances.

Answer 2

A probabilistic comparison of the models, e.g. involving some likelihood computed from the $\epsilon$ with some data (and derived from this AIC or ratio test), makes little sense.

This is because

1. You already know for certain that the model is gonna be wrong.
2. The residuals that you end up with have no relation with the hypothesised distribution of errors that you use to test different hypotheses. (you do not have a statistical/probabilisitc model)
3. Your goal is not to test a hypothesis (basic/pure science), but to characterize the prediction performance of a simplified model (applied science).

---

Most often people describe models in terms of the percent of error for predictions.

Examples:

• Sludge pipe flow pressure drop prediction using composite power-law friction factor-Reynolds number correlations based on different non-Newtonian Reynolds numbers

  It is shown that these correlations can be used to predict pressure drop to within ±20% for a given sludge concentration and operating condition.

• Predicting the effective viscosity of nanofluids based on the rheology of suspensions of solid particles

  The present model suits with the 501 viscosity values with mean deviations lower than 5% and 75% of them are within the correlation coefficient 0.78–1.

• Application of artificial intelligence to modelling asphalt –rubber viscosity

  Figure 2 presents a comparison between measured viscosity ($\rho$) and the viscosity calculated by Einstein model. A difference between calculated and measured values confirms that there is an elevated physical interaction between asphalt base and rubber particles.

• Bond contribution method for estimating henry's law constants

  A correlation coefficient (r2) of 0.94 was determined for the relationship between known LWAPCs (log water‐to‐air partition coefficients) and bond estimated LWAPCs for the 345 compound data set.

Basically you can google any model that is a simplification of reality and you will find people describing their discrepancy with reality in terms of correlation coefficients, or percent of variation.

---

I want to test the hypothesis that "phenomenon A" involving x_3,i contributes measurably to the production of y. Model f incorporates phenomenon A while g and h do not, so if my hypothesis were true, I would predict that model f performs significantly better than either g or h.

For such comparison you could consider the measured performance as a sample, a sample taken out of a larger (hypothetical) population of performance.

So you sort of wish to describe the parameters of the population distribution of the errors $\epsilon$ and compare those. This you might consider as probabilistic. For instance, you could phrase it as 'the average error of the model is $y \pm x$'. Your hypothesis is about those parameters that describe the distribution of the errors.

However this view is a bit problematic, since often the "sample" that is used to measure performance, is not really a random selection (e.g. it are measurements along a predifined range or among a selected practical set of items). Then any quantification of the error in the estimate of general peformance should not be based on a model for random selection (e.g. using variance in the sample to describe te error of the estimate). So it still makes little sense to use a probabilistic model to describe the comparisons. It might be sufficient to just state descriptive data, and make your "estimate" about generalization based on logical arguments.