8
$\begingroup$

Regression and machine learning are used in the natural sciences to test hypotheses, estimate parameters, and make predictions by fitting models to data. However, when I have an a priori model, I don't want to do any fitting---for example, a model of a deterministic physical system calculated from first principles. I simply want to know how well my model matches the data, and then understand which parts of the model contribute significantly to the match. Could someone point me towards a statistically rigorous way of doing this?

In more specific terms, suppose I have a physical system for which I measured a dependent variable $y_i$ ($i$ ranges from 1 to $n$, the sample size) under varying conditions described by three independent variables $x_{1,i}$, $x_{2,i}$, and $x_{3,i}$. Although the real system that generated the data is complicated, I made some simplifying assumptions to derive a theoretical model $f$ for the system, such that

$y_i = f(x_{1,i}, x_{2,i}, x_{3,i}) + \epsilon_i$,

where $f$ is a non-linear (and not linearizable) function of the independent variables and $\epsilon_i$ is the difference between the model-predicted and measured values. $f$ is completely pre-specified; no fitting is done and no parameters are estimated. My first goal is to determine if $f$ is a reasonable model for the process that produced the measured values $y_i$.

I also developed simplified models $g(x_{1,i}, x_{2,i})$ and $h(x_{1,i})$, which are nested in $f$ (if that matters in this case). My second goal is to determine if $f$ matches the data significantly better than $g$ or $h$, suggesting that the features that differentiate model $f$ from models $g$ and $h$ play an important role in the process that generates $y_i$.

Ideas so far

Perhaps if there were some way to determine the number of parameters or number of degrees of freedom for my mathematical model, it would be possible to use existing procedures like a likelihood ratio test or AIC comparison. However, given the nonlinear form of $f$ and the absence of any obvious parameters, I'm not sure if it's reasonable to assign parameters or to assume what constitutes a degree of freedom.

I've read that measures of goodness-of-fit, such as the coefficient of determination ($R^2$), can be used to compare model performance. However, it's not clear to me what the threshold for a meaningful difference between $R^2$ values might be. Further, because I don't fit the model to the data, the mean of the residuals is not zero and may be different for each model. Thus, a well-matching model that tends to underpredict the data might yield as poor a value of $R^2$ as a model that was unbiased but poorly matched to the data.

I've also read a bit about goodness-of-fit tests (e.g., Anderson-Darling), but as statistics is not my field, I'm not sure how well this type of test suits my purpose. Any guidance would be appreciated.

$\endgroup$
  • $\begingroup$ Are there any parameter values for the nonlinear function f() that need to be determined from a fit to the data, or is the function f() completely pre-specified? $\endgroup$ – EdM Feb 13 at 21:10
  • $\begingroup$ @EdM Thanks for that! Edited the question to clarify that f is completely pre-specified. It's like a black box that produces the response y from the input variables, and I want to know how well it's doing compared to competing black boxes. An analagous situation might be trying to evaluate the match between the output of a numerical simulation and measurements made in the real physical system. $\endgroup$ – jbacks Feb 13 at 22:40
4
$\begingroup$

In this situation you are essentially comparing the distributions of the $\epsilon_i$ among the 3 models. So you need to examine issues like:

  1. Are the mean values of the $\epsilon_i$ different among the 3 models, and is any of these mean values different from 0? (That is, is there a bias in any of the models and do the 3 models differ in bias?)
  2. Is there any systematic relation of the $\epsilon_i$ to the values predicted from the corresponding model, or to the values of the independent variables $x_{1,i},x_{2,i}, x_{3,1}$? You should consider all three independent variables here even if the particular model only used 1 or 2 of them.
  3. Are there significant differences in the variances of the $\epsilon_i$ among the 3 models?

The details of how best to approach these questions will depend on the nature of your data. For example, if values of $y_i$ are necessarily positive and have typical measurement errors proportional to their values (as often is the case in practice), it might make sense to do this analysis on differences between log-transformed $y_i$ and log-transformed predictions from each of your models.

Visual analysis of the distributions of the $\epsilon_i$ among the 3 models, for example with density plots, would be an important first step.

Depending on the nature of the data, standard parametric or non-parametric statistical tests for differences in mean values, applied to the $\epsilon_i$ for the 3 models, would address Issue 1.

Issue 2 is essentially what is done to examine the quality of any fitted model; in your case this analysis might show domains of the independent variables over which one or more of your pre-specified models does not work well. Plots of $\epsilon_i$ versus predicted values and independent-variable values, with loess curves to highlight trends, for each of your models would be useful.

If there is no bias in any models and analysis of Issue 2 shows no problems, then the remaining Issue 3 is whether any of the models is superior in terms of precision/variance. In the ideal case with normally distributed $\epsilon_i$ within each model, F-tests could test for equality of variances.

$\endgroup$
  • $\begingroup$ Thinking of the residual distribution as the object of comparison is a useful shift in perspective! a) Would you know of any published analyses that use a similar method? I feel like my situation is unusual. Any published precedent would be helpful. b) The mean of each residual distribution is non-zero and visibly different for two of my models, and I expect ANOVA would confirm this. Knowing this, would it still be sensible to examine the differences among the variance of each residual distribution (Issue 3)? Could patterns exposed via Issue 2 invalidate a comparison of variances? $\endgroup$ – jbacks Feb 14 at 20:26
  • 1
    $\begingroup$ @jbacks I don't know of a published precedent but I don't think this approach would be a hard sell if there is a solid theoretical basis for your model(s). In this theory-based analysis, focus on the reasons for the systematic bias (non-zero mean error, Issue I) between predictions and observations. That would seem to get most directly at the relative worth of the models. Issue II (any patterns of error magnitude/direction related to independent variable values or predicted values) should illustrate where your models are going astray. Comparisons of model variances are of less interest. $\endgroup$ – EdM Feb 14 at 20:45
  • 1
    $\begingroup$ @jbacks also do consider working with observations/predictions in a transformed scale such as logarithmic. A bias in error terms in a non-transformed scale might be reduced or removed following transformation. Note that the use of percent errors, suggested in another answer, is equivalent to looking at differences between log-transformed predictions and observations. You'll have to judge whether that would be appropriate for this situation. $\endgroup$ – EdM Feb 14 at 20:52
  • $\begingroup$ This seems plausible, and I'm going to give it a shot. Thank you again for your insight. $\endgroup$ – jbacks Feb 15 at 18:51
1
$\begingroup$

A probabilistic comparison of the models, e.g. involving some likelihood computed from the $\epsilon$ with some data (and derived from this AIC or ratio test), makes little sense.

This is because

  1. You already know for certain that the model is gonna be wrong.
  2. The residuals that you end up with have no relation with the hypothesised distribution of errors that you use to test different hypotheses. (you do not have a statistical/probabilisitc model)
  3. Your goal is not to test a hypothesis (basic/pure science), but to characterize the prediction performance of a simplified model (applied science).

Most often people describe models in terms of the percent of error for predictions.

Examples:

Basically you can google any model that is a simplification of reality and you will find people describing their discrepancy with reality in terms of correlation coefficients, or percent of variation.


I want to test the hypothesis that "phenomenon A" involving x_3,i contributes measurably to the production of y. Model f incorporates phenomenon A while g and h do not, so if my hypothesis were true, I would predict that model f performs significantly better than either g or h.

For such comparison you could consider the measured performance as a sample, a sample taken out of a larger (hypothetical) population of performance.

So you sort of wish to describe the parameters of the population distribution of the errors $\epsilon$ and compare those. This you might consider as probabilistic. For instance, you could phrase it as 'the average error of the model is $y \pm x$'. Your hypothesis is about those parameters that describe the distribution of the errors.

However this view is a bit problematic, since often the "sample" that is used to measure performance, is not really a random selection (e.g. it are measurements along a predifined range or among a selected practical set of items). Then any quantification of the error in the estimate of general peformance should not be based on a model for random selection (e.g. using variance in the sample to describe te error of the estimate). So it still makes little sense to use a probabilistic model to describe the comparisons. It might be sufficient to just state descriptive data, and make your "estimate" about generalization based on logical arguments.

$\endgroup$
  • $\begingroup$ These examples are helpful! I'm a little confused though by your assertion that my goal does not involve a hypothesis test. As I frame it, I want to test the hypothesis that "phenomenon A" involving x_3,i contributes measurably to the production of y. Model f incorporates phenomenon A while g and h do not, so if my hypothesis were true, I would predict that model f performs significantly better than either g or h. $\endgroup$ – jbacks Feb 14 at 21:49
  • 2
    $\begingroup$ @jbacks for such comparison you could consider the measured performance as a sample taken out of a large population of performance. So you sort of wish to described the parameters of the population distribution of the errors $\epsilon $ and compare those. This you might consider as probabilistic. For instance, you could phrase it as 'the average error of the model is $x \pm y $'. Your hypothesis is about those parameters. $\endgroup$ – Sextus Empiricus Feb 14 at 22:30
  • $\begingroup$ Thank you for expanding on that comment with your edit. Between this perspective and the other answer, I think I have a plausible way forward. Much appreciated! $\endgroup$ – jbacks Feb 15 at 18:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.