1
$\begingroup$

I'm using the chi-squared goodness-of-fit measure to evaluate instances of a fairly complicated model, but the evaluation is unbelievably sensitive to small variations in the model parameter values. My understanding of the definition and use of this goodness-of-fit measure, and my choice to use it, comes from the book Numerical Recipes, specifically the chapter on data modeling. I work in scientific computing (where this book is a standard reference), not statistics. I'm looking for guidance, as a novice in statistical model evaluation, in what goodness-of-fit measures would be more appropriate to my situation.

Here's a simplified example that demonstrates the same sensitivity as I'm seeing in my problem.

  • The model is of a "weighted" range on the number line. It has 3 parameters / degrees of freedom: a beginning value, an ending value, and a weight. $(a,b,w)$ is the parameter vector representing the range $[a,b]$ with weight $w$.
  • I make 10 observations of any given model instance. Each observation is the weight times the length of overlap between the model instance's range and a particular unit-length interval. My 10 observations are on the intervals $[0,1], [1,2], \ldots, [8,9], [9,10]$. I know a priori that the noise on my observations is Gaussian with unit variance.
  • The hidden ground-truth model instance is $(1,9,300)$. So my 10 observations are $$(0.192, 298.839, 301.493, \ldots, 300.273, 300.144, 0.243),$$ which is just $(0, 300, 300, \ldots , 300, 300, 0)$ plus noise.

I've got two model instances I want to talk about:

  1. The oracle instance $(1,9,300)$ predicts these 10 values: $$(0, 300, 300, \ldots , 300, 300, 0)$$
  2. The candidate instance $(1,8.9,300)$ predicts these 10 values: $$(0, 300, 300, \ldots, 300, 270, 0)$$

The chi-squared test statistic, as I'm using it, is: $$\sum_{i=1}^{10} \left(\frac{\mathrm{obs}_i - \mathrm{pred}_i}{\sigma_i} \right)^2$$ where $\sigma_i$ is known to be 1 for all $i$.

As I understand it, the chi-squared goodness-of-fit measure for a test statistic $X$ is $$Q(x) = \int_X^\infty f_k(x)\,\mathrm{d}x = 1 - F_k(X)$$ where $f_k$ is the chi-squared PDF with $k$ degrees of freedom, and $F_k$ is the CDF. $k$ is the number of observations minus the number of model degrees of freedom: $k=10-3=7$ in this example.

The test statistic for the oracle instance is 9.45, and for the candidate instance it's 918.10. The goodness-of-fit values I get are 0.22 for the oracle (pretty reasonable), and $6{\cdot}10^{-194}$ for the candidate instance.

It seems to me that this particular choice of goodness-of-fit measure is way too sensitive: the candidate instance was off by a minuscule value in the parameter space, but the nature of my observations hugely inflates the chi-squared value, and so my goodness-of-fit measure basically ends up screaming: "this model instance is HORRIBLE!" This measure won't provide reasonable values for model instances that are anything but extremely precisely correct.

This measure works fine as an objective function for model-fitting: less-correct model instances give even lower goodness-of-fit values. But when I report on the performance of multiple fitting algorithms or models, for example, I want to use the goodness-of-fit values of the best instances I can compute as objective measures by which to compare them. And these values are just laughably small, so tiny that it seems silly to claim that one technique is better than another on the basis of comparing, say, $10^{-194}$ to $10^{-280}$. Their statistical interpretation, also, is a little suspect; this value $Q(x)$ that I compute is a probability of a sort, and probabilities that small just seem meaningless.

So: what would be a better choice of measure (or measures) to describe the goodness-of-fit of instances of this model? And, so I can learn from my mistakes: what aspects of the problem should have led me to choose that alternative measure in the first place?

[At the risk of making a very long question even longer, here are my answers to questions in the comments:]

I have used precisely this value $Q(x)$ (actually, the negative of its log) as my objective function for model fitting. The nature of the problem makes it so that I can only use iterative model refinement to fit, and the example of (1,8.9,300) is analogous to what my fitting process might find after many iterations: not exactly identical to the ground truth, but pretty close, and predicting the vast majority of observations correctly.

The essential problem I'm confronting is that, while the oracle instance is definitely a better fit than the candidate instance, the candidate instance was also quite good, and yet the claimed Q(x) is incredibly low. My sense of "quite good" comes from a couple different angles:

  • Two of the three parameter values were exactly right, and the third was off by less than 10%.
  • Nine of the ten predicted values were correct (well within the bounds, for each observation, of the known observation noise), and the tenth was off by only 10% (though very far down the tail of the noise distribution on the actual observation).

It seems like the chi-squared GoF function is overreacting, in a way. I was hoping that this measure would provide some absolute, objective basis for discussing the quality of a fit, but this one seems to reject anything that isn't identical to the ground truth.

$\endgroup$
  • $\begingroup$ Your situation isn't terribly clear to me but your interpretation of p-value as goodness of fit is certainly one problem. Don't use p-values to decide whether a fit is good or bad. What would made a model a 'good fit' or 'bad fit' for your purposes? (What, if possible, is your loss function?) $\endgroup$ – Glen_b Jul 20 '14 at 23:00
  • $\begingroup$ @Glen_b Sorry, my use of the term "p-value" was probably incorrect and misleading; I've edited the question to remove it. I also tried to explain my context and motivation a little more. Please let me know if there are other specific questions I can answer; as a stats outsider it's hard for me to translate into language that is useful. $\endgroup$ – Jadrian Miles Jul 21 '14 at 4:27
  • $\begingroup$ @Glen_b As for "what would make a model a good fit"... the essential problem I'm confronting is that, while the oracle instance is definitely a better fit than the candidate instance, the candidate instance was off from the ground truth by less than 10% in only one of three parameter values, and yet the claimed Q(x) is incredibly low. It seems like the GoF function is overreacting, in a way. I was hoping that a GoF measure would provide some absolute, objective basis for discussing the quality of a fit, but this one seems to reject anything that isn't identical to the ground truth. $\endgroup$ – Jadrian Miles Jul 21 '14 at 4:33
  • $\begingroup$ @Glen_b and if I understand what you mean by loss function... I have used precisely this value Q(x) (actually, the negative of its log) as my objective function for model fitting. The nature of the problem makes it so that I can only use iterative model refinement to fit, and the example of (1,8.9,300) is analogous to what my fitting process might find after many iterations: not exactly identical to the ground truth, but pretty close, and predicting the vast majority of observations correctly. $\endgroup$ – Jadrian Miles Jul 21 '14 at 4:35
  • $\begingroup$ You don't have an hypothesis testing problem. The hypothesis test answers a different question to the one you're interested - but answers that different question well enough. What you want to do is figure out how to answer your actual underlying question, not answer a different question. Your actual question is probably something along the lines of 'is it close enough?'... (ctd) $\endgroup$ – Glen_b Jul 21 '14 at 5:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.