10
$\begingroup$

I have a nonlinear model $y=\Phi(f(x,a)) + \varepsilon$, where $\Phi$ is the cdf of the standard normal distribution and f is nonlinear (see below). I want to test the goodness of fit of this model with parameter $a$ to my data $(x_1,y_1),(x_2,y_2),\dots,(x_n,y_n)$, after having used maximum likelihood estimation to find $a$. What would be an appropriate test? I would like to use this test to label a bad fit as bad and determine whether more data should be collected.

I've looked into using deviance, which compares this model against the saturated model, with its corresponding test of goodness of fit using the $\chi^2_{n-1}$ distribution. Would this be appropriate? Most of what I have read about deviance applies it to GLMs, which is not what I have. If the deviance test is appropriate, what assumptions need to hold to make the test valid?

Update: $f = \frac{x-1}{a\sqrt{x^2+1}}$ for $x>1,a>0$ in case this helps.

Improve this question
$\endgroup$
4
  • 1
    $\begingroup$ The answer depends on the purpose of the analysis and on the underlying probability model you have used; there is no unique or best mathematical answer. For example, we would measure goodness of fit differently for a model of the form $y=\Phi(f(x,a)+\varepsilon)$ than for one of the form $y=\Phi(f(x,a))+\varepsilon$ (with iid errors $\varepsilon$). $\endgroup$
    – whuber ♦
    Commented Sep 1, 2011 at 14:42
  • $\begingroup$ Thanks. I've clarified my question. I'm aware that there is no best answer, however, I'd still like to know whether deviance is appropriate for testing goodness-of-fit here, and if not, what is another test that would be appropriate for marking a fit as very poor and saying more data needs to be collected (assuming the model is correct) or saying the model does not describe the data. $\endgroup$
    – spadequack
    Commented Sep 2, 2011 at 3:22
  • 1
    $\begingroup$ Is your target variable $y \in {0,1}$ or is it continuous? If the former, then you could frame the model as $p(y=1) = \Phi(f(x,a))$ instead of having the additive error term, and compare predicted with actual $y=0$ and $y=1$ to get the true and false positive rates, or compare to a baseline model where $p(y=1) = \bar{y}$, or deviance, or several other alternatives. If the latter, what's the distribution you're assuming for the residual? $\endgroup$
    – jbowman
    Commented Dec 3, 2011 at 19:45
  • 1
    $\begingroup$ Voting to close because the request for clarification has gone unanswered. $\endgroup$
    – whuber ♦
    Commented Dec 28, 2012 at 5:30

3 Answers 3

Reset to default
1
$\begingroup$

Use the "npcmstest" package in library "NP" if you are using the R platform. Warning: The function may take several minutes to evaluate your model.

You can also consider an information-theoretic comparison of the response distribution and the predictive distribution (i.e. KL divergence, cross-entropy, etc.)

Improve this answer
$\endgroup$
2
  • $\begingroup$ It seems that method requires a model from either lm or glm. How would this work for a nonlinear model? (Yes, I'm using R.) I've added what $f$ is to my question in case that helps. $\endgroup$
    – spadequack
    Commented Sep 3, 2011 at 23:08
  • $\begingroup$ @are you using gam or the like (mgcv package)? If not, you should check it out. $\endgroup$
    – suncoolsu
    Commented Sep 3, 2011 at 23:22
1
$\begingroup$

Here's how I would do it, basically a likelihood ratio test. But remember they "key" to understanding a goodness of fit test, is to understand the class of alternatives that you are testing against. Now we have the likelihood for each individual data point as:

$$p(y_i|x_i,a,I)=g(\epsilon_i)=g(y_i-f_i)$$

Where $g(\epsilon)$ is the likelihood of the error term in your model, and $f_i=\frac{x_i-1}{a\sqrt{x^2_i+1}}$ is the model prediction for the ith data point, given $x_i$ and $a$. Now for each data point $(x_i,y_i)$ we can choose an $a$ such that $f_i=y_i$ - the "saturated model" as you call it. So you're $\chi^2$ test is appropriate here, if you only want to test to alternatives within the class of those with the same error likelihood, $g(\epsilon)$, and you have independence of each of the likelihoods (i.e. knowing another $x_j,y_j$ would be of no help in predicting $y_i$, given $a$).

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ This is not going to work, since the degrees of freedom of the likelihood ratio test grows as $O(n)$ for the saturated model. $\endgroup$
    – StasK
    Commented Mar 2, 2012 at 21:10
0
$\begingroup$

In linear regression context, goodness of fit testing is often conducted against a more complicated alternative. You have a linear regression -- throw in some polynomial terms to test if the linear form is enough. Since you already have a nonlinear functional form, the complicated alternative you would need to consider would have to be that of non-parametric regression. I won't try to provide an introduction to the topic, as it requires a mindset of its own, and it is worth a separate proper introduction. For the test of parametric vs. nonparametric regressions, Wooldridge (1992) or Hardle and Mammen (1993), they do very similar things. Hardle also wrote a great book on the topic.

Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.