How to assess goodness of fit of a particular nonlinear model? I have a nonlinear model $y=\Phi(f(x,a)) + \varepsilon$, where $\Phi$ is the cdf of the standard normal distribution and f is nonlinear (see below). I want to test the goodness of fit of this model with parameter $a$ to my data $(x_1,y_1),(x_2,y_2),\dots,(x_n,y_n)$, after having used maximum likelihood estimation to find $a$. What would be an appropriate test? I would like to use this test to label a bad fit as bad and determine whether more data should be collected.
I've looked into using deviance, which compares this model against the saturated model, with its corresponding test of goodness of fit using the $\chi^2_{n-1}$ distribution. Would this be appropriate? Most of what I have read about deviance applies it to GLMs, which is not what I have. If the deviance test is appropriate, what assumptions need to hold to make the test valid?
Update: $f = \frac{x-1}{a\sqrt{x^2+1}}$ for $x>1,a>0$ in case this helps.
 A: Use the "npcmstest" package in library "NP" if you are using the R platform. Warning: The function may take several minutes to evaluate your model.
You can also consider an information-theoretic comparison of the response distribution and the predictive distribution (i.e. KL divergence, cross-entropy, etc.)
A: Here's how I would do it, basically a likelihood ratio test.  But remember they "key" to understanding a goodness of fit test, is to understand the class of alternatives that you are testing against.  Now we have the likelihood for each individual data point as:
$$p(y_i|x_i,a,I)=g(\epsilon_i)=g(y_i-f_i)$$
Where $g(\epsilon)$ is the likelihood of the error term in your model, and $f_i=\frac{x_i-1}{a\sqrt{x^2_i+1}}$ is the model prediction for the ith data point, given $x_i$ and $a$.  Now for each data point $(x_i,y_i)$ we can choose an $a$ such that $f_i=y_i$ - the "saturated model" as you call it.  So you're $\chi^2$ test is appropriate here, if you only want to test to alternatives within the class of those with the same error likelihood, $g(\epsilon)$, and you have independence of each of the likelihoods (i.e. knowing another $x_j,y_j$ would be of no help in predicting $y_i$, given $a$).  
A: In linear regression context, goodness of fit testing is often conducted against a more complicated alternative. You have a linear regression -- throw in some polynomial terms to test if the linear form is enough. Since you already have a nonlinear functional form, the complicated alternative you would need to consider would have to be that of non-parametric regression. I won't try to provide an introduction to the topic, as it requires a mindset of its own, and it is worth a separate proper introduction. For the test of parametric vs. nonparametric regressions, Wooldridge (1992) or Hardle and Mammen (1993), they do very similar things. Hardle also wrote a great book on the topic.
