5
$\begingroup$

I'm trying to fit a model estimating waiting time using negative binomial regression, but I'm not sure how to assess the goodness of fit for my model. I would like to compare the negative binomial model to a Poisson model. I have approximately $4,000$ data points. Any suggestions?

Thanks!

$\endgroup$
  • 1
    $\begingroup$ What is your goal? Are you trying to compare your model to another negative binomial regression model? Are you trying to compare to to some other GLM model with a different link function? $\endgroup$ – TrynnaDoStat Aug 12 '14 at 14:36
  • $\begingroup$ I'm trying to compare to some other GLM model with a different link function--for example, Poisson Regression. $\endgroup$ – monkeybiz7 Aug 12 '14 at 14:38
  • $\begingroup$ How big is your sample? $\endgroup$ – TrynnaDoStat Aug 12 '14 at 15:00
  • $\begingroup$ Something like 4000 data points. $\endgroup$ – monkeybiz7 Aug 12 '14 at 15:00
  • $\begingroup$ One easy way to judge whether a Poisson model might be sufficient would be to look at the size of an estimated overdispersion parameter. If its close to 1, the Poisson may be adequate. $\endgroup$ – Glen_b Sep 14 '14 at 22:48
2
$\begingroup$

Generally speaking, a good fitting model means does a good job generalizing to data not captured in your sample. A good way to mimic this is through cross-validation (CV). To do this, you subset your data into two parts: a testing data set and a training data set. Based on your sample size, I would recommend randomly putting 70% of your data into a testing data set and the remaining 30% in a training data set.

Now, build both the Poisson model and the negative binomial model based on your training data set. Calculate the predicted values for the data in your testing data set and compare it to the actual values in the following way:

$\sum_{i=1}^{n_2} (Y_i - \hat{Y}_i)^2$

where $n_2$ is the sample size of your training data set, $Y_i$ is the actual value of the dependent variable, and $\hat{Y}_i$ is the predicted value of the dependent variable.

Whichever model provides a lower value for the above expression is the preferred model.

Now, there is a modification of this called k-folds CV. What it will do is split your data into $k$ approximately equal subsets (called "fold") and will predict each fold using the remaining folds as training data. Setting $k=4$ seems reasonable to me.

The relevant R function for this is cv.glm() in the boot package. More information here: http://stat.ethz.ch/R-manual/R-patched/library/boot/html/cv.glm.html

$\endgroup$
  • $\begingroup$ Thanks. So sum of squared error is the metric I should use? $\endgroup$ – monkeybiz7 Aug 12 '14 at 15:36
  • $\begingroup$ Is it possible to use chi-squared goodness of fit on the validation set? If so, what sort of distribution should I compare my residuals against? $\endgroup$ – monkeybiz7 Aug 12 '14 at 16:37
0
$\begingroup$

I would suggest to use approaches such as the Akaike information criterion or Bayesian information criterion and compare the returned values of your two models (GLM vs. NBR).

Also, using cross-validation to see which model performs worse could be an option and is commonly used, at least to get an impression about how a learned model performs.

$\endgroup$
  • $\begingroup$ Thanks for your answer. The problem is I'm not sure what metric to use for assessing performance during cross-validation. $\endgroup$ – monkeybiz7 Aug 12 '14 at 15:08
  • $\begingroup$ Quoting from the wikipedia article on AIC: Given a set of candidate models for the data, the preferred model is the one with the minimum AIC value. So the AIC/BIC are already your metric and the objective is to minimize the value. $\endgroup$ – Shadow Aug 13 '14 at 11:58
  • 2
    $\begingroup$ It appears AIC is more for comparing models, and not for figuring out how well (in strict terms) your model is doing. That would be more like RMS, right? $\endgroup$ – monkeybiz7 Aug 13 '14 at 15:26
0
$\begingroup$

So, if you just want to know if your fit is significant, you can compute the p-value.

First, find a good metric for your problem. For distributions a typically used is the Kolmogorov-Smirnov distance: $KS(f,g)=max|f(x)-g(x)|$. Now, call $E$ the cdf of your data and $P$ the analytical cdf of your fit, then $KS_0=KS(E,P)$.

Now, we want to compute the probability of obtaining a $KS>KS_0$, given that we assume that your fit is correct. We can easily do that by sampling $n$ times a set of points of size 4000 from your fitted distribution; then we fit the set, and we compute the $KS$ between the sampled set and the fit of the sampled. The p-value is simply the proportion of the $n$ sets where this $KS>KS_0$.

If the resulted p-value$>0.05$ (or some significance level that you need to set) then your data is compatible with your fit.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.