# Measuring goodness-of-fit in a model that combines two distributions

I have data with a double peak that I'm trying to model, and there's enough overlap between the peaks that I can't treat them independently. A histogram of the data might look something like this:

I've created two models for this: one uses two Poisson distributions, and the other uses two negative binomial distributions (to account for overdispersion). What's the appropriate way to tell which model fits the data more accurately?

My initial thought is that I could use a Kolmogorov-Smirnov test to compare each model to the data, then do a likelihood ratio test to see if one is a significantly better fit. Does this make sense? If so, I'm not exactly sure how to perform the likelihood ratio test. Is chi-squared appropriate, and how many degrees of freedom do I have?

If it helps, some (very simplified) R code for the models might look something like this:

## inital data points

#create model data
model.pois = c(rpois(1000000,200),rpois(500000,250))
model.nb = c(rnbinom(1000000,200,0.5),rnbinom(500000,275,0.5)

#Kolmogorov-Smirnov test
#use ks.boot, since it's count data that may contain duplicate values
kpois = ks.boot(model.pois,a)
knb = ks.boot(model.nb,a)

#here's where I'd do some sort of likelihood ratio test
# . . .


Edit: Here's an image that may explain the data and the distributions I'm fitting better. It's totally clear from the visualization that the second model (using the negative binomial dist to account for overdispersion) is a better fit. I'd like to show this quantitatively, though.

(red - data, green - model)

• do you know the probability distribution of the values in each bin ? The y axis label makes me think that this could be Poissonian or Multinomial ? (assuming a model gives you the mean in each bin) Commented Sep 12, 2010 at 7:47
• The data is essentially drawn from two Poisson processes, but there are hidden variables that I can't correct for, leading to overdispersion. Thus, a negative binomial is definitely a better model. (see the new image/text I added above). I need to show that my nb model fits better quantitatively. Commented Sep 12, 2010 at 11:36
• How about a metric like Mean Squared Error between actual vs predicted values?
– user28
Commented Sep 12, 2010 at 11:42
• hrmm - I like that idea, Srikant. It's a lot simpler than what I was thinking, but still makes sense. Throw into a an answer below so I can credit it and send some rep your way. I'm still interested in hearing other methods, but this may work for now. Commented Sep 12, 2010 at 18:13

You can use a metric such as Mean Squared Error between actual vs predicted values to compare the two models.

You can't compare them directly since the Negative Binomial has more parameters. Indeed the Poisson is "nested" within the Negative Binomial in the sense that it's a limiting case, so the NegBin will always fit better than the Poisson. However, that makes it possible to consider something like a likelihood ratio test but the fact that the Poisson is at the boundary of the parameter space for the negative binomial may affect the distribution of the test statistic.

In any case, even if the difference in number of parameters wasn't a problem, you can't do K-S tests directly because you have estimated parameters, and K-S is specifically for the case where all parameters are specified. Your idea of using the bootstrap deals with this issue, but not the first one (difference in number of parameters)

I'd also be considering smooth tests of goodness of fit (e.g. see Rayner and Best's book), which, for example, can lead to a partition the chi-square goodness of fit test into components of interest (measuring deviations from the Poisson model in this case) - taken out to say fourth order or sixth order, this should lead to a test with good power for the NegBin alternative.

(Edit: You could compare your poisson and negbin fits via a chi-squared test but it will have low power. Partitioning the chi-square and looking only at say the first 4-6 components, as is done with smooth tests might do better.)

• Thanks. That clarifies a bunch of things and opens up a whole slew of new questions which I'll have to do some research on. I guess my main question is, does what you're saying mean that something more simple, like just taking root mean squared error, isn't a valid way to approach this problem? I'll grant that it's probably not as robust and won't give me a p-value, but it's something I could do quickly while I try to track down a copy of the book you reference. Any thoughts would be appreciated. Commented Sep 13, 2010 at 2:58
• imagine that you had a set of points (x,y) and you were considering whether you might fit a straight line or a quadratic. If you compared the RMSE, the quadratic would always beat the straight line, because the line is a quadratic with one parameter set to zero: if the least squares estimate of the parameter is exactly zero (which has zero probability for continuous response), it's a tie, and in every other case the line loses. It's the same with the Poisson vs the negative binomial - a free Negative Binomial can always fit at least as well as a free Poisson. Commented Sep 14, 2010 at 10:28
• Nice explanation - I get what you're saying now. I think my case is a little different, because I'm not doing regression to get a fit, but rather, I'm basing the extra NB parameter on outside information (I expect the var/mean ratio to be N). Since Poisson is the special case where N=1, what I'm really comparing is the choice of N. I agree that if I was doing regression, the NB would always be able to find a better fit, because it's less constrained. In my case, where I'm choosing a value for N up front, it would certainly be possible to choose some crazy value of N that makes the fit worse. Commented Sep 14, 2010 at 13:40
• I'm certainly going to read up on the smooth tests of goodness of fit that you suggested though. Thanks for the informative answers. Commented Sep 14, 2010 at 15:31
• Sorry about not realizing that the data didn't come into the choice of overdispersion parameter. There may be some argument for doing it your way, but if the external estimate is likely to reflect what you actually observe, the NB still may have some advantage depending on circumstances. Commented Sep 15, 2010 at 3:28