6
$\begingroup$

In a social experiment that I was conducting, I was trying to count the number of people each user contacted in a period of 10 days. The population size was 100 for the experiment. Based on the values that I calculated, I fit a negative binomial distribution to the data (the Q-Q plot is given below).

Conventional wisdom says that most networks amongst humans follow a power law distribution. I am guessing that my population size is too small to make a full conclusion about anything but is there any kind of relation between a negative binomial distribution and a power law distribution? I am asking this because I read a few days back that Normal Distribution and Gamma distribution (whose discrete analogue is the negative binomial) have a special role in that many other distributions can be derived from the Gamma distribution. I am wondering if this is true even with the power law distribution. I am a beginner in statistics so kindly point me in the right direction if I am out of track.

alt text

$\endgroup$
  • $\begingroup$ Might you be able to remember where you 'read a few days back'? $\endgroup$ – onestop Dec 1 '10 at 20:48
  • $\begingroup$ @onestop: Sorry. I read that in some research paper but I found a quick link on wiki http://en.wikipedia.org/wiki/Generalized_gamma_distribution for a generalized gamma distribution. Please feel free to correct me though. $\endgroup$ – Legend Dec 1 '10 at 21:04
  • 1
    $\begingroup$ This probability plot strongly suggests a mixture of two distributions, one located between 5 and 60 and the other between 50 and 150+. This casts doubt on any simpler characterization, like negative binomial. $\endgroup$ – whuber Dec 1 '10 at 21:27
  • $\begingroup$ @whuber: That is interesting. Would you have any suggestions on how to proceed? Do I start looking for ways to separate the distributions? $\endgroup$ – Legend Dec 2 '10 at 1:42
  • $\begingroup$ Not necessarily. It depends on why you're doing this fitting. At this stage it seems you are exploring the data. Let the data behavior guide you. Can you find any possible cause or explanation for a mixture? Would there be a meaningful difference between values in the 0-60 range versus values in the 40-150+ range? In short, what can you learn about your social experiment from this? $\endgroup$ – whuber Dec 2 '10 at 13:50
2
$\begingroup$

There are many power-law distributions, so you have a lot of possible models. You might start by trying to fit a log-series distribution, which is a limiting case of the negative binomial.

Don't think a priori that you have a mixture distribution as suggested by whuber until you've estimated model parameters and done at least a goodness of fit test. Long-tail distributions, like power-law, log-series, Zipf, etc., typically have what look like outliers in the right-hand tail; their separation from the bulk of the observations is just an artifact of (relatively) small sample size. Mixtures are a pain in the butt to estimate, since some regions overlap. You can often avoid that sort of problem by stepping up your modeling one level with something like Poisson regression, assuming you have some covariate (predictor) data about each user -- this basically does the mixing for you.

The Johnson, Kemp, and Kotz reference given at the end of the referenced Wikipedia article has everything you'd ever want to know about all these distributions, including many methods of parameter estimation.

$\endgroup$
  • 1
    $\begingroup$ This probability plot does not exhibit any outliers at all, but it does indicate a region of sparse probability density around 40-60. This will not be detected by most GoF tests, which will be invalid anyway. Distribution fitting is exploratory, not confirmatory, and so requires more flexibility and creativity than is afforded by parameter estimation and rote application of hypothesis tests. If a mixture is suggested by theory--and it might be here--and if capturing that in a model might be important, then it is worthwhile considering despite the complications that ensue. $\endgroup$ – whuber Dec 2 '10 at 13:44
  • $\begingroup$ (A GoF test, to be valid, must quantitatively account for how many possible distributions one has attempted to fit to the data.) $\endgroup$ – whuber Dec 2 '10 at 13:46

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.