Fitting a distribution to data (e.g. with maximum likelihood), or testing goodness of fit (e.g. with Kolmogorov-Smirnov) assumes that the data are randomly drawn from a population. But what if the data do not satisfy this assumption?

I want to fit a theoretical distribution (log-normal, generalized beta, etc.) to an earnings distribution that comes from administrative data. The earnings distribution is provided as binned data, based on which I can calculate points in the empirical distribution. Specifically, I can calculate $F(X_i)$ where $X_i$ are upper limits of the bins.

But bins are clearly not iid. Does that matter in maximum likelihood fitting, or in goodness of fit testing?

Another question is of sample size. For example, sample size is necessary to calculate Kolmogorov-Smirnov statistic. But what is the sample size in this case, the number of bins, or the population size (because the data is based on the entire population, not only a sample)?

  • $\begingroup$ If you know how many records are in each bin, you could treat this as that number of i.i.d. samples interval censored to lie within the interval corresponding to the bin. How do you mean the bins are not i.i.d.? Is there some kind of correlation between records? I.e. what is your concern? That all the incomes are from the same country or that multiple from the same household go into different bins? $\endgroup$ – Björn Nov 6 '18 at 16:45
  • $\begingroup$ @Björn, here's an example. The data shows that there are exactly 21,816,123 earners with earnings between \$.01 and \$4,999.99. The total number of all earners is 163,520,606. Therefore, F(\$4,999.99) = 21,816,123/163,520,606 = .1334. Similarly, F(\$9,999.99) = 35,346,000/163,520,606=.216, and so on. So these values of $F(X_i)$ are the values to which I want to fit a theoretical distribution. But $X_i$ are not iid. And values of $F(X_i)$ are not calculated based on the sample of $X_i$; they are calculated based on the actual population and its assignment to these bins. $\endgroup$ – Damir Nov 6 '18 at 17:53
  • $\begingroup$ Any reason why what I suggest does not work? That would appropriately take into account number of records in each bin. $\endgroup$ – Björn Nov 6 '18 at 17:55
  • $\begingroup$ @Björn, I am not sure that it would not work, but I am skeptical because it implies that the sample size equals population size, whereas the data offer only a couple of dozens of values of $F(X)$. $\endgroup$ – Damir Nov 6 '18 at 18:49
  • $\begingroup$ Would you please post a link to the binned data? I would like to try fitting this. $\endgroup$ – James Phillips Nov 7 '18 at 16:34

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.