2
$\begingroup$

Suppose, we have a set of 10,000 individuals. Each individual falls into exactly one of 200 categories. [Edit: The categories are phenotypes (different potential outcomes) of the one property that is observed]. Observing the individuals, we can count for each category the number of corresponding individuals. This number is always a non-negative integer. Suppose, that around 30 categories are empty.

Now I would like the find a distribution describing the number of individuals in each category. I did some tests with a lognormal distribution using R and the qqplots seemed to perform well.

However, I did not know how to respect the zero values which are clearly forbidden for lognormal distributed data. During my tests, I modelled the 'zero' phenomenon separately.

To this purpose, I cut off the zero values and then fitted the lognormal distribution. Going reverse (i.e. generating numbers according to the original distribution), I first used random numbers from the binomial distribution with $n=1$, deciding whether I get a zero or a non-zero. Second, I used numbers from the fitted lognormal distribution for the non-zero values.

  1. Is the lognormal approach generally a bad idea for data with zeroes?
  2. How would you evaluate my approach?
  3. Are there better ways to find a distribution for discrete, non-negative data?
$\endgroup$
4
  • $\begingroup$ Are the categories ordered, or just numbered that way? $\endgroup$ Jan 12 '15 at 19:16
  • $\begingroup$ Note that the data are counts, not continuous, so you should probably be considering distributions on the nonnegative integers. It's a little unclear to me -- the counts are from a multivariate distribution. Are you trying to fit the vector of counts in some way? How are you ordering them? Can you explain with a short numerical example what it is you're actually doing? $\endgroup$
    – Glen_b
    Jan 13 '15 at 5:24
  • $\begingroup$ The categories are phenotypes (different potential outcomes) of the one property that is observed and each individual falls into exactly one category. Hence, the categories are indexed, but there is no semantic ordering relation. $\endgroup$
    – user66430
    Jan 13 '15 at 12:04
  • 1
    $\begingroup$ As an example, I have observed 30 categories with 0 individuals, 5 categories with 1 individual, 4 categories with 2 individuals, 10 categories with 3 individuals, ..., and 1 category with 740 individuals. The goal is to describe how many individuals fall into a random category. This number should be independent from the specific category. $\endgroup$
    – user66430
    Jan 13 '15 at 12:12
2
$\begingroup$

Your data is the classic use case for the multinomial distribution.

The binomial distribution models $N$ independent and identical 'experiments' with outcomes in $\{0, 1\}$, where "1" is often taken to indicate "success," and probability of success $p$ that is the same in all experiments.

The multinomial distribution is its generalization to $K$ integer outcomes, $\{0, \dots, k\}$, and outcome probabilities $\{p_1, \dots, p_K\}$ and $p_0 = 1 - \sum_{k=1}^K p_k$. Your data follows a multinomial distribution with $N = 10,000$, $K = 199$ (taking one category arbitrarily as a baseline). You would like to estimate the vector of probabilities $p$.

The most intuitive estimator for each probability is just the fraction of the sample that falls into each category. If $N_k$ is the number of individuals observed to be in category $k$, then $\widehat{p_k} = \frac{N_k}{N}$. This also happens to be the maximum likelihood estimator.

Think of rolling a 200-sided die 10,000 times.

$\endgroup$
4
  • 1
    $\begingroup$ If the question concerned the distribution of counts when sampling from this population, the answer would indeed be multinomial. But since the number of individuals in each category is arbitrary, it seems there is no basis to suppose those numbers themselves should be modeled as if they themselves were a multinomial sample of some (hyper) population. Indeed, nothing appears to be gained by that additional level of complexity, anyway: both descriptions use $199$ independent parameters. $\endgroup$
    – whuber
    Jan 12 '15 at 20:48
  • $\begingroup$ @whuber I'm mainly just addressing the line "Now I would like the find a distribution describing the number of individuals in each category." $\endgroup$ Jan 12 '15 at 21:06
  • $\begingroup$ But the multinomial doesn't do that. The answer to the quoted question is literally the observed frequency distribution itself. To simplify that would require additional assumptions or knowledge about the counts, or relationships among the individuals, etc. $\endgroup$
    – whuber
    Jan 12 '15 at 22:18
  • $\begingroup$ @whuber I don't see how else you would model this data: a draw from the multinomial distribution is a table of category counts. The problem as I see it is that the category counts cannot be independent with a known and fixed sample size. The pairwise dependence is probably weak and the sample size is quite large, so you could probably get away with a pooled-$\lambda$ Poisson distribution, but that's an extreme oversimplification that I don't think accomplishes anything. The only other option I can think of is a copula, which in my understanding is fraught with danger for count data. $\endgroup$ Jan 14 '15 at 1:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy