Suppose I have N smarties, each of which is one of C distinct colours.

Suppose further that N is known and largish (10,000) but C is not, and that for each colour C there are $c_i$ smarties of that colour. Whilst the distribution of $c_i$ is unknown, we have the vague assumption that no $c_i$ will be very large or small, eg no one colour will make up more more than 99 or less than 1 percent of the total number of smarties.

If I take a sample of size n and determine that it contains $x_i$ smarties of each colour, how can I produce a good estimate the total number of classes?

Note -> I'm familiar with http://arxiv.org/pdf/0708.2153.pdf and http://www.jstor.org/stable/2290471 but was wondering if anyone had anything else to contibute. Also I'm concerned with the case where N is large but not infinite, which is fairly distinct from the first paper.


So by reading

Haas, Peter J., and Lynne Stokes. "Estimating the number of classes in a finite population." 
 Journal of the American Statistical Association 93.444 (1998): 1475-1487.

I came up with

def random_sample(n,data):
    indices = np.random.choice(np.arange(0,len(data)),size=n,replace=False)
    # That space efficiency wins no awards.
    return data[indices]

def estimate_n_classes(sample,population_size):
    n = float(len(sample))
    dn = float(len(np.unique(sample))) 
    f = Counter([count for label, count in Counter(sample).iteritems()])
    q = n / population_size
    result = ((1 - ((1 - q) * f[1] / n))) ** (-1) * dn
    return result

This is an implementation of $D_{uj1}$ from the paper which was sufficient for my needs.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.