Calculating optimal number of bins in a histogram for n, where n ranges from 30-350

Question

I'm interested in finding as optimal of a method as I can for determining how many bins I should use in a histogram. My data should range from 30 to 350 objects at most, and in particular I'm trying to apply thresholding (like Otsu's method) where "good" objects, which I should have fewer of and should be more spread out, are separated from "bad" objects, which should be more dense in value. A concrete value would have a score of 1-10 for each object. I'd had 5-10 objects with scores 6-10, and 20-25 objects with scores 1-4. I'd like to find a histogram binning pattern that generally allows something like Otsu's method to threshold off the low scoring objects. However, in the implementation of Otsu's I've seen, the bin size was 256, and often I have many fewer data points that 256, which to me suggests that 256 is not a good bin number. With so few data, what approaches should I take to calculating the number of bins to use?

Answer 1

The Freedman-Diaconis rule is very robust and works well in practice. The bin-width is set to $h=2*\text{IQR}*n^{-1/3}$. So the number of bins is (max-min)/$h$.

In R, you can use hist(x,breaks="FD")

Answer 2

Maybe the paper "Variations on the histogram" by Denby and Mallows will be of interest:

This new display which we term "dhist" (for diagonally-cut histogram) preserves the desirable features of both the equal-width hist and the equal-area hist. It will show tall narrow bins like the e-a hist when there are spikes in the data and will show isolated outliers just like the usual histogram.

They also mention that code in R is available on request.

Answer 3

If you use too few bins, the histogram doesn't really portray the data very well. If you have too many bins, you get a broken comb look, which also doesn't give a sense of the distribution.

One solution is to create a graph that shows every value. Either a dot plot, or a cumulative frequency distribution, which doesn't require any bins.

If you want to create a frequency distribution with equally spaced bins, you need to decide how many bins (or the width of each). The decision clearly depends on the number of values. If you have lots of values, your graph will look better and be more informative if you have lots of bins. This wikipedia page lists several methods for deciding bin width from the number of observations. The simplest method is to set the number of bins equal to the square root of the number of values you are binning.

This page from Hideaki Shimazaki explains an alternative method. It is a bit more complicated to calculate, but seems to do a great job. The top part of the page is a Java app. Scroll past that to see the theory and explanation, then keep scrolling to find links to the papers that explain the method.

Answer 4

I'm not sure this counts as strictly good practise, but I tend to produce more than one histogram with different bin widths and pick the histogram which histgram to use based on which histgram fits the interpretation I'm trying to communicate best. Whilst this introduces some objectivity into the choice of histogram I justify it on the basis I have had much more time to understnad the data than the person I'm giving the histogram to so I need to give them a very concise message.

I'm also a big fan of presenting histograms with the same number of points in each bin rather than the same bin width. I usually find these represent the data far better then the constant bin width although they are mopre difficult to produce.

Answer 5

Did you see the Shimazaki-Shinomoto method?

Although it seems to be computationally expensive, it may give you good results. It's worth giving it a try if computational time is not your problem. There are some implemantations of this method in java, MATLAB, etc, in the website below (Stack doesn't allow me to link you directly), which runs fast enough.

http://176.32.89.45/~hideaki/res/histogram.html

Best of luck!

Answer 6

If I need to determine the number of bins programmatically I usually start out with a histogram that has way more bins than needed. Once the histogram is filled I then combine bins until I have enough entries per bin for the method I am using, e.g. if I want to model Poisson-uncertainties in a counting experiment with uncertainties from a normal distribution until I have more than something like 10 entries.

Answer 7

I think Sturges' rule can be used for n < 200; where n is the number of observations