Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I'm interested in finding as optimal of a method as I can for determining how many bins I should use in a histogram. My data should range from 30 to 350 objects at most, and in particular I'm trying to apply thresholding (like Otsu's method) where "good" objects, which I should have fewer of and should be more spread out, are separated from "bad" objects, which should be more dense in value. A concrete value would have a score of 1-10 for each object. I'd had 5-10 objects with scores 6-10, and 20-25 objects with scores 1-4. I'd like to find a histogram binning pattern that generally allows something like Otsu's method to threshold off the low scoring objects. However, in the implementation of Otsu's I've seen, the bin size was 256, and often I have many fewer data points that 256, which to me suggests that 256 is not a good bin number. With so few data, what approaches should I take to calculating the number of bins to use?

share|improve this question

7 Answers 7

up vote 16 down vote accepted

The Freedman-Diaconis rule is very robust and works well in practice. The bin-width is set to $h=2*\text{IQR}*n^{-1/3}$. So the number of bins is (max-min)/$h$.

In base R, you can use hist(x,breaks="FD")

For other plotting libraries without this option (e.g. ggplot2), you can calculate binwidth as:

bw <- diff(range(x)) / (2 * IQR(x) * length(x)^(1/3)))
### for example
ggplot() + geom_histogram(aes(x), binwidth = bw)
share|improve this answer
    
As a note, by default (so, if you don't specify breaks) R –  nico Jan 7 '11 at 21:36
    
@nico. The default in R is breaks="Sturges" which does not always give good results. –  Rob Hyndman Jan 8 '11 at 21:48
    
for whatever reason my comment was truncated... I meant to write "by default (so, if you don't specify breaks) R uses the Sturges algorithm"... odd! –  nico Jan 9 '11 at 0:15

If you use too few bins, the histogram doesn't really portray the data very well. If you have too many bins, you get a broken comb look, which also doesn't give a sense of the distribution.

One solution is to create a graph that shows every value. Either a dot plot, or a cumulative frequency distribution, which doesn't require any bins.

If you want to create a frequency distribution with equally spaced bins, you need to decide how many bins (or the width of each). The decision clearly depends on the number of values. If you have lots of values, your graph will look better and be more informative if you have lots of bins. This wikipedia page lists several methods for deciding bin width from the number of observations. The simplest method is to set the number of bins equal to the square root of the number of values you are binning.

This page from Hideaki Shimazaki explains an alternative method. It is a bit more complicated to calculate, but seems to do a great job. The top part of the page is a Java app. Scroll past that to see the theory and explanation, then keep scrolling to find links to the papers that explain the method.

share|improve this answer
1  
+1 $\sqrt{n}$ is a good place to start –  David Aug 24 '12 at 4:17
    
The link to Hideaki's page is broken. I'm not sure if this is the same thing: toyoizumilab.brain.riken.jp/hideaki/res/histogram.html –  DarenW Dec 22 '12 at 0:27
    
Thanks Daren. I updated the link as you suggest. –  Harvey Motulsky Dec 23 '12 at 6:12

Maybe the paper "Variations on the histogram" by Denby and Mallows will be of interest:

This new display which we term "dhist" (for diagonally-cut histogram) preserves the desirable features of both the equal-width hist and the equal-area hist. It will show tall narrow bins like the e-a hist when there are spikes in the data and will show isolated outliers just like the usual histogram.

They also mention that code in R is available on request.

share|improve this answer

I'm not sure this counts as strictly good practise, but I tend to produce more than one histogram with different bin widths and pick the histogram which histgram to use based on which histgram fits the interpretation I'm trying to communicate best. Whilst this introduces some objectivity into the choice of histogram I justify it on the basis I have had much more time to understnad the data than the person I'm giving the histogram to so I need to give them a very concise message.

I'm also a big fan of presenting histograms with the same number of points in each bin rather than the same bin width. I usually find these represent the data far better then the constant bin width although they are mopre difficult to produce.

share|improve this answer
    
sorry, i should have mentioned that i need to do this in an automated way. the option of "doing it multiple times until i find the one that best suits my purpose" won't work for me. has to be done computationally... –  hatorade Jul 27 '10 at 15:34
2  
I agree - the idea that there is one "optimal" bin width is a huge simplifying assumption. –  hadley Mar 19 '11 at 17:32

Did you see the Shimazaki-Shinomoto method?

Although it seems to be computationally expensive, it may give you good results. It's worth giving it a try if computational time is not your problem. There are some implemantations of this method in java, MATLAB, etc, in the website below (Stack doesn't allow me to link you directly), which runs fast enough.

http://176.32.89.45/~hideaki/res/histogram.html

Best of luck!

share|improve this answer

If I need to determine the number of bins programmatically I usually start out with a histogram that has way more bins than needed. Once the histogram is filled I then combine bins until I have enough entries per bin for the method I am using, e.g. if I want to model Poisson-uncertainties in a counting experiment with uncertainties from a normal distribution until I have more than something like 10 entries.

share|improve this answer

I think Sturges' rule can be used for n < 200; where n is the number of observations

share|improve this answer

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.