Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

I'm interested in finding as optimal of a method as I can for determining how many bins I should use in a histogram. My data should range from 30 to 350 objects at most, and in particular I'm trying to apply thresholding (like Otsu's method) where "good" objects, which I should have fewer of and should be more spread out, are separated from "bad" objects, which should be more dense in value. A concrete value would have a score of 1-10 for each object. I'd had 5-10 objects with scores 6-10, and 20-25 objects with scores 1-4. I'd like to find a histogram binning pattern that generally allows something like Otsu's method to threshold off the low scoring objects. However, in the implementation of Otsu's I've seen, the bin size was 256, and often I have many fewer data points that 256, which to me suggests that 256 is not a good bin number. With so few data, what approaches should I take to calculating the number of bins to use?

share|improve this question
    
I think Sturges' rule can be used for n < 200; where n is the number of observations –  venkasub Jan 4 '11 at 18:57

7 Answers 7

up vote 25 down vote accepted

The Freedman-Diaconis rule is very robust and works well in practice. The bin-width is set to $h=2*\text{IQR}*n^{-1/3}$. So the number of bins is (max-min)/$h$.

In base R, you can use hist(x,breaks="FD")

For other plotting libraries without this option (e.g. ggplot2), you can calculate binwidth as:

bw <- diff(range(x)) / (2 * IQR(x) / length(x)^(1/3)))

### for example
ggplot() + geom_histogram(aes(x), binwidth = bw)
share|improve this answer
    
As a note, by default (so, if you don't specify breaks) R –  nico Jan 7 '11 at 21:36
    
@nico. The default in R is breaks="Sturges" which does not always give good results. –  Rob Hyndman Jan 8 '11 at 21:48
    
for whatever reason my comment was truncated... I meant to write "by default (so, if you don't specify breaks) R uses the Sturges algorithm"... odd! –  nico Jan 9 '11 at 0:15
    
How does one calculate IQR? –  Kurt Mueller Sep 11 '14 at 23:57
    
@KurtMueller IQR means interquartile range. Look for 1st quartile and 3rd quartile and the difference is IQR. IQR already comes with R so you can use it. –  xiaodai Oct 15 '14 at 3:25

If you use too few bins, the histogram doesn't really portray the data very well. If you have too many bins, you get a broken comb look, which also doesn't give a sense of the distribution.

One solution is to create a graph that shows every value. Either a dot plot, or a cumulative frequency distribution, which doesn't require any bins.

If you want to create a frequency distribution with equally spaced bins, you need to decide how many bins (or the width of each). The decision clearly depends on the number of values. If you have lots of values, your graph will look better and be more informative if you have lots of bins. This wikipedia page lists several methods for deciding bin width from the number of observations. The simplest method is to set the number of bins equal to the square root of the number of values you are binning.

This page from Hideaki Shimazaki explains an alternative method. It is a bit more complicated to calculate, but seems to do a great job. The top part of the page is a Java app. Scroll past that to see the theory and explanation, then keep scrolling to find links to the papers that explain the method.

share|improve this answer
1  
+1 $\sqrt{n}$ is a good place to start –  David Aug 24 '12 at 4:17
    
The link to Hideaki's page is broken. I'm not sure if this is the same thing: toyoizumilab.brain.riken.jp/hideaki/res/histogram.html –  DarenW Dec 22 '12 at 0:27
    
Thanks Daren. I updated the link as you suggest. –  Harvey Motulsky Dec 23 '12 at 6:12

Maybe the paper "Variations on the histogram" by Denby and Mallows will be of interest:

This new display which we term "dhist" (for diagonally-cut histogram) preserves the desirable features of both the equal-width hist and the equal-area hist. It will show tall narrow bins like the e-a hist when there are spikes in the data and will show isolated outliers just like the usual histogram.

They also mention that code in R is available on request.

share|improve this answer

I'm not sure this counts as strictly good practise, but I tend to produce more than one histogram with different bin widths and pick the histogram which histgram to use based on which histgram fits the interpretation I'm trying to communicate best. Whilst this introduces some objectivity into the choice of histogram I justify it on the basis I have had much more time to understnad the data than the person I'm giving the histogram to so I need to give them a very concise message.

I'm also a big fan of presenting histograms with the same number of points in each bin rather than the same bin width. I usually find these represent the data far better then the constant bin width although they are mopre difficult to produce.

share|improve this answer
    
sorry, i should have mentioned that i need to do this in an automated way. the option of "doing it multiple times until i find the one that best suits my purpose" won't work for me. has to be done computationally... –  sepiroth Jul 27 '10 at 15:34
2  
I agree - the idea that there is one "optimal" bin width is a huge simplifying assumption. –  hadley Mar 19 '11 at 17:32

If I need to determine the number of bins programmatically I usually start out with a histogram that has way more bins than needed. Once the histogram is filled I then combine bins until I have enough entries per bin for the method I am using, e.g. if I want to model Poisson-uncertainties in a counting experiment with uncertainties from a normal distribution until I have more than something like 10 entries.

share|improve this answer

Did you see the Shimazaki-Shinomoto method?

Although it seems to be computationally expensive, it may give you good results. It's worth giving it a try if computational time is not your problem. There are some implemantations of this method in java, MATLAB, etc, in the website below (Stack doesn't allow me to link you directly), which runs fast enough.

http://176.32.89.45/~hideaki/res/histogram.html

Best of luck!

share|improve this answer

I have 600 observations for Au g/t. Bin size 1 gives me this:enter image description here

Automatic selection (omit the bin range) gives this:enter image description here

The data looks O'K on the first and second graphs, as if there is no problem with data integrity. Only bin size 0.1 (g/t) answers the question: the measurements were both inaccurate and impreciseenter image description here

My judgement: 1. There is no measurement technique on Earth to show up the true value of natural phenomenon. All measurements are approximate, some being close to the true value. It depends on the sampling design, calibration, human qualifications, etc. 2.This is why distribution is skewed rather than symmetrical. 3.Nevertheless, the shape of the distribution should resemble a "bell-like" section, at least approximately. One bell at a time (unless there are several geological environments). 4.Frequency distribution with the bin size manipulation can help to reveal pattern on how accurate and precise had the measurements been done. So that one needs an experimental pick up of the bin size rather than a rule cut on stone.

share|improve this answer
1  
This is more a comment than an answer unless you spell out the moral. I'd say the moral is this: All rules are rules of thumb, some have more statistical basis than others, but most rules may not do what you wish if distributions have very high skewness or kurtosis. So, use your judgment too. –  Nick Cox Feb 19 at 9:04
2  
You have added comments, but I am not clear that any contribute helpful new details to the thread. 1. is fine by me but not the issue here. 2. doesn't follow from 1. as some distributions are nearly symmetrical. 3. is dubious: there are many situations where shapes other than bells are expected. 4. is also dubious as the quality of the original measurements is often not evident from any histogram, but often best examined with attention to the very fine structure of a distribution. –  Nick Cox Feb 19 at 10:06
    
2.Nearly symmetrical is not symmetrical. You can not be nearly pregnant: either pregnant or not. –  Sergo Cusiani Feb 19 at 10:53
1  
Sure, but exact symmetry in data is so very rare that if I had said "symmetrical" someone might reasonably object to that too. It's very easily to be nearly symmetrical; that is why and how we have measures of skewness. –  Nick Cox Feb 19 at 10:58
1  
Noted, but your comment was general and lacked the context you now give. In turn I said "dubious" not "wrong". My major comment remains that your answer doesn't really add much to the thread on how to choose bin size or equivalently number of bins. Having another example in which it's difficult is a spin on the question, not really an answer. –  Nick Cox Feb 19 at 11:14

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.