103
$\begingroup$

I'm interested in finding as optimal of a method as I can for determining how many bins I should use in a histogram. My data should range from 30 to 350 objects at most, and in particular I'm trying to apply thresholding (like Otsu's method) where "good" objects, which I should have fewer of and should be more spread out, are separated from "bad" objects, which should be more dense in value. A concrete value would have a score of 1-10 for each object. I'd had 5-10 objects with scores 6-10, and 20-25 objects with scores 1-4. I'd like to find a histogram binning pattern that generally allows something like Otsu's method to threshold off the low scoring objects. However, in the implementation of Otsu's I've seen, the bin size was 256, and often I have many fewer data points that 256, which to me suggests that 256 is not a good bin number. With so few data, what approaches should I take to calculating the number of bins to use?

$\endgroup$
1
  • $\begingroup$ I think Sturges' rule can be used for n < 200; where n is the number of observations $\endgroup$ – venkasub Jan 4 '11 at 18:57
125
$\begingroup$

The Freedman-Diaconis rule is very robust and works well in practice. The bin-width is set to $h=2\times\text{IQR}\times n^{-1/3}$. So the number of bins is $(\max-\min)/h$, where $n$ is the number of observations, max is the maximum value and min is the minimum value.

In base R, you can use:

hist(x, breaks="FD")

For other plotting libraries without this option (e.g., ggplot2), you can calculate binwidth as:

bw <- 2 * IQR(x) / length(x)^(1/3)

### for example #####
ggplot() + geom_histogram(aes(x), binwidth = bw)
$\endgroup$
16
  • 2
    $\begingroup$ @nico. The default in R is breaks="Sturges" which does not always give good results. $\endgroup$ – Rob Hyndman Jan 8 '11 at 21:48
  • 5
    $\begingroup$ How does one calculate IQR? $\endgroup$ – Kurt Mueller Sep 11 '14 at 23:57
  • 1
    $\begingroup$ @KurtMueller IQR means interquartile range. Look for 1st quartile and 3rd quartile and the difference is IQR. IQR already comes with R so you can use it. $\endgroup$ – xiaodai Oct 15 '14 at 3:25
  • 4
    $\begingroup$ If I am not mistaken, the answer should read num_bins <- diff(range(x)) / (2 * IQR(x) / length(x)^(1/3)) $\endgroup$ – Jasha Apr 18 '18 at 3:59
  • 2
    $\begingroup$ nclass.FD did not exist nine years ago. $\endgroup$ – Rob Hyndman Mar 4 '19 at 20:28
22
$\begingroup$

If you use too few bins, the histogram doesn't really portray the data very well. If you have too many bins, you get a broken comb look, which also doesn't give a sense of the distribution.

One solution is to create a graph that shows every value. Either a dot plot, or a cumulative frequency distribution, which doesn't require any bins.

If you want to create a frequency distribution with equally spaced bins, you need to decide how many bins (or the width of each). The decision clearly depends on the number of values. If you have lots of values, your graph will look better and be more informative if you have lots of bins. This wikipedia page lists several methods for deciding bin width from the number of observations. The simplest method is to set the number of bins equal to the square root of the number of values you are binning.

This page from Hideaki Shimazaki explains an alternative method. It is a bit more complicated to calculate, but seems to do a great job. The top part of the page is a Java app. Scroll past that to see the theory and explanation, then keep scrolling to find links to the papers that explain the method.

$\endgroup$
2
11
$\begingroup$

Maybe the paper "Variations on the histogram" by Denby and Mallows will be of interest:

This new display which we term "dhist" (for diagonally-cut histogram) preserves the desirable features of both the equal-width hist and the equal-area hist. It will show tall narrow bins like the e-a hist when there are spikes in the data and will show isolated outliers just like the usual histogram.

They also mention that code in R is available on request.

$\endgroup$
7
$\begingroup$

Did you see the Shimazaki-Shinomoto method?

Although it seems to be computationally expensive, it may give you good results. It's worth giving it a try if computational time is not your problem. There are some implementations of this method in java, MATLAB, etc, in the following link, which runs fast enough: web-interface

$\endgroup$
1
  • $\begingroup$ Here is an implementation: gist.github.com/salotz/0158a99a75078b47538452111ec0faa2. And yes its more expensive since you pick a range for the number of bins and you must make a histogram for each and then compute a cost, then pick the most minimal costing one. $\endgroup$ – salotz Jan 9 '20 at 18:24
6
$\begingroup$

I'm not sure this counts as strictly good practice, but I tend to produce more than one histogram with different bin widths and pick the histogram which histogram to use based on which histogram fits the interpretation I'm trying to communicate best. Whilst this introduces some subjectivity into the choice of histogram I justify it on the basis I have had much more time to understand the data than the person I'm giving the histogram to so I need to give them a very concise message.

I'm also a big fan of presenting histograms with the same number of points in each bin rather than the same bin width. I usually find these represent the data far better then the constant bin width although they are more difficult to produce.

$\endgroup$
2
  • 1
    $\begingroup$ sorry, i should have mentioned that i need to do this in an automated way. the option of "doing it multiple times until i find the one that best suits my purpose" won't work for me. has to be done computationally... $\endgroup$ – Tony Stark Jul 27 '10 at 15:34
  • 5
    $\begingroup$ I agree - the idea that there is one "optimal" bin width is a huge simplifying assumption. $\endgroup$ – hadley Mar 19 '11 at 17:32
3
$\begingroup$

If I need to determine the number of bins programmatically I usually start out with a histogram that has way more bins than needed. Once the histogram is filled I then combine bins until I have enough entries per bin for the method I am using, e.g. if I want to model Poisson-uncertainties in a counting experiment with uncertainties from a normal distribution until I have more than something like 10 entries.

$\endgroup$
2
$\begingroup$

Please see this answer as a complementary of Mr. Rob Hyndman's answer.

In order to create histogram plots with exact same intervals or 'binwidths' using the Freedman–Diaconis rule either with basic R or ggplot2 package, we can use one of the values of hist() function namely breaks. Suppose we want to create a histogram of qsec from mtcars data using the Freedman–Diaconis rule. In basic R we use

x <- mtcars$qsec
hist(x, breaks = "FD")

Meanwhile, in ggplot2 package we use

h <- hist(x, breaks = "FD", plot = FALSE)
qplot(x, geom = "histogram", breaks = h$breaks, fill = I("red"), col = I("white"))

Or, alternatively

ggplot(mtcars, aes(x)) + geom_histogram(breaks = h$breaks, col = "white")

All of them generate histogram plots with exact same intervals and number of bins as intended.

$\endgroup$
0
$\begingroup$

Another method is Bayesian Blocks from Studies in Astronomical Time Series Analysis. VI. Bayesian Block Representations by Scargle et al.

Bayesian Blocks is a dynamic histogramming method which optimizes one of several possible fitness functions to determine an optimal binning for data, where the bins are not necessarily uniform width.

Bayesian Blocks for Histograms

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.