125
$\begingroup$

I'm interested in finding as optimal of a method as I can for determining how many bins I should use in a histogram. My data should range from 30 to 350 objects at most, and in particular I'm trying to apply thresholding (like Otsu's method) where "good" objects, which I should have fewer of and should be more spread out, are separated from "bad" objects, which should be more dense in value. A concrete value would have a score of 1-10 for each object. I'd had 5-10 objects with scores 6-10, and 20-25 objects with scores 1-4. I'd like to find a histogram binning pattern that generally allows something like Otsu's method to threshold off the low scoring objects. However, in the implementation of Otsu's I've seen, the bin size was 256, and often I have many fewer data points that 256, which to me suggests that 256 is not a good bin number. With so few data, what approaches should I take to calculating the number of bins to use?

Improve this question
$\endgroup$
4
  • $\begingroup$ I think Sturges' rule can be used for n < 200; where n is the number of observations $\endgroup$
    – venkasub
    Jan 4, 2011 at 18:57
  • $\begingroup$ It's tacit in several answers but at risk of emphasizing the obvious: there is no optimum to identify without a criterion or criteria to optimize. Histogram design is often fairly epitomized as a trade-off between too much detail and too little detail, but that is still vague. Sometimes all the detail you can show is interesting and useful and indeed vital. If you want a smooth representation of a distribution, especially a continuous one, you perhaps should go straight to density estimation any way. $\endgroup$
    – Nick Cox
    Dec 31, 2021 at 13:31
  • $\begingroup$ what about just the single integer nbins? $\endgroup$
    – Charlie Parker
    Nov 26, 2022 at 22:18
  • $\begingroup$ for a practical answer see: stackoverflow.com/a/74586599/1601580 $\endgroup$
    – Charlie Parker
    Nov 27, 2022 at 0:24

11 Answers 11

Reset to default
153
$\begingroup$

The Freedman-Diaconis rule is very robust and works well in practice. The bin-width is set to $h=2\times\text{IQR}\times n^{-1/3}$. So the number of bins is $(\max-\min)/h$, where $n$ is the number of observations, max is the maximum value and min is the minimum value.

In base R, you can use: 

hist(x, breaks="FD")

For other plotting libraries without this option (e.g., ggplot2), you can calculate binwidth as: 

bw <- 2 * IQR(x) / length(x)^(1/3)

### for example #####
ggplot() + geom_histogram(aes(x), binwidth = bw)
Improve this answer
$\endgroup$
18
  • 3
    $\begingroup$ @nico. The default in R is breaks="Sturges" which does not always give good results. $\endgroup$
    – Rob Hyndman
    Jan 8, 2011 at 21:48
  • 8
    $\begingroup$ How does one calculate IQR? $\endgroup$
    – Kurt Mueller
    Sep 11, 2014 at 23:57
  • 3
    $\begingroup$ @KurtMueller IQR means interquartile range. Look for 1st quartile and 3rd quartile and the difference is IQR. IQR already comes with R so you can use it. $\endgroup$
    – xiaodai
    Oct 15, 2014 at 3:25
  • 4
    $\begingroup$ If I am not mistaken, the answer should read num_bins <- diff(range(x)) / (2 * IQR(x) / length(x)^(1/3)) $\endgroup$
    – Jasha
    Apr 18, 2018 at 3:59
  • 3
    $\begingroup$ nclass.FD did not exist nine years ago. $\endgroup$
    – Rob Hyndman
    Mar 4, 2019 at 20:28
27
$\begingroup$

If you use too few bins, the histogram doesn't really portray the data very well. If you have too many bins, you get a broken comb look, which also doesn't give a sense of the distribution.

One solution is to create a graph that shows every value. Either a dot plot, or a cumulative frequency distribution, which doesn't require any bins.

If you want to create a frequency distribution with equally spaced bins, you need to decide how many bins (or the width of each). The decision clearly depends on the number of values. If you have lots of values, your graph will look better and be more informative if you have lots of bins. This wikipedia page lists several methods for deciding bin width from the number of observations. The simplest method is to set the number of bins equal to the square root of the number of values you are binning.

This page from Hideaki Shimazaki explains an alternative method. It is a bit more complicated to calculate, but seems to do a great job. The top part of the page is a Java app. Scroll past that to see the theory and explanation, then keep scrolling to find links to the papers that explain the method.

Improve this answer
$\endgroup$
5
13
$\begingroup$

Maybe the paper "Variations on the histogram" by Denby and Mallows will be of interest:

This new display which we term "dhist" (for diagonally-cut histogram) preserves the desirable features of both the equal-width hist and the equal-area hist. It will show tall narrow bins like the e-a hist when there are spikes in the data and will show isolated outliers just like the usual histogram.

They also mention that code in R is available on request.

Improve this answer
$\endgroup$
7
$\begingroup$

Did you see the Shimazaki-Shinomoto method?

Although it seems to be computationally expensive, it may give you good results. It's worth giving it a try if computational time is not your problem. There are some implementations of this method in java, MATLAB, etc, in the following link, which runs fast enough: web-interface

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Here is an implementation: gist.github.com/salotz/0158a99a75078b47538452111ec0faa2. And yes its more expensive since you pick a range for the number of bins and you must make a histogram for each and then compute a cost, then pick the most minimal costing one. $\endgroup$
    – salotz
    Jan 9, 2020 at 18:24
6
$\begingroup$

I'm not sure this counts as strictly good practice, but I tend to produce more than one histogram with different bin widths and pick the histogram which histogram to use based on which histogram fits the interpretation I'm trying to communicate best. Whilst this introduces some subjectivity into the choice of histogram I justify it on the basis I have had much more time to understand the data than the person I'm giving the histogram to so I need to give them a very concise message.

I'm also a big fan of presenting histograms with the same number of points in each bin rather than the same bin width. I usually find these represent the data far better then the constant bin width although they are more difficult to produce.

Improve this answer
$\endgroup$
3
  • 1
    $\begingroup$ sorry, i should have mentioned that i need to do this in an automated way. the option of "doing it multiple times until i find the one that best suits my purpose" won't work for me. has to be done computationally... $\endgroup$
    – Tony Stark
    Jul 27, 2010 at 15:34
  • 6
    $\begingroup$ I agree - the idea that there is one "optimal" bin width is a huge simplifying assumption. $\endgroup$
    – hadley
    Mar 19, 2011 at 17:32
  • $\begingroup$ Subjectivity is what we dislike, judgment is what we respect, especially our own. $\endgroup$
    – Nick Cox
    Dec 31, 2021 at 13:18
3
$\begingroup$

If I need to determine the number of bins programmatically I usually start out with a histogram that has way more bins than needed. Once the histogram is filled I then combine bins until I have enough entries per bin for the method I am using, e.g. if I want to model Poisson-uncertainties in a counting experiment with uncertainties from a normal distribution until I have more than something like 10 entries.

Improve this answer
$\endgroup$
2
$\begingroup$

Please see this answer as a complementary of Mr. Rob Hyndman's answer.

In order to create histogram plots with exact same intervals or 'binwidths' using the Freedman–Diaconis rule either with basic R or ggplot2 package, we can use one of the values of hist() function namely breaks. Suppose we want to create a histogram of qsec from mtcars data using the Freedman–Diaconis rule. In basic R we use

x <- mtcars$qsec
hist(x, breaks = "FD")

Meanwhile, in ggplot2 package we use

h <- hist(x, breaks = "FD", plot = FALSE)
qplot(x, geom = "histogram", breaks = h$breaks, fill = I("red"), col = I("white"))

Or, alternatively

ggplot(mtcars, aes(x)) + geom_histogram(breaks = h$breaks, col = "white")

All of them generate histogram plots with exact same intervals and number of bins as intended.

Improve this answer
$\endgroup$
2
$\begingroup$

Conventional wisdom dictates that a "broken look' resulting from a histogram with many bins is undesirable. This clashes with the need to show individual outliers, digit preference, bimodality, data gaps, and other features. I believe that histograms need to be both summary and descriptive measures. For that reason I use either m=100 or 200 bins regardless of the sample size, with modifications to (1) have unequally spaced bins when the number of distinct data values is not huge and (2) to pool such unequally spaced bins when two distinct data values are closer together than, say, 1/5m of the data span. The result is "spike histograms" which I've implemented in many functions in the R Hmisc package. The key algorithm is here in for example the histboxp function.

Many examples are shown here including interactive spike histograms where data values can be viewed in hover text. You'll also see an example where horizontal lines are added underneath the histogram to show various quantiles as in a box blot.

Spike histograms are similar to rug plots, and the human eye is quite good at summarizing distributional shapes from examining tick mark density in the rug. No significant binning is needed.

Improve this answer
$\endgroup$
1
$\begingroup$

With so few data, what approaches should I take to calculating the number of bins to use?

FD or doane methods (see below) might be more suitable. Experience tells, this depends on the upstream task as well. If binning changes the inference or results of the upstream task. Then, one should find a stable/robust method that does not change with changing data.

numpy's manual page on histogram_bin_edges provides nice list and pros and cons of each approach. The 'auto' option follows partially Rob Hyndman's recommendation Here is the full list

  • ‘auto’ Maximum of the ‘sturges’ and ‘fd’ estimators. Provides good all around performance.
  • ‘fd’ (Freedman Diaconis Estimator) Robust (resilient to outliers) estimator that takes into account data variability and data size.
  • ‘doane’ An improved version of Sturges’ estimator that works better with non-normal datasets.
  • ‘scott’ Less robust estimator that that takes into account data variability and data size.
  • ‘stone’ Estimator based on leave-one-out cross-validation estimate of the integrated squared error. Can be regarded as a generalization of Scott’s rule.
  • ‘rice’ Estimator does not take variability into account, only data size. Commonly overestimates number of bins required.
  • ‘sturges’ R’s default method, only accounts for data size. Only optimal for gaussian data and underestimates number of bins for large non-gaussian datasets.
  • ‘sqrt’ Square root (of data size) estimator, used by Excel and other programs for its speed and simplicity.
Improve this answer
$\endgroup$
0
$\begingroup$

Another method is Bayesian Blocks from Studies in Astronomical Time Series Analysis. VI. Bayesian Block Representations by Scargle et al.

Bayesian Blocks is a dynamic histogramming method which optimizes one of several possible fitness functions to determine an optimal binning for data, where the bins are not necessarily uniform width.

Bayesian Blocks for Histograms

Improve this answer
$\endgroup$
0
$\begingroup$

The MDL histogram density estimation method has the following features:

  1. variable width; the method is not constrained to histograms with fixed bin widths.
  2. adaptive; the number of bins, and bin widths are determined based on data. Very few input parameters are required, and the parameters have little impact on the resulting aesthetics.
  3. principled; the resulting histogram is the normalized maximum likelihood distribution (constrained to histograms).
Improve this answer
$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.