# Information loss by histograms

Recently I became curious about what I imagine to be an old problem: the fidelity of histograms to an underlying data set. CrossValidated has a number of questions on the subject of "optimal histograms" such as:

Calculating optimal number of bins in a histogram

But the answers looked more like heuristics than metrics for quantifying the information that is lost from summarizing data. Can anyone point me in the direction of theoretical treatments (or algorithms) of this problem? As an example of the the type of answer I am looking for, I submitted an answer below.

• You might find example (3) in this answer interesting. Commented Sep 25, 2017 at 1:36
• I had an example in my practice where I analyzed (in a simulation) both a continuous underlying trait and its binned version. Both of them would go into an input to a correlation calculation, and the standard error on the correlation would act as a measure of information, so the increase in the standard error with the categorical version would demonstrate the information loss. (My recollection is that I binned a standard normal to 3 or 4 bins, and that led to almost exactly 50% loss.) If that's relevant, I can dig that out and post as an answer. Commented Sep 25, 2017 at 4:48
• @StasK, thanks for the comment. If it is not too much work to dig out, I would be interested in the approach you took. Commented Sep 25, 2017 at 22:13
• Maybe kernel density estimators will work better: no bins, no headaches Commented Oct 2, 2017 at 17:41

I had a chance to examine the faithful data set in R, to score some plausible metrics: a two-sample Kolmogorov-Smirnov test (distance and p-value), Kullback–Leibler divergence, and Mutual Information for a range of different numbers of bins.

## An overview of the data:

The duration between Old Faithful geyser eruptions can be plotted as a histogram, but what is the trade off of #bins vs. fidelity to the data set?

A 4 bin histogram shows too much averaging

A 14 bin histograms shows the two main modes

And a 54 bin histogram shows all of the data at the measurement accuracy:

## Results of different metrics:

Using the metrics of information loss we can plot how they change with an increasing number of bins:

Here, the KS metrics remain mostly unchanged above 27 bins, KL Divergence (in my implementation) is a useless metric, but mutual information best captures the increasing accuracy with increasing bin count.

Thoughts?

Teh codez:

library(entropy)
library(ggplot2)
library(dplyr)
library(reshape2)
data("faithful")
duration <- faithful$waiting xrange <- seq(min(duration),max(duration),by=1) h.all <- hist(duration,c(xrange[1]-0.5,xrange+0.5)) metrics <- function(nbins){ h <- hist(duration,seq(min(duration)-0.5*(max(duration)-min(duration)+1)/nbins,max(duration)+0.5*(max(duration)-min(duration)+1)/nbins,length.out=1+nbins)) # must force bin breakpoints x <- unlist(mapply(FUN=function(m,c){rep(m,c)},h$mids,h$counts)) # KS test? ks <- ks.test(x,duration,alternative='two.sided',exact=FALSE) # KL divergence? KLD <- KL.empirical(x,duration) # Mutual information? bin.ix.orig <- findInterval(duration,h.all$breaks,rightmost.closed = TRUE)
bin.ix.new <- findInterval(duration,h$breaks,rightmost.closed = TRUE) y2d <- table(data.frame(bin.ix.new,bin.ix.orig)) MI <- mi.empirical(y2d) return(data.frame(KS.distance=as.numeric(ks$statistic),
KS.p.value= ks$p.value, KL.Divergence = KLD, Mutual.Info = MI)) } df.sweep <- data.frame(nbins=seq(3,60)) %>% group_by(nbins) %>% do(metrics(.$nbins)) %>% ungroup()
ggplot(melt(df.sweep,id.vars='nbins'),aes(x=nbins,y=value))+geom_point()+facet_wrap(~variable,scales='free_y')


The optimal method to generate a histogram is to ditch it entirely and use a KDE instead.

Histograms are an anachronism, and with the advent of easy software to generate kernel density estimators there is no longer any reason to ever use them. Thus, in the practical sense, the optimal method to generate a histogram is to ditch it entirely and use a KDE instead. With the latter method, the problem of bin selection is not required. There is a bandwidth parameter that is estimated from the data (e.g., via MLE) and so the method does not depend on any arbitrary choice by the user. For this reason, asking for the optimal selection of bins in a histogram is a bit like asking for the optimal horse-shoe for racing in the Daytona 500.

• I do like KDE myself, but a quick scan around CV suggests that there are multiple rules that are followed for bandwidth selection (e.g. stats.stackexchange.com/questions/194792/… and stats.stackexchange.com/questions/168/…. I don't know how the area well, but wouldn't that suggest that there's still some arbitrariness in choosing which rule to follow?
– mkt
Commented Aug 5, 2019 at 13:31
• I would like to suggest the statement about "anachronism" may be too strong. KDEs have their own problems. In particular, as illustrated at that link, they may provide misleading information about tail behavior, demonstrating that any "optimality" depends on the application. There are places where histograms are ideal: for instance, they lend themselves nicely to chi-squared goodness-of-fit testing. Histograms also make a nice pedagogical bridge from summarizing frequencies to density functions.
– whuber
Commented Aug 5, 2019 at 14:15
• @mkt: All it really means is that there are multiple estimation methods for parameters (which is try in other contexts too). The fact that the data supplies a basis to estimate the bandwidth parameter means that you are not required to arbitrarily choose a parameter value for the KDE. You are still required to choose an estimation method, just as in any other statistical inference problem.
– Ben
Commented Aug 6, 2019 at 1:16
• @whuber: The problem of bad tail behaviour can usually be dealt with by choosing an appropriate kernel that respects the support of the distribution. In the post you link to, the problem is that a lower-truncated distribution is estimated by a KDE using the Gaussian kernel. I guess that could be considered a "problem" with KDEs, but I think of it more as a problem with the kernel selection.
– Ben
Commented Aug 6, 2019 at 1:18
• But that beautifully illustrates the problem, Ben: the distribution is not "truncated;" it merely has positive support. That example shows you usually cannot deal with it by choosing an "appropriate kernel." Indeed, the tail behavior of the kernel determines the tail behavior of the KDE, which may be quite different from the tail behavior of the underlying distribution. Arguably histograms suffer from the same problem (no tails at all!)--but at least they usually expose the issue better than a KDE, which often looks deceptively nice.
– whuber
Commented Aug 6, 2019 at 12:06

I gave a talk a few years ago looking at estimation of polychoric correlations (which few statisticians have heard about -- these are MLE estimates of correlations underlying two ordinal variables, assuming that these underlying variables come from a bivariate normal distribution, and that the observed ordinal variables are a result of coarsening the underlying normal variables). I was simulating some data from such bivairate distribution (slides 7--9) to demonstrate the point, with two two variables categorized in three and four categories, respectively, which is way fewer than what a histogram would typically give you (but is typical with social science data response scales). On slide 15 of, I report estimation results: the standard error of the correlation between the underlying normal variables was 0.0146, while the standard error of the polychoric correlations between the ordinal variables was 0.0207. They estimate the same quantity, so the difference/ratio of the standard errors is the degree of information loss, which is pretty much exactly 50%.