0
$\begingroup$

I have a data set with N values residing in the range [-1, 1]. Assume that N is quite large (1,000 to 10,000). I would like to compute a histogram for this data set with the usual binning requirements of not too few bins (at risk of hiding points of interest) and not too many bins (at risk of introducing a lot of noise). Currently, I'd like to use the Freedman-Diaconis rule for determine bin widths: $h=2\times\text{IQR}\times N^{-1/3}$ (where $h$ is the bin width).

When I compute this value, I get quite a large bin size (~600-900 bins). This number of bins is difficult to fit within a histogram (my histogram must be rendered on a webpage).

I think the problem is my domain data is unscaled. My reasoning for this is as such:

Imagine you have two datasets with gaussian distributions. However, assume the standard deviation for the first is 1, while the standard deviation for the second is 10. Still, they would have an identical shape. Presumably you'd want the histograms for each of these to look the same, with the only difference being the x-axis interval labels. The problem with this is that each dataset would have a different IQR, and thus a different recommendation for the bin width according to Freedman-Diaconis.

It seems like I need to scale my dataset to get a reasonable bin width (that captures the distribution well but isn't affected by its scale). What strategy would I use to scale the IQR?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I think the problem is my domain data is unscaled. My reasoning for this is as such:

No. Scaling changes the bin width with the same scale factor, and so does not change the number of bins. The number of bins is range divided by bin width, and scaling affects both the same!

Improve this answer
$\endgroup$
4
  • $\begingroup$ Right, okay that makes sense. So as Freedman-Diaconis produces a very large bin count, is there a method that produces a more reasonable bin count? The chart I'm trying to fit can reasonably fit about 100 bins. I'm not sure how other visualization libraries would handle this. $\endgroup$
    – widavies
    Apr 30, 2021 at 23:20
  • $\begingroup$ Well, if you cannot use more than 100 you have to go with that. Not clear how some other method can help! $\endgroup$
    – kjetil b halvorsen ♦
    Apr 30, 2021 at 23:22
  • $\begingroup$ I guess the question is, is it sometimes more beneficial to use 25 bins or 75 bins? $\endgroup$
    – widavies
    May 1, 2021 at 0:54
  • $\begingroup$ I don't really understand. Beneficial for what? If the problem is technical, why not just show as many bins as you can? Or maybe look into averaged shifted histograms stats.stackexchange.com/questions/51718/… $\endgroup$
    – kjetil b halvorsen ♦
    May 9, 2021 at 1:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.