# Calculating the optimal number of bins for severely skewed data

I have a data set with a sample size over three million numeric values. Close to 20% are either 0 or 1, with the maximum being nearly 18500. So the data is clearly quite heavily positively skewed.

I am trying to categorize some of this data by putting it into bins of equal width, so I decided to try and find the optimal number of bins. Using the Freedman-Diaconis rule it gave me a value of 126044.0262335108, this is clearly a ridiculously large number of bins for the data.

Breaking the set into the Inter-decile range also proved fruitless giving me [0, 1, 1, 2, 3, 5, 8, 17, 47]

Reading elsewhere the square root of the sample size was suggested, this gave 1732.05081 which is more reasonable. However the method is quite crude.

I also looking into Doane's formula given here. But reading up on this method it seems to have been based on an incorrect hypothesis.

How should I deal with this level of skew in the data?

What is the best way to categorize this data?

• To have an "optimum", you need some goal, so can you say what you are trying to do by binning the data? Also, it may be likely that the data is not so much skewed as "mixed: is the 18500 really in the same (sub-)population as the 0-47? (i.e. you may have outliers rather than a "tail") – GeoMatt22 Apr 15 '17 at 16:50
• I'm not sure that looking into methods for calculating the number of bins for histograms is what you want. If you have millions of values then 126 thousand bins isn't to my mind "clearly ridiculously large". the point of a histogram is usually to approximate a density function, but it doesn't seem like that is what you're trying to do. What do you mean when you say you want to categorize the data? – einar Apr 16 '17 at 8:51
• Why bin at all? It sounds as if your data are zero or positive integers. I would show the frequency of each value as a spike and frequencies on a square root scale. Principle is shown here stats.stackexchange.com/questions/338868/… I've found that this often works well for very skewed distributions. You don't lose sight of the fine structure in the tail and the longer spikes for low values for subdued. – Nick Cox Apr 14 '18 at 14:15