2
$\begingroup$

I have a data set with a sample size over three million numeric values. Close to 20% are either 0 or 1, with the maximum being nearly 18500. So the data is clearly quite heavily positively skewed.

I am trying to categorize some of this data by putting it into bins of equal width, so I decided to try and find the optimal number of bins. Using the Freedman-Diaconis rule it gave me a value of 126044.0262335108, this is clearly a ridiculously large number of bins for the data.

Breaking the set into the Inter-decile range also proved fruitless giving me [0, 1, 1, 2, 3, 5, 8, 17, 47]

Reading elsewhere the square root of the sample size was suggested, this gave 1732.05081 which is more reasonable. However the method is quite crude.

I also looking into Doane's formula given here. But reading up on this method it seems to have been based on an incorrect hypothesis.

How should I deal with this level of skew in the data?

What is the best way to categorize this data?

$\endgroup$
  • 3
    $\begingroup$ To have an "optimum", you need some goal, so can you say what you are trying to do by binning the data? Also, it may be likely that the data is not so much skewed as "mixed: is the 18500 really in the same (sub-)population as the 0-47? (i.e. you may have outliers rather than a "tail") $\endgroup$ – GeoMatt22 Apr 15 '17 at 16:50
  • 1
    $\begingroup$ I'm not sure that looking into methods for calculating the number of bins for histograms is what you want. If you have millions of values then 126 thousand bins isn't to my mind "clearly ridiculously large". the point of a histogram is usually to approximate a density function, but it doesn't seem like that is what you're trying to do. What do you mean when you say you want to categorize the data? $\endgroup$ – einar Apr 16 '17 at 8:51
  • 3
    $\begingroup$ Why bin at all? It sounds as if your data are zero or positive integers. I would show the frequency of each value as a spike and frequencies on a square root scale. Principle is shown here stats.stackexchange.com/questions/338868/… I've found that this often works well for very skewed distributions. You don't lose sight of the fine structure in the tail and the longer spikes for low values for subdued. $\endgroup$ – Nick Cox Apr 14 '18 at 14:15
1
$\begingroup$

Assuming that your goal is to visualise your data, no binning can allow you to appreciate the distribution in the range 0-47 and the remaining cases up to 18500. Even if you can fit the 0-47 range in a single cm of paper, the maximum (18500) will lie over 3 meters away. If you have a very large support, like a corridor wall, you can follow Nick Cox's comment and draw spikes at every value to convey a sense of the real scale of your data - in fact astronomers and science museums often do so to convey the real scale of the Solar System.

If your histogram needs to fit in a piece of paper or a screen, I would transform the variable as PtrZlnk suggests in his answer. Transformation log10(X+k) could work, and square or cubic root and asinh may work, too.

Once your variable is transformed, you could apply any standard binning strategy to the transformed variable, but don't forget to label your axes in the original variable.

| cite | improve this answer | |
$\endgroup$
0
$\begingroup$

If this is machine learning problem you can just try even 50 different binning strategies and cross-validate which one is the best.

If this is rather hypothesis testing problem think of it in cathegories of statistical certainty about value of predicted variable in bin and choose such a bin size that can quarantee that the value ascribed to bin is choosen with precision high enough. This however require more information about your modelling problem to define the exact algorythm of choosing right bin size.

Another idea is to trying to take logarithm od the variable, thus however would require adding some positive number to the variable. Then it is more likely that relation to dependent variable would linear.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.