1
$\begingroup$

This question already has an answer here:

I frequently work with extremely unbalanced data where I care most about the edge cases. For example, right now, I'm looking at some data with almost 30,000 cases where Score is between 0 and .3 and 5,000 where Score ranges from .05 - over 8.

If my data was normally distributed (or even close), I'd run a histogram to get a feel for what the distribution is like, or maybe a summary, so I can see the quantiles, median and mean. But as it is, the histogram is unreadable since the y scale goes up to 30,000 and I'm mostly interested in the bins that are less than 100 and the minimum through the 3rd quantile are all zero.

Is there a standard technique for high-level exploration of this sort of highly skewed data in the same way that a histogram works? To be clear, I'm most interested in working around the skewness (which sabotages my standard techniques), rather than identifying/quantifying it.

$\endgroup$

marked as duplicate by kjetil b halvorsen, Peter Flom Mar 10 '18 at 14:09

This question has been asked before and already has an answer. If those answers do not fully address your question, please ask a new question.

  • 1
    $\begingroup$ You could try a histogram with log scale on the y-axis? $\endgroup$ – kjetil b halvorsen Mar 16 '17 at 20:37
  • 1
    $\begingroup$ Try transforming the variable with log or power function, or box-cox. $\endgroup$ – Aksakal Mar 16 '17 at 20:40
  • 1
    $\begingroup$ That makes sense...is there a way (outside of practice) to make those transformed values readable so I can immediately apply the insight from the transformed histogram? That's always been my hesitation with log transformed values--I can't immediately say, "oh, I see, there's another real drop when the values get to 6" or whatever. I suppose that sort of mental re-transformation just comes with practice? $\endgroup$ – crazybilly Mar 16 '17 at 20:46
  • 1
    $\begingroup$ If the histograms are severely skewed, I would consider looking at 2D cross-histograms, to check whether skewness is related to joint probabilities $\endgroup$ – Laurent Duval Mar 16 '17 at 23:41