# My data are very skew and I can't see any detail in a histogram. How do I see the shape?

I have a vector with income values of all companies that I found (n=1821). The income should look like a lognormal distribution, but if I use the hist function in R (RStudio) the result is this:

As you can see there are many values appear to be near 0, that's because lots of income values are 0 and many are quite small, and then there are a few incomes (about 10) with very high values.

What should I do to more clearly show the shape with a histogram?

Should I delete all 0 values from the vector?

I don't know how to proceed.

• In my university we have a huge database with all companies and their income per year, in the income vector there are lots of 0 and the values are all near the 0 value cause income variable is expressed with millions, there are also some low value and only 10 very high values, those companies are from the clothes industry so those 10 companies are dolce and gabbana gucci and very few others, but there are tons of companies that are very low with their income, this vector has 1800 values and seems hard for r putting them in a graph, my professor wants that I put those values in an histogram. – Jacopo Soppelsa Aug 1 '16 at 22:05
• @Glen_b look my comment and this one, I have to show that my data distribute like a log normal cause my professor said income data distribution is lognormal I tried so many times but as you can see the histogram is horrible and I don't know how to show the real distribution, maybe the problems are the class for the variable. I think I'm going to cut all 0 for what u said, and then how can the distribution look like a lognormal? – Jacopo Soppelsa Aug 1 '16 at 22:07
• @Glen_b but If I do a log transformation from.the data the histogram will look like a normal, but I want the data distributed like a lognormal, there is something I can do without the log transformation? Thanks a lot for the help, you don't know how im looking for it... – Jacopo Soppelsa Aug 1 '16 at 22:09
• @Glen_b I'm going to cut all 0 values now and try again, but if there are too many low value and very few high value like 2 million income for one company and 1000000 for another one the graph can't show this, is there something that can for instance say : "the last class is composed by more than xxx values? Like more than 5000 million for the last class in the right – Jacopo Soppelsa Aug 1 '16 at 22:13
• @Glen_b thanks a lot for these precious info, for wide distribution I mean there are too many values.and the graph can't deal with them, in fact the result is terrible and doesn't show anything, I added more bins in the graph but the situation is still the same. The problem is that my professor is gone for one month and I have to work on this alone but I find this very difficult, tomorrow I will read all your comment and try to do something, cutting all 0 values isn't cheating but I think I need to cut them cause they doesn't have a meaning in this database... – Jacopo Soppelsa Aug 1 '16 at 22:29

## 1 Answer

With or without exact zeros a histogram of a very skew distribution can look like this. It has nothing to do with the spread, nor with the existence of zeros, but with how far above the bulk of the data the largest observation is.

You're dealing with two different problems at once here --

$\:$ a. The distribution is very skew.

$\:$ b. The default number of bins in R is a good deal too low for seeing the shape well in general and much too low when the distribution is skew or has heavy tails.

The second problem is easy to deal with -- use far more bins.

Obtaining a good display with very skewed data is not a single step process. It may take several attempts and some decision-making about how to best represent the information

If the data were actually lognormal with a large $\sigma$ parameter (and so very skew), it can look just like your plot. Here I generated some data (in "x", with n=1800) which has a particular lognormal distribution:

I made that plot have about twice as many bins that you got from the default, but it still didn't show any detail. How skewed is this? Well, the sample mean is about 8000 times as big as the median. Those large values dominate more than just the plot. (I made a second data set that's somewhat less skew, to show the potential value of some of the options I suggest)

The most obvious thing to do would be to plot on a log scale:

Note that the axis labels are in original values not log-currency, just as you'd get with plot(...,log="x") (in fact that's what I used to make this plot, after extracting the results of hist by putting it into a variable); I also added some detail on the x-axis but this isn't really necessary.

If you have exact zeros this approach of plotting on a log-scale is obviously not suitable for them as-is, since you can't take log of exact 0's (which are impossible in a lognormal; clearly you don't a lognormal).

How you might deal with them depends on what you're trying to do with the variable and how many there are.
(What's the actual proportion of exact zeros? You didn't say)

Anyway, here's the simplest step in the process of trying to find a suitable display:

1. try a histogram with a lot of bins - at least a hundred. And plot in a bigger window.

This can sometimes help a lot but if your data is pretty skew, it won't solve the problem:

That may not work (it didn't for my most skewed sample there), so what else is there?

2. Cut off the largest values and list them on the plot, or do two displays, one with the top end cut off and one showing the larger values (equivalently, on one display, show a complete scale break and plot the two parts on two different x-scales). Something a bit like this:

This can often help a lot but if your data is really skewed, it won't solve the problem:

This is about the best plot possible with that data -- you really can't get more detail on the right without losing it all on the left.

3. cut off the exact zeros, show a count of them and plot on the log-scale (but with original currency-scale tick-labels) as in my second diagram above

4. Consider a transformation that can manage zeros, such as cube-root, but still show currency on the axis. (This would involve writing some R-code, so it may be too hard for you at this point. I don't really suggest it in this case, since people are much more used to seeing financial variables like income either on the log-scale, or on the original currency-scale.)

Since someone is bound to ask, here's how I did the log-scale plot (absent additional fiddling with axis ticks):

res <- hist(log(x),n=30)
with(res,plot(counts~exp(mids),type="h",lwd=10,col=8,log="x",lend=2,xlab="income"))


(You need lwd wide enough that the bars just touch. You need lend to make the cars have square ends. You can make a version of "hist" to do this but it takes more work.)