Tell me more ×
Cross Validated is a question and answer site for statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
up vote 2 down vote favorite

I am trying to find out answer for this simple question. I have data (list of numbers). I have 20000 numbers. smallest number is 100 and largest is 15000. I want to interpret this data, how it is distributed or any other suggestions, that how can I play with this data and do different things?

The thing is I have to give a presentation on summary stats on this data and also I want to include plots of this data in my presentation so that I can easily show to everyone, what data looks like, to tell them, which is largest number, smallest, which range of numbers are occurring more.

So I have made histogram in R but for so many numbers, histogram is not a good choice. Also I know how to calculate all this mean, median and standard deviation. This is not a homework, I want to give presentation about the data, and I am not a statistician.

share|improve this question
1  
is this homework? if it is please tag as such. Please let us know what you have tried as well. – RJ- Mar 27 '12 at 12:34
You're only concerned with one variable? You should say "visualize and interpret many cases of a single variable" or "Univariate data summary" because "large set of data" usually implies many variables as well as cases. – Michael Bishop Mar 27 '12 at 16:58
I think a histogram is a better choice w/ lots of data than w/ little; ie, histograms become better as the amount of data increases. – gung Aug 16 '12 at 16:51

4 Answers

up vote 3 down vote

A lot of the expenditure data I work with has a similar distribution to this. Depending on what you need to do we often find it useful to plot a histogram of the logarithm of the data. This can be done by transforming the variable and plotting that, or plotting the original data on a logarithmic axis.

This method can be useful for having another think about your outliers - are they really exceptional cases, or are they just the natural high values you'd expect with what an underlying distribution that is quite regular eg approximately log normal.

For most audiences, it would be a good idea to present the original as well, as otherwise they can easily otherwise walk away with the incorrect idea that your data is symmetrically distributed when it is far from it.

Then, a quantile-quantile plot is the best way of comparing the distribution of your variable to some reference distribution. A qq-plot comparing the log of our "spend" variable compared to a normal distribution is included below. This shows that our distribution isn't actually log-normal, but for a reasonable amount of its range, including the higher values, it is a reasonable approximation. Interpreting qq-plots needs a bit of practice but there are plenty of explanations out there.

eg

library(ggplot2)
qplot(TotalSpend) # this is just a histogram
qplot(TotalSpend, log="x")
qplot(sample=log(TotalSpend), stat="qq") # or qqnorm() if you prefer

enter image description here

share|improve this answer
+1, there's a lot of good information here. One question, in trying to think through this, is there / what is the substantive difference b/t a hist of ln(data) & a hist of data plotted on ln(x)? Also, re: difficulty interpreting qq-plots, I like to pair them w/ a plot of the kernel density & possibly the theoretical dist. – gung Aug 16 '12 at 16:56
up vote 2 down vote

A histogram is a fine choice when you have lots of data. I'm guessing your problems are that with a histogram:

  1. your high outliers won't be noticed.

  2. the visual is too small because the x-axis is stretched long enough to capture the high outliers.

Solve problem 1 with a note below the graph: "There are large outliers which are not visible."

Solve problem 2 by reducing the range, something like: hist(x, xlim=range(0:2000))

@chl pointed out that my description was a bit unclear so I've edited it lightly. He/she also points out that you'll probably need to adjust other parameters to make a good looking histogram, e.g. "breaks" which affects bin size. Of course there are other packages you could use for your plotting. lattice and ggplot2 are two popular alternatives to the base R graphics.

In addition to calculating the standard deviation, you might use a function like describe from package Hmisc. It provides output like this:

ih$bweightoz : birthweight in ounces 
      n missing  unique    Mean     .05     .10     .25     .50 
  16548    4197     136   115.4    75.0    86.7   103.0   117.0 
    .75     .90     .95 
  129.0   142.0   151.0 

lowest :  48.01  64.00  64.01  65.00  66.00
highest: 187.00 188.00 190.00 191.00 192.01
share|improve this answer
I don't understand Point 2: hist will automatically set x-axis limits to fit the observed range of the data. Do you instead mean varying bin size or number? – chl♦ Mar 27 '12 at 20:08
@chl, I think this is one of those rare occasions where I wouldn't want the x-axis limits to cover the entire range of the data. He has outliers which are so far from the rest of his data (he said mean=500, max=15,000) that showing the entire range will make it hard to see the bulk of the distribution. You are right that he may need to vary bin size or number as well. – Michael Bishop Mar 28 '12 at 0:01
up vote 1 down vote

A box plot shows the summary statistics. Try a box plot.

share|improve this answer
Boxplots are particularly useful when you want to compare the distribution of your variable across two sub-groups (ex: in women vs in men). You then have several boxplots side-by-side, which is more easy to interpret visually than having several histograms. – dominic999 Mar 27 '12 at 13:11
Welcome to this site, Naomi. Sorry, I left this post notice to your intention without explaining why (I was in a hurry...). It would be good if you could explain why you would recommend using a boxplot rather than, say, an histogram or a density plot. – chl♦ Mar 27 '12 at 20:03
1  
Boxplots, histograms and density plots are all useful for examining the distribution of data. It often is useful to plot data more than one way. I meant to try box plots in addition to rather than instead of the other ways suggested. The outliers will still be a problem. Have the outliers been checked for reasonableness? I have seen cases where the outliers turned out to be copying errors or other erroneous data. – Naomi B. Robbins Mar 28 '12 at 0:36
Thanks, Naomi, for the added clarifications. (You can edit your answer instead of adding comment--this should work even if you haven't registered your account yet.) – chl♦ Mar 28 '12 at 12:08
up vote 1 down vote

A histogram would do exactly that. For descriptive stats, the mean, standard deviation and skewness, as well as the IQR are the most commonly used. Depending on the type of distribution, some of those become more or less relevant -- a mean isn't really representative in a highly skewed distribution, for instance. You would probably prefer the median, which is less sensitive to extremes.

share|improve this answer
Thanks.As you said "Depending on the type of distribution" How can I know the distribution of my data? – Rajiv Mar 27 '12 at 12:44
By looking at the histogram. If you get a near symmetric bell curve, then the mean is a good representative measure. If you have a skewed or a zero-inflated distribution, then the mean is less relevant. – dominic999 Mar 27 '12 at 13:08
In histogram some values are very far away. Like I have some numbers equal to 15000 and 20000 and most of them are equal to 500, my mean is 501. So the histogram is not clearly visible. – Rajiv Mar 27 '12 at 13:14
So you have a skewed distribution... Difficult to help without seeing the histogram though. You could try to change the limits of your x axis to see just the main part of the curve (or, if it's easier technically, just exclude temporarily the points which are above a certain threshold, then creating a new histogram). – dominic999 Mar 27 '12 at 13:46
Yes I can change the limits of x axis, but I need to show this plot (Or some other plot) in presentation. So that I can explain easily as histogram is not clear for large values. – Rajiv Mar 27 '12 at 13:49
show 1 more comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.