1
$\begingroup$

I have a large dataset (> 100,000 rows) of ecological data. In some of my first attempts to visualize the data, I used bar plots with calculated means and error bars (see below plots). My go-to for error bars is usually the standard-error, and this first round of plots had uniformly small error bars. One of my colleagues suggested that since my n was so large, I would always have a small SE, and that SD is a better metric to characterize variance with large datasets.

Following her advice, I created the same plots as above, only this time using the SD for my error bars, and now I have huge error bars; larger than the means themselves. It seems like SE creates error bars that are too small to be a useful indicator of variance, and SD creates error bars that are too big. I'm not sure if the data is just too noisy to work with, or I don't have a good way to begin to visualize the variance with this large dataset.

As an example

library(plotrix)
library (ggplot2)

A<-rnorm(100, mean=5, sd=5)
B<-rnorm(100, mean=2, sd=2)
C<-rnorm(100, mean=4, sd=6)
Group<-rep(1:5, 20)

SmallData<-data.frame(A,B,C,Group)


Smallmean<-aggregate(A ~ Group, data=SmallData, FUN=mean)
Smallsd<-aggregate(A ~ Group, data=SmallData, FUN=sd)
Smallse<-aggregate(A ~ Group, data=SmallData, FUN=std.error)

Smallmean$SD<-Smallsd$A
Smallmean$SE<-Smallse$A


# With a small n, both SD and SE are similar 
ggplot(Smallmean, aes(x=Group, y=A)) + 
  geom_bar(stat="identity", color="black", 
           position=position_dodge()) +
  geom_errorbar(aes(ymin=A-SD, ymax=A+SD), width=.2,
                position=position_dodge(.9)) 



ggplot(Smallmean, aes(x=Group, y=A)) + 
  geom_bar(stat="identity", color="black", 
           position=position_dodge()) +
  geom_errorbar(aes(ymin=A-SE, ymax=A+SE), width=.2,
                position=position_dodge(.9)) 





#Much larger n and SD
A<-rnorm(1000000, mean=5, sd=200)
B<-rnorm(1000000, mean=2, sd=2)
C<-rnorm(1000000, mean=4, sd=6)
Group<-rep(1:5, 200000)

BigData<-data.frame(A,B,C,Group)


Bigmean<-aggregate(A ~ Group, data=BigData, FUN=mean)
Bigsd<-aggregate(A ~ Group, data=BigData, FUN=sd)
Bigse<-aggregate(A ~ Group, data=BigData, FUN=std.error)

Bigmean$SD<-Bigsd$A
Bigmean$SE<-Bigse$A


# With a large n, SD and SE are very different. Not sure which is a better way to create error bars
ggplot(Bigmean, aes(x=Group, y=A)) + 
  geom_bar(stat="identity", color="black", 
           position=position_dodge()) +
  geom_errorbar(aes(ymin=A-SD, ymax=A+SD), width=.2,
                position=position_dodge(.9)) 



ggplot(Bigmean, aes(x=Group, y=A)) + 
  geom_bar(stat="identity", color="black", 
           position=position_dodge()) +
  geom_errorbar(aes(ymin=A-SE, ymax=A+SE), width=.2,
                position=position_dodge(.9)) 

Given large datasets, is there a preferred method to calculate variance? Specially geared towards data visualizations as above?

$\endgroup$
  • 1
    $\begingroup$ your colleague was right; for large datasets SE should be small else the data gathering process was not consistent. What kind of data is it? if it is time series maybe you can try simple line plots and see if there is a trend. You can try other methods such as kernel density estimation to estimate the underlying probability distribution function. $\endgroup$ – joydeep bhattacharjee Oct 29 at 3:16
  • 1
    $\begingroup$ Standard error of what? Standard deviation and standard error, despite their similar names, have distinct roles in statistics. // What do you want to say with your standard error or standard deviation? $\endgroup$ – Dave Oct 29 at 3:26
  • $\begingroup$ I was referring to the Standard Error of the mean. And I'm hoping to be able to quickly summarize (visually) how good a representation the mean is of each grouping. So that when I compare them in the above plots, I have an idea if there is s a significant difference between each group. $\endgroup$ – Vint Oct 29 at 12:58
2
$\begingroup$

Note that "standard error" in this case is short for "standard error of the mean". That means that it's measuring the variation associated with your sample mean. Of course, that variation is much lower than the variation associated with an individual sample.

In your example, you give a case where the standard deviation is very large compared with the mean. So, it is very reasonable that the bars for the standard deviation are wide compared with the magnitude of the mean.

So, it depends on what you are trying to accomplish. Do you want to give an idea of the location of the population mean? Then use the standard error of the mean or perhaps a different confidence interval. (Note that the standard error of the mean is roughly equivalent to a 68% confidence interval for the population mean.)

Do you want to give an idea of the variation among individual observations? Then use the standard deviation.

For the example you give and the impression of the conversation, the standard deviation sounds good. If your data are similarly symmetric, sort of normally distributed, then the standard deviation gives you a fair assessment of the variability among individual observations.

For less well-behaved data you could use a boxplot with the geom_boxplot() command instead of a bar plot. There are other choices to make there but the defaults are at least reasonable as a quick starting point.

| cite | improve this answer | |
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.