How to characterize variance with large datasets

Question

I have a large dataset (> 100,000 rows) of ecological data. In some of my first attempts to visualize the data, I used bar plots with calculated means and error bars (see below plots). My go-to for error bars is usually the standard-error, and this first round of plots had uniformly small error bars. One of my colleagues suggested that since my n was so large, I would always have a small SE, and that SD is a better metric to characterize variance with large datasets.

Following her advice, I created the same plots as above, only this time using the SD for my error bars, and now I have huge error bars; larger than the means themselves. It seems like SE creates error bars that are too small to be a useful indicator of variance, and SD creates error bars that are too big. I'm not sure if the data is just too noisy to work with, or I don't have a good way to begin to visualize the variance with this large dataset.

As an example

library(plotrix)
library (ggplot2)

A<-rnorm(100, mean=5, sd=5)
B<-rnorm(100, mean=2, sd=2)
C<-rnorm(100, mean=4, sd=6)
Group<-rep(1:5, 20)

SmallData<-data.frame(A,B,C,Group)


Smallmean<-aggregate(A ~ Group, data=SmallData, FUN=mean)
Smallsd<-aggregate(A ~ Group, data=SmallData, FUN=sd)
Smallse<-aggregate(A ~ Group, data=SmallData, FUN=std.error)

Smallmean$SD<-Smallsd$A
Smallmean$SE<-Smallse$A


# With a small n, both SD and SE are similar 
ggplot(Smallmean, aes(x=Group, y=A)) + 
  geom_bar(stat="identity", color="black", 
           position=position_dodge()) +
  geom_errorbar(aes(ymin=A-SD, ymax=A+SD), width=.2,
                position=position_dodge(.9)) 



ggplot(Smallmean, aes(x=Group, y=A)) + 
  geom_bar(stat="identity", color="black", 
           position=position_dodge()) +
  geom_errorbar(aes(ymin=A-SE, ymax=A+SE), width=.2,
                position=position_dodge(.9)) 





#Much larger n and SD
A<-rnorm(1000000, mean=5, sd=200)
B<-rnorm(1000000, mean=2, sd=2)
C<-rnorm(1000000, mean=4, sd=6)
Group<-rep(1:5, 200000)

BigData<-data.frame(A,B,C,Group)


Bigmean<-aggregate(A ~ Group, data=BigData, FUN=mean)
Bigsd<-aggregate(A ~ Group, data=BigData, FUN=sd)
Bigse<-aggregate(A ~ Group, data=BigData, FUN=std.error)

Bigmean$SD<-Bigsd$A
Bigmean$SE<-Bigse$A


# With a large n, SD and SE are very different. Not sure which is a better way to create error bars
ggplot(Bigmean, aes(x=Group, y=A)) + 
  geom_bar(stat="identity", color="black", 
           position=position_dodge()) +
  geom_errorbar(aes(ymin=A-SD, ymax=A+SD), width=.2,
                position=position_dodge(.9)) 



ggplot(Bigmean, aes(x=Group, y=A)) + 
  geom_bar(stat="identity", color="black", 
           position=position_dodge()) +
  geom_errorbar(aes(ymin=A-SE, ymax=A+SE), width=.2,
                position=position_dodge(.9)) 

Given large datasets, is there a preferred method to calculate variance? Specially geared towards data visualizations as above?

Answer 1

Note that "standard error" in this case is short for "standard error of the mean". That means that it's measuring the variation associated with your sample mean. Of course, that variation is much lower than the variation associated with an individual sample.

In your example, you give a case where the standard deviation is very large compared with the mean. So, it is very reasonable that the bars for the standard deviation are wide compared with the magnitude of the mean.

So, it depends on what you are trying to accomplish. Do you want to give an idea of the location of the population mean? Then use the standard error of the mean or perhaps a different confidence interval. (Note that the standard error of the mean is roughly equivalent to a 68% confidence interval for the population mean.)

Do you want to give an idea of the variation among individual observations? Then use the standard deviation.

For the example you give and the impression of the conversation, the standard deviation sounds good. If your data are similarly symmetric, sort of normally distributed, then the standard deviation gives you a fair assessment of the variability among individual observations.

For less well-behaved data you could use a boxplot with the geom_boxplot() command instead of a bar plot. There are other choices to make there but the defaults are at least reasonable as a quick starting point.