# How to characterize variance with large datasets

I have a large dataset (> 100,000 rows) of ecological data. In some of my first attempts to visualize the data, I used bar plots with calculated means and error bars (see below plots). My go-to for error bars is usually the standard-error, and this first round of plots had uniformly small error bars. One of my colleagues suggested that since my n was so large, I would always have a small SE, and that SD is a better metric to characterize variance with large datasets.

Following her advice, I created the same plots as above, only this time using the SD for my error bars, and now I have huge error bars; larger than the means themselves. It seems like SE creates error bars that are too small to be a useful indicator of variance, and SD creates error bars that are too big. I'm not sure if the data is just too noisy to work with, or I don't have a good way to begin to visualize the variance with this large dataset.

As an example

library(plotrix)
library (ggplot2)

A<-rnorm(100, mean=5, sd=5)
B<-rnorm(100, mean=2, sd=2)
C<-rnorm(100, mean=4, sd=6)
Group<-rep(1:5, 20)

SmallData<-data.frame(A,B,C,Group)

Smallmean<-aggregate(A ~ Group, data=SmallData, FUN=mean)
Smallsd<-aggregate(A ~ Group, data=SmallData, FUN=sd)
Smallse<-aggregate(A ~ Group, data=SmallData, FUN=std.error)

Smallmean$$SD<-Smallsd$$A
Smallmean$$SE<-Smallse$$A

# With a small n, both SD and SE are similar
ggplot(Smallmean, aes(x=Group, y=A)) +
geom_bar(stat="identity", color="black",
position=position_dodge()) +
geom_errorbar(aes(ymin=A-SD, ymax=A+SD), width=.2,
position=position_dodge(.9))

ggplot(Smallmean, aes(x=Group, y=A)) +
geom_bar(stat="identity", color="black",
position=position_dodge()) +
geom_errorbar(aes(ymin=A-SE, ymax=A+SE), width=.2,
position=position_dodge(.9))

#Much larger n and SD
A<-rnorm(1000000, mean=5, sd=200)
B<-rnorm(1000000, mean=2, sd=2)
C<-rnorm(1000000, mean=4, sd=6)
Group<-rep(1:5, 200000)

BigData<-data.frame(A,B,C,Group)

Bigmean<-aggregate(A ~ Group, data=BigData, FUN=mean)
Bigsd<-aggregate(A ~ Group, data=BigData, FUN=sd)
Bigse<-aggregate(A ~ Group, data=BigData, FUN=std.error)

Bigmean$$SD<-Bigsd$$A
Bigmean$$SE<-Bigse$$A

# With a large n, SD and SE are very different. Not sure which is a better way to create error bars
ggplot(Bigmean, aes(x=Group, y=A)) +
geom_bar(stat="identity", color="black",
position=position_dodge()) +
geom_errorbar(aes(ymin=A-SD, ymax=A+SD), width=.2,
position=position_dodge(.9))

ggplot(Bigmean, aes(x=Group, y=A)) +
geom_bar(stat="identity", color="black",
position=position_dodge()) +
geom_errorbar(aes(ymin=A-SE, ymax=A+SE), width=.2,
position=position_dodge(.9))


Given large datasets, is there a preferred method to calculate variance? Specially geared towards data visualizations as above?

• your colleague was right; for large datasets SE should be small else the data gathering process was not consistent. What kind of data is it? if it is time series maybe you can try simple line plots and see if there is a trend. You can try other methods such as kernel density estimation to estimate the underlying probability distribution function. – joydeep bhattacharjee Oct 29 at 3:16
• Standard error of what? Standard deviation and standard error, despite their similar names, have distinct roles in statistics. // What do you want to say with your standard error or standard deviation? – Dave Oct 29 at 3:26
• I was referring to the Standard Error of the mean. And I'm hoping to be able to quickly summarize (visually) how good a representation the mean is of each grouping. So that when I compare them in the above plots, I have an idea if there is s a significant difference between each group. – Vint Oct 29 at 12:58