I have data where I believe that most of it is normally distributed, but a few extreme outliers are drawn from a different distribution. To estimate the mean and variance of the normal part of my mostly normal data, I'd hoped to exclude the outliers by eliminating the top and bottom 10% quantiles from the data, and to use the
MASS::fitdist function in R. Perhaps unsurprisingly to experts, this results in a biased (incorrect) estimate of the variance. The problem is easy to demonstrate in R. (Ignore for now the possibility that there are any non-normally distributed subpopulations in the data, and suppose the data is completely normally distributed.)
# generate random data set.seed(0) all_normal <- rnorm(1000000, mean=0, sd=10) # eliminate the bottom 10% and top 10% of the values trunc <- 0.1 lb <- quantile(all_normal, trunc) ub <- quantile(all_normal, 1-trunc) not_normal <- all_normal[all_normal > lb & all_normal < ub] # fit a distribution to the truncated data print(MASS::fitdistr(not_normal, 'normal'))
This results in
mean sd -0.007818535 6.617770945 ( 0.007398893) ( 0.005231807)
The best-fit standard deviation is $6.618\pm0.005$, but the actual standard deviation is obviously 10 in my numeric example. My truncation based approach is not suitable.
Is there a generally recognized approach to fitting truncated data? Is there for example a theoretical correction factor, computable from my
trunc value, that I can use to recover an unbiased estimate of the standard deviation from top- and bottom-quantile truncated-data? It appears that there are formulas for recovering an estimated standard deviation from trimmed data, but what if I want to estimate the mean and standard deviation simultaneously? Can I assume that the estimated mean and the estimated standard deviation do not co-vary?