# How to use Tukey's Biweight Function to appropriately weight outliers to generate a normal distribution

I am working with a distribution that has outliers beyond 1.5*3rd Qu.. I'm using Shankar, et al. Recommendations for the validation of immunoassays used for detection of host antibodies against biotechnology products (2008), as a guide and it suggests (if outliers cannot be removed):

Data transformation is often needed in order to satisfy the distributional assumptions of the statistical analysis (e.g., ANOVA). Low and high outliers arising from analytical or biological abnormalities should preferably be excluded, or appropriately down-weighted (e.g., by use of Median and Median Absolute Deviation or Tukey’s biweight function) in the determination of a screening cut point.

The distribution can be created with:

set.seed(12)
tmp <- rnorm(100, mean = 100, sd = 1)
set.seed(12)
outliers <- rnorm(10, mean = (1.5 * summary(tmp)[["3rd Qu."]]), sd = 1)
## Bind the columns together
dat <- data.frame(cbind(c(tmp, outliers),
c(rep("Expected", each = length(tmp)),
rep("Outlier", each = length(outliers)))))
colnames(dat) <- c("Units", "Category")

## Convert ECL values to numeric
dat$Units <- as.numeric(as.character(dat$Units))


The distribution looks like this:

I tried to normalize through a log transformation, as well as a median-based transformation, but I wasn't able to generate a normal distribution.

How could I appropriately weight the distribution to normalize it?

• Why do you want to force the distribution to be normal? You could use a nonparametric method if the distribution is not normal. If you know that the outliers are inappropriate values removing them makes more sense then down-weighting them. Normalize is not the right term. You are trying to make the distribution approximately Gaussian (i.e. normal). – Michael R. Chernick Apr 23 '18 at 0:16
• The usual Tukey criterion for high values that should be thought about out is values exceeding upper quartile $+$ 1.5 IQR. If you're using 1.5 $\times$ upper quartile as a cut-off for inspection, that would be different. – Nick Cox Apr 23 '18 at 1:31
• Two clumps will, broadly speaking, remain two clumps whatever you do by way of plausible transformation. – Nick Cox Apr 23 '18 at 1:33