0
$\begingroup$

I am working with a distribution that has outliers beyond 1.5*3rd Qu.. I'm using Shankar, et al. Recommendations for the validation of immunoassays used for detection of host antibodies against biotechnology products (2008), as a guide and it suggests (if outliers cannot be removed):

Data transformation is often needed in order to satisfy the distributional assumptions of the statistical analysis (e.g., ANOVA). Low and high outliers arising from analytical or biological abnormalities should preferably be excluded, or appropriately down-weighted (e.g., by use of Median and Median Absolute Deviation or Tukey’s biweight function) in the determination of a screening cut point.

The distribution can be created with:

set.seed(12)
tmp <- rnorm(100, mean = 100, sd = 1)
set.seed(12)
outliers <- rnorm(10, mean = (1.5 * summary(tmp)[["3rd Qu."]]), sd = 1)
## Bind the columns together
dat <- data.frame(cbind(c(tmp, outliers),
                        c(rep("Expected", each = length(tmp)), 
                          rep("Outlier", each = length(outliers)))))
## Add Column Names
colnames(dat) <- c("Units", "Category")

## Convert ECL values to numeric
dat$Units <- as.numeric(as.character(dat$Units))

The distribution looks like this: enter image description here

I tried to normalize through a log transformation, as well as a median-based transformation, but I wasn't able to generate a normal distribution.

How could I appropriately weight the distribution to normalize it?

Improve this question
$\endgroup$
3
  • 1
    $\begingroup$ Why do you want to force the distribution to be normal? You could use a nonparametric method if the distribution is not normal. If you know that the outliers are inappropriate values removing them makes more sense then down-weighting them. Normalize is not the right term. You are trying to make the distribution approximately Gaussian (i.e. normal). $\endgroup$
    – Michael R. Chernick
    Apr 23, 2018 at 0:16
  • $\begingroup$ The usual Tukey criterion for high values that should be thought about out is values exceeding upper quartile $+$ 1.5 IQR. If you're using 1.5 $\times$ upper quartile as a cut-off for inspection, that would be different. $\endgroup$
    – Nick Cox
    Apr 23, 2018 at 1:31
  • 2
    $\begingroup$ Two clumps will, broadly speaking, remain two clumps whatever you do by way of plausible transformation. $\endgroup$
    – Nick Cox
    Apr 23, 2018 at 1:33

1 Answer 1

Reset to default
1
$\begingroup$

First, ANOVA does not make any assumptions about the distribution of the dependent variable, it makes assumptions about the distribution of the errors. That said, if your dependent variable really looks like that, you shouldn't be trying to fit it into ANOVA, but using a method that does not make assumptions about the distribution of the residuals.

One solution would be to do separate regressions on the two groups.

Another would be to use quantile regression.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.