# Removing outliers non-normal data Hi,

I was wondering if anyone had advice on removing outliers. In a practical experiment relative telomere length in DNA samples was measured in duplicate. Expected values are around 1, and the peak of samples between 0.0-0.5 are fairly certain failed samples. Since this data isn't normally distributed and the samples are likely fails, I was looking for a valid method to get rid of obvious outliers. Any advice how to do this for this data? I have tried Median Absolute Deviation already, but the peak remains.

Thanks!

• When binned, the pdf (probability density function) of your data appears quite lumpy and anything but normally distributed. It may be that parametric models and assumptions aren't met and won't work. One option is to revert to nonparametric methods and models that are robust to outliers, e.g., estimators like the median (not the mean), quantile or kernel (not OLS) regression as well as assuming underlying extreme value probability distributions such as the Weibull, Cauchy or Levy (not the normal). Adopting these approaches should, for the most part, eliminate the need to delete outliers. – Mike Hunter Jul 26 '17 at 11:13

The "failed" samples are not really "outliers", they are clearly a normal part of your experimental process. What you need to do is correctly model this process.

For example, it may be appropriate to assume a mixture of normals distribution. You would let $X_i \sim^{\text{iid}} \text{Bern}(p)$ be 0 if sample $i$ is a "failure", 1 otherwise, and then assume different conditional normal distributions for each outcome:

$$Y_i|(X_i = 0) \sim N(\mu_f,\sigma_f^2)$$ $$Y_i|(X_i = 1) \sim N(\mu_s,\sigma_s^2)$$

You only observe the $Y_i$ but you can estimate the parameter(s) of interest (which seems to be $\mu_s$, the mean of successful samples) using an algorithm like EM.

In addition to the presented solution by @Kim I would transform data in order to normalize it which is a good solution instead of deleting outliers. See these answers to know more about the used transformations link.