I have performed linear regression (lm) on two modified p-value types: q-value and Benjamini-Hochberg. Results gives two astronomical outliers, however, after removal of those, new outliers are always present. Could someone please replicate the code and see if issue prevails? What could be the possible source of an issue?

Here is the full code for easy copy/paste:


p = 50
m = 10
pval = c(rbeta(m,1,100), runif(p-m,0,1))

BHpval <- p.adjust(pval,method="BH")
qval_ <- qvalue(pval)

fit2 <- lm(qval_$qvalues ~ BHpval)
  • 1
    $\begingroup$ Why are you using regression for that? Why would you believe they would be linearly related? $\endgroup$ – Glen_b -Reinstate Monica Nov 12 at 6:25
  • $\begingroup$ I believe you have asked a version of this question somewhere before -- (possibly on stackoverflow and perhaps from a different account). Please link to that question, since advice/comments given there will be highly relevant. $\endgroup$ – Glen_b -Reinstate Monica Nov 12 at 8:58

Hard to give good advice without seeing data. However, the meaning of an outlier is not necessarily what you are assuming it is. For example, near outliers are most frequently encountered for non normal distributions. Those do not justify removal, nor do far outliers. As a first step in investigating outliers, one should attempt to identify the distribution type. For example, a reciprocal normal distribution could have a number of outliers merely because it is reciprocal normal, and not because there is something wrong that requires removal of data. If the distribution is nasty, e.g., Cauchy, removing one outlier might just make the next one pop up.

As a second step, one can attempt to create a normal distribution from the data. For example, if the data is log-normal, taking the logarithm and testing that for outliers should reduce the number of outliers substantially, and in the normal case near outliers occur for 0.698% of the realizations, whereas far outliers occur for only $0.000234$%.

As a third step, one can, for example, compute the binomial probability of an upper outlier being $\geq$ to the observed value. If that is significant (low probability $\leq \alpha$), only then is it likely an erratic outlier for that distribution.

As a fourth and final step, the good scientist identifies a source for the erratic behaviour; instrument glitch, intermittent fault, outside source contamination, theoretical error, methodological misstep and only then, if an appropriate cause can be shown, does he (whatever, groan) remove it.

  • $\begingroup$ Data is artificially generated with code i shared. $\endgroup$ – Raimonds R Nov 12 at 8:09
  • 1
    $\begingroup$ @RaimondsR Great, but I do not use R, rather I use Mathematica, so your question needs translation into pseudocode for appeal to a wider audience. $\endgroup$ – Carl Nov 12 at 10:40

Not the answer you're looking for? Browse other questions tagged or ask your own question.