# Can something show bias but not be significant?

"A slight bias (although not significant) for
50/30 microRNAs to target genes with related
functions using miRanda (Figure 2B)"


However, if there is a bias: let's say one mean is greater than the other, but that it is not significant, can you say there is a bias at all? Couldn't it be (equally?) likely that the mean be lower than the other given that the two distributions aren't significantly indistinguishable?

I understand that you may argue that if there was more data then that mean may eventually become significant, but the dataset is in the thousands and is likely to not to get much larger?

Does the term 'slight bias' have any statistical meaning?

-
In Fig 2b, you see the 3' and 5' UTR predictions for human Miranda were the worst performing in terms of their calibration (O/E ratio in the vicinity of ~10%). The authors' point is that the precision of the predictions were weak and the prediction intervals indicate that this estimate may have still been consistent with the expected 3' / 5' UTR Miranda location (or expression... or whatever they're measuring here). When they say "slight" they are downplaying the fact that it's actually somewhat big (but not statistically significant... after correcting for multiple comparisons...) –  AdamO Apr 25 at 21:23

In short, the authors, like almost all scientists, are (presumably) using the Null Hypothesis Significance Testing approach, with all it's associated pitfalls.

These methods can tell us the probability of this data being observed given that the null hypothesis is true (the null hypothesis here being 'no bias at all'), and we're allowed to conclude that the null hypothesis is incorrect if the probability of the data given the null hypothesis is less than 5%.

Bayesian approaches, on the other hand, could answer the question you ask - but rather than providing a proper answer and explaining the distinction, I'm going to point you towards one of John Kruschke's papers on the subject.

-

This is how I understand such sentences:

When stating that there was a 'slight bias', the authors talk about descriptive statistics, simply saying that in the sample there were more Xs than Ys. By talking about (the lack of) significance, they move to the realm of inferential statistics, inquiring whether this observed bias could be explained by chance.

I think that the legitimacy of including such descriptive report in a scientific paper mostly depends on how it is interpreted.

-

"Slight bias" doesn't have a specific meaning, but what the authors are doing is talking about the magnitude and direction of their effect measure, and decoupling that from it's significance.

Their results show a slight effect, and that alone may be interesting, even without significance - if it agrees with other studies, what they mean by slight, etc.

Also, keep in mind that, while we treat it as binary, "not significant" is not an on-off switch. It's probably not equally likely that the effect is in the other direction, and in some cases of non-significance (p = 0.0506) it's actually quite unlikely, but still non-significant.

To use two other examples from research I've seen:

The association between many environmental exposures and rare disease outcomes may never be significant - the outcomes are too rare, the effects small but non-zero, the sample sizes too small, and the studies simply too expensive to run. So you may have twenty non-significicant results, but if they're all in a single direction, that's information.

In a study I conducted, the hazard ratio of a study turned out to be 1.97 with a confidence interval from 0.96 to 4.01. Not significant, but my sample size was dictated by the size of a particular data set, not the power needed to see an effect of that size. And the vast bulk of the evidence is that there's a positive effect - while it is possible that the effect is zero or a little below, the most likely estimate is above zero, and probably still worth reporting.

-