Scientifically reasonable or not ? exclusion of very, very uncertain values from statistical analysis I have 5 treatments A1 .. A5 and 4 independent individuals per treatment, whose parameter X is of interest to me.
My objective is to compare X among treatments, and see if one or more treatment differ significantly from the others.
To do this, I measure X in each individual, 5 times.
I then get 5*4*5=100 values, that reduce to 20 when the mean is calculated at the "individual" level.
When plotting each individual's mean against treatment, I clearly see one treatment, say A2, is obviously different from A1, A3 and A4, but not A5.
But when doing a Tukey's HSD test following an ANOVA (model = X ~ treatment), all treatments are assigned the same letter...
The problem is : A5's individuals means are very different, in the sense that the confidence interval for A5's mean overlap all other treatments' CI. This would not be a problem if I was confident about the individuals means in A5 but, for some known reason, the uncertainties associated with most of the individuals means in A5 are huge, to the point they do not differ significantly from zero.
When applying the same statistics to only A1 .. A4 (i.e. excluding A5 individuals from the tests), A2 comes out significantly different from the other treatments means.
Is it reasonable, and scientifically acceptable, to exclude A5 individuals from the model, on the basis of the uncertainty in the estimate of their respective X values ?
I think the problem may be about heteroscedasticity across treatments.
 A: In order to decide whether it is reasonable to exclude data points, you must make a reasonable assessment about why you see a larger variability in A5 treatment group. Is there one subject that has a replicate mean that is very different than the 3 other subjects? If so, look at the individual replicate values and see if there are any values that are clearly out of range and are more likely due to technical variation rather than biological variation. Those data points can be removed.
If this does not solve the problem then examine whether the A5 treatment has larger differential effects on the subjects. If the A5 subjects were able to be re-measured with one of the other treatments, would you expect to see the same variance? If the A5 treatment is causing the larger variance then there is some other factor that might be important. You could also add more subjects to the A5 group to compute a more stable treatment mean estimate.
In any case, you should be able to perform pairwise tests between all groups and show that A2 is pairwise different from A1, A3 and A4.
