What is the effect of sequentially removing outliers on the p-value of a t-test and can graphing changes in p-value provide useful information
In the data we compare soil pH and soil moisture data collected by students and teaching assistants. We begin by ranking student-collected data points based on IQR score. This score was the absolute value of the difference between a data point and the median divided by the interquartile change: |(ni - median)/IQR|. Once ranked, we run a t-test comparing student and teaching assistant data. After the initial test, we sequentially remove student outliers from the analysis. After each data point is removed, the t-test is re-run. The figures below chart p-value on the y-axis and the number of IQR-ranked data points removed from the analysis.
How I am interpreting these figures:
I would remove outliers that fit the following conditions: A) the left side of the figure has a very steep slope and B) that slope decelerates rapidly, observed as a sharp angle in the line. This appears to be the case in the soil moisture data on the left.
A curving negative slope without sharp bends on the left side of the figure (e.g. pH figure) indicates outliers that should not be removed from the analysis since they align with the rest of the dataset. While these points have high IQR values, there is nothing to indicate that they do not just represent tails on a typical distribution.
- Is this a valid qualitative analysis to address our question?
- Is the interpretation of these figures accurate?
- Is there anything else we can glean from these figures, particularly any patterns in the middle or right side of the x-axes?