5
$\begingroup$

What is the effect of sequentially removing outliers on the p-value of a t-test and can graphing changes in p-value provide useful information

In the data we compare soil pH and soil moisture data collected by students and teaching assistants. We begin by ranking student-collected data points based on IQR score. This score was the absolute value of the difference between a data point and the median divided by the interquartile change: |(ni - median)/IQR|. Once ranked, we run a t-test comparing student and teaching assistant data. After the initial test, we sequentially remove student outliers from the analysis. After each data point is removed, the t-test is re-run. The figures below chart p-value on the y-axis and the number of IQR-ranked data points removed from the analysis.

enter image description here

How I am interpreting these figures:

I would remove outliers that fit the following conditions: A) the left side of the figure has a very steep slope and B) that slope decelerates rapidly, observed as a sharp angle in the line. This appears to be the case in the soil moisture data on the left.

A curving negative slope without sharp bends on the left side of the figure (e.g. pH figure) indicates outliers that should not be removed from the analysis since they align with the rest of the dataset. While these points have high IQR values, there is nothing to indicate that they do not just represent tails on a typical distribution.

My questions:

  1. Is this a valid qualitative analysis to address our question?
  2. Is the interpretation of these figures accurate?
  3. Is there anything else we can glean from these figures, particularly any patterns in the middle or right side of the x-axes?
Improve this question
$\endgroup$
1
  • 4
    $\begingroup$ This seems like a bizarre procedure. Why did you do this ? You ask "Is this a valid qualitative analysis to address our question?" but I don't even know what your question is. In what sense even is the analysis "qualitative" ? $\endgroup$
    – Robert Long
    Nov 19, 2020 at 21:35

1 Answer 1

Reset to default
4
$\begingroup$

Is this a valid qualitative analysis to address our question?

I'm not sure what you question is, but I'm confident the answer is "absolutely not". You're performing several tests and removing data without good reason except for "they are outliers" -- which is not a precise term in my own opinion anyway.

Is the interpretation of these figures accurate?

In all likelihood, no. Your method seems very ad hoc and so it is difficult to comment on properties of this method of estimation. I'm extremely dubious about any inferences made from this approach. Can you simulate your approach to demonstrate some desirable properties?

Is there anything else we can glean from these figures, particularly any patterns in the middle or right side of the x-axes?

I would hesitate from inferring anything from these. What you're doing sounds very much to me like data dredging if not p-hacking.

Why don't you begin by telling is your research question. These students are collecting soil samples. Why? Why are you interested in comparing TA vs. Student data? Are you not confident in the student's abilities?

Improve this answer
$\endgroup$
6
  • $\begingroup$ (+1) I don't get how such analyses are motivated. $\endgroup$
    – Robert Long
    Nov 19, 2020 at 21:38
  • $\begingroup$ @RobertLong They are motivated by the fact one can, and if one did the outcome would not appear any different from a more successful analysis that did not entail these practices $\endgroup$
    – benxyzzy
    Nov 20, 2020 at 12:16
  • 4
    $\begingroup$ @benxyzzy but just because one can doesn't mean one should $\endgroup$
    – Robert Long
    Nov 20, 2020 at 18:47
  • $\begingroup$ @RobertLong I agree, but the perverse incentive is clear $\endgroup$
    – benxyzzy
    Nov 21, 2020 at 8:23
  • $\begingroup$ @benxyzzy it's not clear to my little brain. Please could you explain this incentive ? $\endgroup$
    – Robert Long
    Nov 21, 2020 at 9:40

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.