2
$\begingroup$

I have read several articles concerning subgroup interaction interpretation but none of them have the situation I am dealing with. Suppose we have an experiment with a control vs treatment influence on some target value and there are 2 subgroups across the population: male and female.

We run a regression analysis and get p-values for:

  1. The overall effect of treatment on whole population
  2. The individual effect in subgroups (male p-value and female p-value)
  3. The interaction p-value.

So the analysis went as follows. First, we looked at the interaction p-value to test the null hypothesis for the interaction: the treatment had the same effect on the subgroups. The p-value was above the pre-determined threshold (alpha) and we did not reject the null hypothesis.

Second, we looked at the p-values for subgroup experiment to test null hypothesis for each one: treatment had no effect on the target value. Here we found out that the p-value for the male group was lower than alpha and the p-value for the female group was above alpha. So we had to reject the null hypothesis for the male group and accept that there is a statistically significant test of treatment. However, we did not reject the null hypothesis for females.

So we arrived to the conclusion that the effect in subgroups is different. But! An earlier interaction analysis showed us the contrary: that the groups do not differ in relation to treatment effect.

How do I get over this contradiction? This is a real life example, the sample groups sizes are both over a thousand observations each and are quite close to one another.

Improve this question
$\endgroup$

2 Answers 2

Reset to default
6
$\begingroup$

Welcome to CV. There are a few errors in your analysis, or, rather, in your interpretation of your analysis.

First, you tested interpretation and found it was not significant. You concluded that the "treatment has the same effect" in the subgroups. This is not a correct interpretation of a nonsignificant p value. This is the fallacy known as "accepting the null".

Second, you looked at each subgroup separately and compared their p values. This is also a mistake. See

Gelman and Stern: The Difference between `significant' and 'not significant' is not, itself, statistically significant . Rather than comparing p values, you should compare effect sizes.

Third, you conclude that there is a contradiction. But there is not.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thank you for your welcome and answer. 1. The conclusion should be "there is not enough evidence to reject null hypothesis", am I right? $\endgroup$
    – Boris Smirnov
    Commented Feb 23 at 13:14
  • 1
    $\begingroup$ @BorisSmirnov see my answer. I include a couple examples from other posts I have made here and a tale about how the $p$ value, interpreted completely on it's own, can lead to substantial errors in thinking. The short story is this: you need a lot more information to determine anything of value from your study. $\endgroup$
    – Shawn Hemelstrand
    Commented Feb 23 at 13:30
  • $\begingroup$ @ShawnHemelstrand / PeterFlom I respectfully disagree with the "wording" (though perhaps not the intended spirit) of the statement "comparing p-values is a mistake". You both seem to be arguing against comparing Reject/Non-reject operations, rather than comparing the p-value itself. However, the p-value, when taken at face value as a measure of how atypical a dataset is w.r.t. a model, yields very useful information; and I would argue that in the case where effect sizes operate across different units, then a p-value (not the decision made by it) is much more informative than an effect size. $\endgroup$
    – Tasos Papastylianou
    Commented Feb 24 at 8:45
  • 1
    $\begingroup$ I can't speak for Peter, but for the same reasons you should never take a $p$ value at face value, the same can be said about effect sizes. My main concern with the post was that the entire question was wrapped around $p$ values, which by nature lends itself to very grave mistakes. $\endgroup$
    – Shawn Hemelstrand
    Commented Feb 24 at 9:20
  • 1
    $\begingroup$ I agree with @ShawnHemelstrand. Also, I'd say that a p value measures how unusual data is compared to a null hypothesis. So, by comparing two p values, you could say "If the null is true, this data set is less likely than that one". That is rarely of interest. $\endgroup$
    – Peter Flom
    Commented Feb 24 at 11:55
6
$\begingroup$

The Dangers of P-Value Hunts

Peter already noted issues in your interpretation that I completely agree with (see my answer here, here, or here, which all show how meaningless $p$ values can become without understanding the effect sizes or the functional relationships of the variables we are estimating). Rather than state the case here like I normally do, I'll share a story the damage of such thinking from Hauer (2007):

Our story begins in 1976 when a consultant submitted a report about the safety repercussions of RTOR (Right Turn on Red) to the Governor and General Assembly of Virginia. The studies then extant were deemed deficient and the consultant did his own before–after study at 20 intersections with the results in Table 1.

enter image description here

Looking at the data in Table 1, persons without training in statistics would think that after RTOR was allowed, these intersections were somewhat less safe. However, the consultant concluded, quite correctly, that the change was not statistically significant. The Commissioner of the Virginia Department of Highways and Transportation sent the consultant’s report to the Governor and in the letter of transmittal wrote: “we can discern no significant hazard to motorists or pedestrians from implementation of the general permissive rule (i.e. of RTOR). No significant increase in traffic crashes has been noted following adoption of right-turn-on-red in any state including Virginia”...

More published studies followed. One study in 1977 found that there were 19 crashes involving right turning vehicles before and 24 after allowing RTOR and “this increase in accidents in not statistically significant, and therefore it cannot be said that this increase in RTOR accidents is attributable to RTOR”. And so the sequence of small studies all pointing in the same direction but with statistically not significant results continued to accumulate, till that last study which I followed was published in 1983. While 287 crashes to right turning vehicles were expected, 313 were counted. The authors concluded, once again, that there was no significant difference in vehicular crashes.

In the 1980s, researchers finally looked at the total years of accumulated evidence rather than single studies and uncovered the unflattering truth (Preusser, 1982):

Estimates of the magnitude of the increases ranged from 43% to 107% for pedestrian accidents and 72% to 123% for bicyclist accidents.

The wholescale adoption of this procedure, completely based off statistical significance, literally killed a large percentage of the population that didn't have to die.

Thoughts

Please inspect the data visually, find out the mean differences and their standard errors / confidence intervals, and determine what the data actually says about your groups. As a reminder, a $p$ value is just a probability of a given estimate (or more extreme) given the null hypothesis is true (Wasserstein & Lazar, 2016). One can neither accept the null nor the alternative hypothesis from a low $p$ value, nor can they ascertain what the magnitude of the effect is from a $p$ value alone. So as Peter noted, please do some more digging.

  • Hauer, E. (2004). The harm done by tests of significance. Accident Analysis & Prevention, 36(3), 495–500. https://doi.org/10.1016/S0001-4575(03)00036-8
  • Preusser, D.F., Leaf, W.A., DeBartdo, K.B., Blomberg, R.D., Leoy, M.M., 1982. The effect of right-turn-on-red on pedestrian and bicycle accidents. J. Safety Res. 13, 45–55.
  • Wasserstein, R. L., & Lazar, N. A. (2016). The ASA statement on p-values: Context, process, and purpose. The American Statistician, 70(2), 129–133. https://doi.org/10.1080/00031305.2016.1154108
Improve this answer
$\endgroup$
3
  • $\begingroup$ Thank you. The effect sizes argument is valuable. If I understand the moral of the cited story correctly, one should have noticed piling evidence though statistically insignificant in parts and complete a more thorough analysis. Let's say that in my example the observed effect in males is of practical significance. Let's also more accurately state our conclusions in terms of hypothesis: we didn't find evidence to reject interaction, but have alpha% probability that treatment has effect in males. I would greatly appreciate to hear your interpretation of the results to learn from it. $\endgroup$
    – Boris Smirnov
    Commented Feb 23 at 14:00
  • $\begingroup$ The issue afoot is not how thorough the analysis is (though of course more data and a more rigorous test of hypotheses is always better). The problem is stating things like "no differences" when there clearly are some present, or vice versa (again, my examples linked in my answer show some clear issues with this). The $p$ value is just a probability, one that tentatively says that "the differences are not zero" if low enough. $\endgroup$
    – Shawn Hemelstrand
    Commented Feb 23 at 14:07
  • $\begingroup$ It appears you are also mixing up $a$ with $p$, where the former is an error cutoff for determining "low enough" $p$ values that lead to a lower rate of incorrect NHST decisions. I have some more strong sentiments about this specific issue and personally adopt a neoFisherian approach to $p$ values, as discussed in the latter half of my answer here. $\endgroup$
    – Shawn Hemelstrand
    Commented Feb 23 at 14:07

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.