# Is this heteroskedasticity or a cautionary tale about arbitrary p-values or…?

As a sandbox experiment trying to conceptually mimic the following finding from a paper

A higher incidence of early risk factors accounted for racial differences related to any juvenile arrest, as well as differences in violence- and theft-related arrests. However, increased exposure to early risk factors did not explain race differences in drug-related arrests.

let us consider the following data set (in R):

 x y 1 1 1.000000 2 2 2.000000 3 3 3.000000 4 4 4.000000 5 5 5.000000 6 6 6.000000 7 7 7.000000 8 8 8.000000 9 9 9.000000 10 10 10.000000 11 1 5.028417 12 2 7.329881 13 3 6.759483 14 4 5.022186 15 5 2.098116 16 6 2.603654 17 7 5.879392 18 8 5.518071 19 9 6.492925 20 10 6.638523 

the second half of which was generated by rnorm(10,5,3), and which overall looks like this:

This dataset seems to [re]produce the "paradox" from the paper well enough, because while second-half subset yield a non-significant regression fit:

Residuals:
Min      1Q  Median      3Q     Max
-3.2223 -0.2383  0.2950  1.1233  2.1095

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)  5.15365    1.26463   4.075  0.00356 **
x            0.03335    0.20381   0.164  0.87409
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.851 on 8 degrees of freedom
Multiple R-squared:  0.003335,  Adjusted R-squared:  -0.1212
F-statistic: 0.02677 on 1 and 8 DF,  p-value: 0.8741


the combined dataset yields a significant one

Residuals:
Min      1Q  Median      3Q     Max
-3.0732 -1.1432 -0.2372  1.4106  3.7197

Coefficients:
Estimate Std. Error t value Pr(>|t|)
(Intercept)   2.5768     0.9256   2.784  0.01225 *
x             0.5167     0.1492   3.463  0.00277 **
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

Residual standard error: 1.916 on 18 degrees of freedom
Multiple R-squared:  0.3999,    Adjusted R-squared:  0.3666
F-statistic:    12 on 1 and 18 DF,  p-value: 0.002772


So how would you best describe this phenomenon where an independent variable is a good explanatory one for a sizeable part of the data, but not for the remainder? You can obviously spin this result to fit your prior bias.

• Could you clarify what is surprising here? One could make a philosophical argument that this is true of most statistical analyses. E.g. not all smokers develop lung cancer - more likely, smoking has an effect only in subgroups that are already frail because of genetic factors, bad health status, environment... We just don't know exactly which subgroups are those, and thus have to rely on marginal effects. – juod Nov 27 '17 at 16:43
• @juod: I don't find it surprising. As I was typing my "replication" experiment in R I was 95% convinced I'd get what I've got in the end. I'm mainly asking how is phenomenon usually called? "All statistics works like this"? – SX welcomes ageist gossip Nov 27 '17 at 16:58
• I see. In biostatistics, a likely phrasing for such results would be "x interacts with group", or "the effect of x is moderated by group", where group is some variable identifying those two strata. (See also en.wikipedia.org/wiki/Moderation_(statistics)). I'm not aware if other fields have a more general term for this though. – juod Nov 27 '17 at 17:09
• This seems to be very much like Simpson's paradox. – Michael R. Chernick Nov 27 '17 at 23:31