1
$\begingroup$

I have a dataset with 53k samples and 12 predictors, divided in this way:

  • 15k - Group 1
  • 10k - Group 2
  • 8k - Group 3
  • 7k - Group 4
  • 4k - Group 5
  • 4k - Group 6
  • 2k - Group 7
  • 2k - Group 8
  • 1k - Group 9
  • 1k - Group 10

The dependent variable is binary (0/1) and the groups are in decreasing order of likelihood of Y=1, basically. So the distribution of Y is skewed towards 0s in the lower groups and 1s in the higher groups (for example: G1 has 12k 1s and 3k 0s).

After running a Logistic Regression on each group separately and the entire dataset, these are the coefficients and p-values for predictor P:

               Coefficient      p-value
Group 1           0.4392         0.0905
Group 2           0.1923         0.1333
Group 3           0.1222         0.6540
Group 4           0.101          0.5599
Group 5          -0.353          0.7000
Group 6          -0.005          0.6889
Group 7          -0.0208         0.9343
Group 8           0.184          0.9268
Group 9           0.2722         0.4466
Group 10          0.1009         0.3230
Overall           0.8842          4*e^-16

Now, I've never had to run a Logistic Regression on subsets of a dataset and the entire dataset, but is it possible that the coefficient is so strong and significant for the whole dataset while being insignificant in all the other groups?

My intuition is yes, because there's more data, but anyone have any ideas?

The code for the Logistic Regression is very basic:

glm.fit.1 <- glm(Y~p1 + p2 + p3 + p4 + p5 + p6 + p7 + p8 + p9 + p10 + p11 + p12, data=group1, family=binomial)

Repeated for each group + entire dataset

$\endgroup$
  • 1
    $\begingroup$ I don't understand your output- are those coefficients for one of the predictors across the 10 groups? Which predictor? $\endgroup$ – atiretoo Jul 31 '17 at 23:53
  • $\begingroup$ Hi. Yes, coefficients. And I'll edit the post to make it clearer. And does the predictor matter? It's more of a general question I guess $\endgroup$ – AMC Aug 1 '17 at 2:06
  • $\begingroup$ Well, which other predictors are in the model affects the estimate of a coefficient, but I don't think that would change the effect of decreasing sample size. The sample size argument is basically correct, but I'm surprised that a predictor with a strong effect with 53K datapoint is really not signifiant at 15K data points. The other possibility is that there is an interaction between the predictor and group membership, or between another predictor and group membership. Essentially the model for each group is fitting interactions between group and each predictor. $\endgroup$ – atiretoo Aug 1 '17 at 2:17
  • $\begingroup$ Sorry, forgot to say, they all have the exact same predictors. It is definitely possible that there is an interaction between Group and P. Higher Groups (from top to bottom) have higher values/more non-0s for P. I'm not necessarily an expert, but I can't see anything else wrong with this simple model, and I got the same basic results from two different sources (R and Python). My theory is that the samples aren't evenly distributed within each group, so the p-value is deemed statistically insignificant until there's enough data points, but this is more logic than stats based $\endgroup$ – AMC Aug 1 '17 at 3:17
4
$\begingroup$

When you estimate the effect of p separately within each group, that's a fundamentally different test from estimating it once across the whole dataset. In the second case, you're just looking at the main effect of p in a model that doesn't include an interaction between p and group, forcing one estimate on the whole dataset (even though your subset analyses suggest that there may indeed be an interaction between p and group).

Here are some pictures to illustrate. I'm just plotting it as though it were an OLS regression, since the underlying issue is the same (if you like, imagine the y-axis is logits if you want to think about it in terms of logistic regression).

This first plot shows the relationship between outcome y and predictor x estimated separately in each of the three groups. You can see some times it appears positive (group c), some times negative (group a), some times it looks like there's no effect at all (group b).

interaction

This plot shows the same data, but now the effect of x is estimated on all of the points at once. Suddenly it looks like a clear, positive effect!

no interaction

Of course, it's not always the case that effects will come out stronger when you look at the data all together vs. by group. You can just as easily see a clear effect within each group that disappears overall. The point is that you're estimating very different things when you look at the data by group vs. all together. There's no reason to expect that you should see the same pattern of results.

Here's the code to generate the above plots in R:

set.seed(24601)

group <- rep(c(1,2,3), each=50)
x <- c(1:50, 21:70, 41:90)
y <- ifelse(group == 1, x*-.5 + rnorm(50, sd = 15),
            ifelse(group == 2, rnorm(50, sd = 15),
                   ifelse(group == 3, x*.5 + rnorm(50, sd = 15), NA)))
my_data <- data.frame(group=factor(group, labels = c("a", "b", "c")), x, y)

# plot 1
ggplot(my_data, aes(y=y, x=x, color = group, fill = group)) + 
  geom_point(alpha = .5) + 
  stat_smooth(method = "lm") +
  theme_bw()

# plot 2
ggplot(my_data, aes(y=y, x=x, color = group, fill = group)) + 
  geom_point(alpha = .5) + 
  stat_smooth(aes(group= 1), method = "lm", color = "black") +
  theme_bw()
$\endgroup$
  • 1
    $\begingroup$ Fantastic visuals. $\endgroup$ – Matthew Drury Aug 1 '17 at 4:10
  • $\begingroup$ This is amazing, thank you so much! Made it really easy to understand and the visuals are awesome. My hunch was that they were completely separate things, but because they are technically "the same" it was really confusing. And I have never read anything about running regressions on subsets v a full dataset. Thanks again. Also thanks for the plot code! $\endgroup$ – AMC Aug 1 '17 at 12:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.