1
$\begingroup$

I'm facing this issue with a logit model with R. I got my coefficients significant (already removed the Linear dependent variabiles and the not significant variables), as:

Call:
glm(formula = k ~ a + b + c + d + e + f + g + h + i + j, 
    family = binomial(link = "logit"), data = bz)

Deviance Residuals: 
     Min        1Q    Median        3Q       Max  
-2.33397  -1.11978   0.06133   1.12395   2.47743  

Coefficients:
                    Estimate Std. Error z value Pr(>|z|)    
(Intercept)        -0.442743   0.037655 -11.758  < 2e-16 ***
a                  -0.042182   0.002231 -18.911  < 2e-16 ***
b                   0.514025   0.037674  13.644  < 2e-16 ***
c                  -2.640015   0.166331 -15.872  < 2e-16 ***
d                   1.505434   0.090759  16.587  < 2e-16 ***
e                   1.503102   0.096854  15.519  < 2e-16 ***
f                  -1.262869   0.116334 -10.856  < 2e-16 ***
g                   0.745737   0.179957   4.144 3.41e-05 ***
h                   0.312694   0.021166  14.774  < 2e-16 ***
i                  -0.440660   0.032558 -13.535  < 2e-16 ***
j                   0.773453   0.036602  21.131  < 2e-16 ***
---
Signif. codes:  0 ‘***’ 0.001 ‘**’ 0.01 ‘*’ 0.05 ‘.’ 0.1 ‘ ’ 1

(Dispersion parameter for binomial family taken to be 1)

    Null deviance: 83178  on 59999  degrees of freedom
Residual deviance: 79092  on 59989  degrees of freedom
AIC: 79114

Number of Fisher Scoring iterations: 4

I've worked with a balanced dataset if it's important (i.e. 30,000 of 1, and 30,000 of 0, having only 30,000 1 and much more 0). Also, doing it with my full dataset, the numbers of 0 is so much high that the model detect perfectly them, but it's very bad to detect the 1 (if I'm doing something wrong, please tell me without problem).

I decided to see if it predicts well its dependent variable:

probabilities <- mod %>% predict(bz, type = "response")
predicted.classes <- ifelse(probabilities > 0.5, "1", "0")

prop.table(table(predicted.classes,bz$k))

predicted.classes         0         1
                0 0.3039000 0.1946333
                1 0.1961000 0.3053667

mean(predicted.classes == bz$millennials_01)
[1] 0.6092667

So it has many significant coefficients, but not too much good capabilities of prediction.

What does it mean?

$\endgroup$
3
  • 1
    $\begingroup$ Your sample size is HUGE. Knowing that a log-odds ratio is statistically different from 0 is trivial when the sample size is 10,000. Actually getting good predictions requires more careful thought and carefully specified objectives and methods. $\endgroup$
    – AdamO
    Commented Jun 24, 2019 at 18:34
  • $\begingroup$ @AdamO thanks for your comment. I've tried several attempts with 3,000 sample (X 2) and the result are almost identical. Something changes with 300(X2) sample, some of the p-values accept H0, but the prediction proportion are the same. Could you specify your sentence about getting good predictions? I'll gladly read it. $\endgroup$
    – s__
    Commented Jun 25, 2019 at 6:56
  • $\begingroup$ 1) P > 0.05 $\not \Rightarrow$ accept $H_0$. 2) inference is a function of sample size (power) 3) null hypothesis significance testing and prediction have no relation. Developing a good prediction model means achieving high out-of-sample calibration and discrimination. Search the last two terms for more clarity. $\endgroup$
    – AdamO
    Commented Jun 25, 2019 at 14:11

1 Answer 1

2
$\begingroup$

First, 61% accuracy isn't necessarily so bad. You're outperforming random guessing based on the prior distribution that would give about 50% accuracy. Perhaps you know that 61% is inadequate for your application, but don't dismiss 61% just because it is not as high as 99.9% like people have gotten on MNIST.

What it means, though, that your parameters are significant but your accuracy is poor is that you are not accounting for everything influencing class membership. Are there interactions between variables such as $a b$ or higher-order terms such as $a^2$? Are there additional covariates that were not included in your data but could influence the outcome, perhaps an $L$ variable?

As you read about improving machine learning performance, be careful about the difference between in-sample and out-of-sample performance. You have an issue of bias rather than variance. Consequently, while ridge, LASSO, or elastic net regularization may seem like appealing techniques to explore to boost performance, they are not what you seek.

$\endgroup$
2
  • $\begingroup$ Thanks for your answer (+1). How to detect relationship? My variables have high correlation, but the scatter is not "linear". Are you suggesting to put transformation of variables in the model like k = a + b+ .. + ab? Thanks. $\endgroup$
    – s__
    Commented Jun 25, 2019 at 6:59
  • $\begingroup$ Yes, it would be something like k = a + b + ... + i + j + a^2 + ab. Visual examination of your data should help guide which to include. $\endgroup$
    – Dave
    Commented Jun 25, 2019 at 10:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.