2
$\begingroup$

I run a backward variable selection logistic regression and find out the SAS program selected 12 variables and give me the output like this:

enter image description here enter image description here enter image description here enter image description here enter image description here

It is funny that my configuration of the SAS backward selection is to make the SC(SBC) lower. Overall, the result looks like the SC result suppose that intercept with covariates is worse. But other two guys AIC and -2logL supports that Intercept with corvar is better model.

May you guys make some judge ment on this result and test output?

Improve this question
$\endgroup$
8
  • 1
    $\begingroup$ Have you checked out the answers to this question - stats.stackexchange.com/questions/3984/aic-and-sc-value? $\endgroup$
    – Peter Ellis
    Commented Mar 24, 2012 at 0:54
  • $\begingroup$ @PeterEllis, I just checked. I am doing predicting model, so my model is good with those covariate....? And I　find a page interpreting that aic and sc are kind of "indicators" but checking the model with -2 log likelihood and degree of freedom improvement is better decision making. ats.ucla.edu/stat/sas/library/comp_repeated.htm $\endgroup$
    – Wenhao.SHE
    Commented Mar 24, 2012 at 1:38
  • $\begingroup$ @Wenhao.SHE I would strongly suggest that you, as the human, select the variables that go into the model based on logic and theory. The best a machine can do is data mine. Stepwise selection methods are notorious for capitalizing on sample idiosyncrasies. As for your fit statistics, have you considered looking at something like the area under the ROC curve (c-statistic)? I find it's a much more intuitive measure of model fit. $\endgroup$
    – Will
    Commented Mar 24, 2012 at 2:17
  • $\begingroup$ @Will, I just post the ROC curve. May you have a look?> I try to predict whether tomorrow a forex pair is up or down. as you can see, the variable is frequecy of key trading related words. It makes sense for human, right? May you give more comments after viewing the roc? Thanks $\endgroup$
    – Wenhao.SHE
    Commented Mar 24, 2012 at 2:23
  • $\begingroup$ @Wenhao.SHE, I see. There isn't any sort of theory here, this really is just data mining. I would use the AIC in this situation. As to the c-statistic, a value of .5 means your model's predictions are no better than chance while a value of 1 would mean perfect prediction. It's just a nice intuitive way for humans to pick one model over another. For your purposes it probably isn't as useful as the AIC. This isn't my subject area but I would hope for a c stat higher than .75. $\endgroup$
    – Will
    Commented Mar 24, 2012 at 3:06

1 Answer 1

Reset to default
1
$\begingroup$

Looking at the SC in conjunction with -2logL is suggesting that your model is overparametised. That is, you can obtain a simpler model with less parameters but still a good fit to the data. You will get a better SC if you set the slstay= option equal to the quantile of a chi-square with 1 degree of freedom at the difference between SC and -2LogL for the intercept only model. For your output we have a difference of $447.224-440.919=6.305$ (from which I infer your sample size is $\exp(6.305)=547$). Now we set the p-value equal to $Pr(\chi^2_1>6.305)=0.012$ (more generally set it to $Pr(\chi^2_1>\log(n))$. This ensures that every model selection step will decrease the SC (approximately) without having to manually re-fit each model. If you haven't specified the slentry= option then it defaults to $0.05$. Thus my prediction is that you have some variables with p-values less than $0.05$ but greater than $0.012$.

Improve this answer
$\endgroup$
16
  • $\begingroup$ I just updated my question with image showing that you are definitely right that I have some variable with p-value > 0.12. So, what is your suggestion? lower the stay from 0.05 to somewhere below 0.02? or even 0.01? I also post roc and lift graph. May you have a look. You are definitely a master of stat. Admiring...poor me $\endgroup$
    – Wenhao.SHE
    Commented Mar 24, 2012 at 2:21
  • $\begingroup$ I read your comment again and I guess the right choice of stay should be 0.12 to lower the sc down to be consistent with -2loglikelihood result, right? I just tried and the sc just fall down below the 447.244. And the model is significanlly simplified now. SHould I say it is a good model now with roc and lift chart I added? $\endgroup$
    – Wenhao.SHE
    Commented Mar 24, 2012 at 2:37
  • $\begingroup$ 0.012 not 0.12 you missed a decimal $\endgroup$
    – probabilityislogic
    Commented Mar 24, 2012 at 2:38
  • $\begingroup$ Sorry for my careless.....Thanks again. It should be 0.012. Now the sc and -2 log shows same direction of indication. May I say it is a good modeling? I try to predict the up and down of a forex pair in next day by frequency of keywords on a forum of forex. The words left is the Intercept "problem" "game" and "direction". $\endgroup$
    – Wenhao.SHE
    Commented Mar 24, 2012 at 2:48
  • $\begingroup$ @probabilityislogic, can I just ask (from someone who clearly knows more than I do) why does it matter if a simpler model could work just as well in this case? If the goal is merely prediction and not understanding then problems like collinearity go away. Who cares how we partial out the variance when all that matters is that the variance is partialled out. I say this not to be be abrasive but because I assume there must be something I'm missing here. $\endgroup$
    – Will
    Commented Mar 24, 2012 at 3:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.