3
$\begingroup$

I am validating a logistic regression model. This is the first time i am validating a model. I am using split sampling method. I have split data randomly into two parts - 70% development and 30% validation data sets (70:30). Then i run logistic regression on development data set using SAS and rank their probabilities in descending order and split data into 10 groups (deciles). Check the percentage of responses in the upper deciles. My question - Is there any thumb rule to assess the model on the basis of percentage of responses in the upper 3-4 deciles? Someone said top 3 deciles should cover atleast 65% of responses. Is it correct? I have checked Hosmer and Lemeshow Goodness-Fit-Test.

I have formulated an equation including intercept and coefficients derived from development dataset and run it on validation dataset. The code is shown below for reference -

Data validation_Output; Set validation_Set; resp_xb1= -1.3844+(1.4708)A_Flg+(2.9829) B_Flg+(.0317)* C + (-0.2372)*D +(-0.3359)*E; exppres1=exp(resp_xb1); p_resp1=exppres1/(1+exppres1); run;

Then run PROC RANK and PROC SQL to calculate deciles on validation dataset.

I have decile scores on development and validation sets. Should the significant variables in both the datasets be same? OR Concordant?

$\endgroup$
6
$\begingroup$

Binning of continuous variables is never the answer. Your post raises a huge number of issues. Many of these are covered in my handouts at http://biostat.mc.vanderbilt.edu/CourseBios330. Note that the Hosmer-Lemeshow test is no longer competitive with continuous calibration curve methods. Data splitting requires perhaps 15,000 observations to be reliable. Look at resampling methods in my handout.

$\endgroup$
2
$\begingroup$

When you're validating a model, you want to see how well the model's predictions match the validation data's actual outcomes. Start simple and work your way up from there. The first thing I would do is compare the ratio of actual to predicted outcomes (assigning one outcome 0 and the other 1 so that you can calculate a numerical measure).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.