Problems with interpretation of a CLASS variable with many levels in a logistic regression model

Question

Thank you in advance.

I am trying to build a model to predict the possibility of ordinal accident levels in a nuclear station if an accident happens. The accidents include four levels: Site_Area_Emergency, Alert, Unusual Event and Non_Emergency (In order of from Strongest to Weakest, Non Emergency is the weakest). I　have three variable as potential predictor variables: The Age of that power station and the Facility_or_Organization type of that power station and power output. The age variable and power output are numerical variable. The Facility_or_Organization is obviously a nominal variable (like Kansas, New York, etc.)

Therefore, based on the stuff I described above, I chose to build a Ordinal Logistic Model. I use the backward selection method. It turns out that only age and Facility_or_Organization type matter.

My report is attached as a pdf file. It is a pdf file and quite long, therefore, I uploaded it to a share website, here is the link. Sorry for disturbing.

http://minus.com/mvsqZH/43f

My question is that: the overall degree of freedom is 67 because we have 65 different Facility_or_Organization types. The probability > chi_square for facility overall is quite small like 0.0001 but for each Facility_or_Organization type it is quite high. I took screen shots for you to have a look.

My Question:

May be you already noticed that in the second last table called "Type 3 analysis of Effects". The Pr>ChiSq for Facility_Or_Organization is <0.001. However, in the last table called "Analysis of Maximum Likelihood Estimate" the Pr>ChiSq for each Fasicility is quite high around 0.96. For example, for first Facility ARKANSAS NUCLEAR, the Pr>ChiSq is 0.9670. I just want to know why the Pr>ChiSq for whole Facility_Or_Organization is quite low but for each of them it is high?

Answer 1

First of all, you shouldn't use backwards selection (nor forwards nor stepwise). The p-values will be too small, standard errors too small, parameter estimates biased away from 0. Not good.

Second, I notice that the facility_or_organization variable has 65 DF, indicating that you had 65 different facilities. It also looks like you had two incidents per facility - this is drastic overfitting

Third, in the pdf, note the ominous warnings Warning: The maximum likelihood estimate may not exist. Warning: The LOGISTIC procedure continues in spite of the above warning. Results shown are based on the last maximum likelihood iteration. Validity of the model fit is questionable.

SAS' notes are often ignorable, it's warnings rarely are.

Finally, to properly fit this model, you have to either forget about the facility variable, or group the facilities somehow, so the model isn't overfit.