3
$\begingroup$

Thank you in advance.

I am trying to build a model to predict the possibility of ordinal accident levels in a nuclear station if an accident happens. The accidents include four levels: Site_Area_Emergency, Alert, Unusual Event and Non_Emergency (In order of from Strongest to Weakest, Non Emergency is the weakest). I　have three variable as potential predictor variables: The Age of that power station and the Facility_or_Organization type of that power station and power output. The age variable and power output are numerical variable. The Facility_or_Organization is obviously a nominal variable (like Kansas, New York, etc.)

Therefore, based on the stuff I described above, I chose to build a Ordinal Logistic Model. I use the backward selection method. It turns out that only age and Facility_or_Organization type matter.

My report is attached as a pdf file. It is a pdf file and quite long, therefore, I uploaded it to a share website, here is the link. Sorry for disturbing.

http://minus.com/mvsqZH/43f

My question is that: the overall degree of freedom is 67 because we have 65 different Facility_or_Organization types. The probability > chi_square for facility overall is quite small like 0.0001 but for each Facility_or_Organization type it is quite high. I took screen shots for you to have a look.

enter image description here

My Question:

May be you already noticed that in the second last table called "Type 3 analysis of Effects". The Pr>ChiSq for Facility_Or_Organization is <0.001. However, in the last table called "Analysis of Maximum Likelihood Estimate" the Pr>ChiSq for each Fasicility is quite high around 0.96. For example, for first Facility ARKANSAS NUCLEAR, the Pr>ChiSq is 0.9670. I just want to know why the Pr>ChiSq for whole Facility_Or_Organization is quite low but for each of them it is high?

Improve this question
$\endgroup$
2
  • $\begingroup$ Although a section begins "My question is," I cannot discern any question there. What do you need to know? $\endgroup$
    – whuber ♦
    Apr 14, 2012 at 1:18
  • 1
    $\begingroup$ @whuber Sorry, I edited it. May be you already noticed that in the second last table called "Type 3 analysis of Effects". The Pr>ChiSq for Facility_Or_Organization is <0.001. However, in the last table called "Analysis of Maximum Likelihood Estimate" the Pr>ChiSq for each Fasicility is quite high around 0.96. For example, for first Facility ARKANSAS NUCLEAR, the Pr>ChiSq is 0.9670. I just want to know why the Pr>ChiSq for whole Facility_Or_Organization is quite low but for each of them it is high? $\endgroup$
    – Wenhao.SHE
    Apr 14, 2012 at 1:25

1 Answer 1

Reset to default
2
$\begingroup$

First of all, you shouldn't use backwards selection (nor forwards nor stepwise). The p-values will be too small, standard errors too small, parameter estimates biased away from 0. Not good.

Second, I notice that the facility_or_organization variable has 65 DF, indicating that you had 65 different facilities. It also looks like you had two incidents per facility - this is drastic overfitting

Third, in the pdf, note the ominous warnings Warning: The maximum likelihood estimate may not exist. Warning: The LOGISTIC procedure continues in spite of the above warning. Results shown are based on the last maximum likelihood iteration. Validity of the model fit is questionable.

SAS' notes are often ignorable, it's warnings rarely are.

Finally, to properly fit this model, you have to either forget about the facility variable, or group the facilities somehow, so the model isn't overfit.

Improve this answer
$\endgroup$
4
  • $\begingroup$ "The p-values will be too small, standard errors too small, parameter estimates biased away from 0" - if the design matrix is orthogonal (granted, this is rarely satisfied, but it is an assumption of most regression models) is this still true? In that case, forward and backward selection will yield the same result, which is the same result as just testing all of the predictors at once and removing the non-significant ones. So, I'm thinking the answer is 'No', although I could be missing something. $\endgroup$
    – Macro
    Apr 18, 2012 at 13:43
  • $\begingroup$ @FrankHarrell can probably give you more detail on this than I can, it is his book Regression Modelling Strategies that got me interested in this problem. I don't have the book handy, but as I recall, he did not have to assume non-orthogonality to show the results above. However, in this paper David Cassell and I show an example which I believe has orthogonality and where problems still occur with forward and backward. $\endgroup$
    – Peter Flom
    Apr 18, 2012 at 16:25
  • $\begingroup$ Oops, I think that should be @Frank FrankHarrell, although I am not sure if it matters. $\endgroup$
    – Peter Flom
    Apr 18, 2012 at 16:37
  • 1
    $\begingroup$ Any variable selection will cause a distortion in all statistical quantities. But the issue may be also related to how arbitrary is the choice of the reference cell when looking at individual tests. The global test ("chunk test") is invariant with respect to reference cell choice. $\endgroup$
    – Frank Harrell
    Apr 19, 2012 at 1:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.