I am trying to assess predictive validity of a discrete choice model. When with a testing set consisting of 6 choices in each set, it has the hit ratio of 33% (i.e., 33% of the time it correctly guesses the actual choice in the testing set).

How good or bad is it? Understandably, if the model was tested on sets of 2 choices that would be bad (because we have 50% chance of guessing at random). Conversely, if tested on 100 choices in a set, that would be great.

  • $\begingroup$ How good or bad it is depends on the question you ask. Personally, i would say this is pretty bad. $\endgroup$ – itsnotgood Jan 28 '17 at 2:59

From the information you provide, one can't determine how good or bad this level of predictive validity is. Suppose 45% of your subjects had selected Choice A. Using your current metric, one could far outperform the model you describe by simply predicting Choice A for all subjects, thereby yielding a "hit rate" of 45%.

Correct Classification Rate is a very intuitive but very overrated method of assessing predictive accuracy. You can find information on other methods at threads such as the following: 1, 2, and 3.

  • $\begingroup$ Thanks rolando2, I think I can avoid the first problem you describe by randomising the order of choices, so it is impossible to "just select choice A". $\endgroup$ – k-zar Jan 28 '17 at 22:19
  • $\begingroup$ Also, there are a few of these methods described int he links. Which ones would you recommend for choice data? $\endgroup$ – k-zar Jan 28 '17 at 22:20

I don't think it will be ever possible to tell whether a given level of predictive validity is "good" or "bad". But at the very least you could try to determine whether the predictive validity of your model is significantly better than what you would obtain by chance.

For example, you could follow a cross-validation procedure:
(1) randomly split your sample into 2 groups;
(2) estimate your model on one group;
(3) use model estimates to predict choices by the other group;
(4) compare observed and predicted choices => This would give you a measure of predictive validity;
(5) repeat this procedure a large number of times => This would give you a distribution for the predictive validity;
(6) check whether the chance level is within the 95% confidence interval).


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.