# Assessing predictive validity of choice models

I am trying to assess predictive validity of a discrete choice model. When with a testing set consisting of 6 choices in each set, it has the hit ratio of 33% (i.e., 33% of the time it correctly guesses the actual choice in the testing set).

How good or bad is it? Understandably, if the model was tested on sets of 2 choices that would be bad (because we have 50% chance of guessing at random). Conversely, if tested on 100 choices in a set, that would be great.

• How good or bad it is depends on the question you ask. Personally, i would say this is pretty bad. Jan 28, 2017 at 2:59

From the information you provide, one can't determine how good or bad this level of predictive validity is. Suppose 45% of your subjects had selected Choice A. Using your current metric, one could far outperform the model you describe by simply predicting Choice A for all subjects, thereby yielding a "hit rate" of 45%.

Correct Classification Rate is a very intuitive but very overrated method of assessing predictive accuracy. You can find information on other methods at threads such as the following: 1, 2, and 3.

• Thanks rolando2, I think I can avoid the first problem you describe by randomising the order of choices, so it is impossible to "just select choice A". Jan 28, 2017 at 22:19
• Also, there are a few of these methods described int he links. Which ones would you recommend for choice data? Jan 28, 2017 at 22:20

I don't think it will be ever possible to tell whether a given level of predictive validity is "good" or "bad". But at the very least you could try to determine whether the predictive validity of your model is significantly better than what you would obtain by chance.

For example, you could follow a cross-validation procedure:
(1) randomly split your sample into 2 groups;
(2) estimate your model on one group;
(3) use model estimates to predict choices by the other group;
(4) compare observed and predicted choices => This would give you a measure of predictive validity;
(5) repeat this procedure a large number of times => This would give you a distribution for the predictive validity;
(6) check whether the chance level is within the 95% confidence interval).