# Evaluation of Predicted Probabilities from Multinomial Logistic Regresion

I'm currently working to develop a model which predict a traveler's choice of transportation mode (transit, auto, bike, walk) using data from the U.S. Census which has been aggregated to custom origin/destination zones. Thus, the data available to me is not traveler specific but instead contains variables representing the characteristics specific to each zonal pair (pop density, modal travel time, etc.) as well as the total number of traveler between each pair and the modal split for that pair. See below for a toy example of the data,

Zone1 Zone2 Zone1_Var1 Zone2_Var1 TT_Mode1 TT_Mode2 Tot_Trips Auto_Trips Transit_Trips
1     2     123        456        55       45       100       75         25


I'm specifically interested in estimating the proportions of each trip (e.g. from above .75 and .25). I had considered using a multinomial (for all four modes) logistic regression to estimate the proportions of each mode. To do so, I expanded the data so that there was an observation for each trip in each zonal pair with a variable indicating the mode chosen (taken from the percentage of total trips) to pass into the multinom function from the nnet package in R. For example (using data from above),

ID Zone1 Zone2 Zone1_Var1 Zone2_Var1 TT_Mode1 TT_Mode2 Mode
1  1     2     123        456        55       45       Auto
2  1     2     123        456        55       45       Auto
3  1     2     123        456        55       45       Auto
...
76 1     2     123        456        55       45       Transit
77 1     2     123        456        55       45       Transit
...
100 1     2     123        456        55       45       Transit


Model call:

mn_fit <- multinom(as.formula(paste("mode", "~", paste(predictors, collapse = " + "))), mode_choice_list\$train_mode_choice, max_iter = 200, trace= T)


However, this produces the interesting scenario where validating the model on a test set of the data leads it to predict the same mode (when asked to classify) for all zonal pairs since the variables do not vary within a zonal pair. Alternatively, I can have the model return the predicted probabilities for each mode which should be analogous to the observed modal splits.

This leads to my question: what is the best way to evaluate a set of predicted probabilities vs. a set of observed probabilities (ie modal splits). I've looked into scoring rules (e.g. brier, logarithmic, etc. ) and they seem promising. However, I'd like to confirm that utilizing these rules in evaluating various specifications of the mode choice model (e.g. different variables, variable transformations, etc.) is a valid route to take. If they are, are they able to account for the frequency weights of each zonal pair (i.e. there are a different number of trips between each zonal pair) which I would assume should be accounted for when evaluating performance.

Additionally, I'd welcome any feedback regarding my structuring of the data or use of the multinom package if it is incorrect or on another class of model that might be more suited to the data/problem at hand.

"I'm specifically interested in estimating the proportions of each trip (e.g. from above .75 and .25)"

This does not seem clear to me. Why are you interested in estimating the proportion of trips taken when you already have this data? It would seem to me that the real question is "what is the impact of pop. density, modal travel time, etc. on the probability of a given transportation option being chosen." To solve this you can use multinomial logistic regression.

Do you have information on the distribution of your independent variables? If you do, my suggestion would be to generate a large random sample using this information, and generating set of target outcomes using the proportions that you already know. You can then train a cross validated model on this information to estimate the effect each of your variables has on the probability of each mode of transportation being chosen.

It is important to note that mlogit models utilize relative risk ratios (the exponentated coefficients) and not odds ratios. That is, your coefficients will tell you the change in probability that a unit change in your variable has on one outcome occurring over the base category you specify. I highly recommend reading the ats example here.

• Thanks for the quick answer, I'm interested in estimating a model so I can provide it with new (future) values for the independent values specific to each zonal pair and apply the estimated proportions to a future number of total trips resulting from another model. While the effects of the ind. variables in this would likely be interesting, I'm more concerned with predictive power. I guess my question boils down to: what is the evaluation method I should use during the CV you mentioned in order to determine which identify the best performing model. – Jack897 Jan 25 '17 at 19:48
• Also, I appreciate the last note on the coefficients. I hadn't realized they weren't odds ratio. – Jack897 Jan 25 '17 at 19:49
• I would try different models and see what works best for your data. Multinomial logistic regression is an option, and you could also try linear (or quadratic) discriminant analysis, multinomial naive bayes, neural nets, multiclass SVM extensions, and others. If you aren't restricted to using R I would check out scikit-learn in Python. You can structure your code so that preliminary CV fitting and predicting can be done very quickly on many algorithms. Since you are doing classification it is relatively simple to compare models based on accuracy, precision, recall, and overall F1 scores. – James Steele Jan 25 '17 at 20:39
• I'm aware of the various multinomial models available. As I said in my previous post, I'm interested in assessing the model's ability to predict proportions. Accuracy, precision, recall, etc. all require an actual prediction to be made so that they can be computed. For example, accuracy is calculated as (TP+TN)/N, where N is the number of data points. I can't compute these using just the predicted probabilities, which is why I was interested in scoring rules as a measure of model performace. – Jack897 Jan 25 '17 at 21:48
• You can compute the metrics using the predicted probabilities. Just assign a category for whichever predicted probability is highest. If that doesn't suit you, you can manipulate your decision rule which defines what probability is required for an observation to be assigned to the category. That is the idea behind ROC curves, though i'm not sure if/how this is possible with mlogit. – James Steele Jan 26 '17 at 19:39