I'm currently working to develop a model which predict a traveler's choice of transportation mode (transit, auto, bike, walk) using data from the U.S. Census which has been aggregated to custom origin/destination zones. Thus, the data available to me is not traveler specific but instead contains variables representing the characteristics specific to each zonal pair (pop density, modal travel time, etc.) as well as the total number of traveler between each pair and the modal split for that pair. See below for a toy example of the data,
Zone1 Zone2 Zone1_Var1 Zone2_Var1 TT_Mode1 TT_Mode2 Tot_Trips Auto_Trips Transit_Trips 1 2 123 456 55 45 100 75 25
I'm specifically interested in estimating the proportions of each trip (e.g. from above .75 and .25). I had considered using a multinomial (for all four modes) logistic regression to estimate the proportions of each mode. To do so, I expanded the data so that there was an observation for each trip in each zonal pair with a variable indicating the mode chosen (taken from the percentage of total trips) to pass into the
multinom function from the
nnet package in R. For example (using data from above),
ID Zone1 Zone2 Zone1_Var1 Zone2_Var1 TT_Mode1 TT_Mode2 Mode 1 1 2 123 456 55 45 Auto 2 1 2 123 456 55 45 Auto 3 1 2 123 456 55 45 Auto ... 76 1 2 123 456 55 45 Transit 77 1 2 123 456 55 45 Transit ... 100 1 2 123 456 55 45 Transit
mn_fit <- multinom(as.formula(paste("mode", "~", paste(predictors, collapse = " + "))), mode_choice_list$train_mode_choice, max_iter = 200, trace= T)
However, this produces the interesting scenario where validating the model on a test set of the data leads it to predict the same mode (when asked to classify) for all zonal pairs since the variables do not vary within a zonal pair. Alternatively, I can have the model return the predicted probabilities for each mode which should be analogous to the observed modal splits.
This leads to my question: what is the best way to evaluate a set of predicted probabilities vs. a set of observed probabilities (ie modal splits). I've looked into scoring rules (e.g. brier, logarithmic, etc. ) and they seem promising. However, I'd like to confirm that utilizing these rules in evaluating various specifications of the mode choice model (e.g. different variables, variable transformations, etc.) is a valid route to take. If they are, are they able to account for the frequency weights of each zonal pair (i.e. there are a different number of trips between each zonal pair) which I would assume should be accounted for when evaluating performance.
Additionally, I'd welcome any feedback regarding my structuring of the data or use of the multinom package if it is incorrect or on another class of model that might be more suited to the data/problem at hand.