I have a question about determining which models are "better" and how to assess that info.
Let's say I have three models, each which predicts our bid on won ping. Our bid is a continuous variable and won ping is either 0 (no) or 1 (yes). From the models (logit) and the system that I'm modeling, I constructed the following table with the number of bids, our win rate, predicted win rate, and average bid. Essentially, we run the model our bids and whether we win and use that to predict our win rate in the future bids.
#_of_bids win_rate pred.win_rate avg.bid
Mod_1 1792 46% 62% $1.54
Mod_2 851 2% 2% $0.98
Mod_3 17037 6% 0% $2.83
According to the above table, there is a "significant" discrepancy between our win rate and the predicted win rate. As a result, I ran through the model and came up with the following "diagnostics."
Mod_1
- Won ping rate = 45.7%
- Bid distribution issues = politively skewed, but no single bid repeats significantly
- Nagelkerke's R squared = 0.23 -> The predictor only exlained 23% of the variance found in the
response variable (won ping) -> poor
- Cross-Validation (10-fold) = 0.693 -> how accurate were the model predictions (0 to 1)
- ROC Curve = 0.746 -> fair(C) -> how well the model fits the data (0 to 1)
However, I was left wondering if the discrepancy between the win rate and predicted win rate, plus the validation statistics were enough to conclude that only that model was off (aka performed poorly or didn't predict the actual win rate properly). Therefore, I ran through the same process for the other two models and came up with the following stats.
Mod_2
- Won ping rate = 2.3%
- Bid distribution issues = 53.5% of our bids are $0.26 or $0.26
- Nagelkerke's R squared = 0.40 -> The predictor only exlained 23% of the variance found in the
response variable (won ping) -> average
- Cross-Validation (10-fold) = 0.974 -> how accurate were the model predictions (0 to 1)
- ROC Curve = 0.914 -> excellent(A) -> how well the model fits the data (0 to 1)
Mod_3
- Won ping rate = 5.7%
- Bid distribution issues = 12.9% of our bids were either $0.25 or $0.26
- Nagelkerke's R squared = 0.03 -> The predictor only exlained 3% of the variance found in the
response variable (won ping) -> horrible
- Cross-Validation (10-fold) = 0.942 -> how accurate were the model predictions (0 to 1)
- ROC Curve = 0.339 -> horrible -> how well the model fits the data (0 to 1)
So in trying to figure out if it was just model 1 that was off, I looked at these various statistics. However, I'm not sure about what the proper steps are.
Questions
- Should I assume that because the win rate and predicted win rates were so different in the first table that the model performed poorly?
- While the validation stats for model 1 are poor, they are not much better for model 2 or 3. What explains that result?
- How should one go about comparing two or three different models to one another in terms of overall performance (aka fits the data/predictive quality)?