How to validate and compare models predicting a binary variable?

Question

I have a question about determining which models are "better" and how to assess that info.

Let's say I have three models, each which predicts our bid on won ping. Our bid is a continuous variable and won ping is either 0 (no) or 1 (yes). From the models (logit) and the system that I'm modeling, I constructed the following table with the number of bids, our win rate, predicted win rate, and average bid. Essentially, we run the model our bids and whether we win and use that to predict our win rate in the future bids.

            #_of_bids    win_rate    pred.win_rate    avg.bid      
    Mod_1       1792          46%       62%         $1.54    
    Mod_2        851          2%	    2%         $0.98
    Mod_3       17037         6%         0%         $2.83 

According to the above table, there is a "significant" discrepancy between our win rate and the predicted win rate. As a result, I ran through the model and came up with the following "diagnostics."

Mod_1
    - Won ping rate = 45.7%
    - Bid distribution issues = politively skewed, but no single bid repeats significantly
    - Nagelkerke's R squared = 0.23 ->  The predictor only exlained 23% of the variance found in the 
      response variable (won ping)  ->  poor
    - Cross-Validation (10-fold) = 0.693  ->  how accurate were the model predictions (0 to 1)  
    - ROC Curve = 0.746 ->  fair(C)  ->  how well the model fits the data (0 to 1)

However, I was left wondering if the discrepancy between the win rate and predicted win rate, plus the validation statistics were enough to conclude that only that model was off (aka performed poorly or didn't predict the actual win rate properly). Therefore, I ran through the same process for the other two models and came up with the following stats.

Mod_2   
    - Won ping rate = 2.3%
    - Bid distribution issues = 53.5% of our bids are $0.26 or $0.26
    - Nagelkerke's R squared = 0.40 ->  The predictor only exlained 23% of the variance found in the 
      response variable (won ping)  ->  average
    - Cross-Validation (10-fold) = 0.974  ->  how accurate were the model predictions (0 to 1)  
    - ROC Curve = 0.914 ->  excellent(A)  ->  how well the model fits the data (0 to 1)

Mod_3
    - Won ping rate = 5.7%
    - Bid distribution issues = 12.9% of our bids were either $0.25 or $0.26
    - Nagelkerke's R squared = 0.03 ->  The predictor only exlained 3% of the variance found in the 
      response variable (won ping)  ->  horrible
    - Cross-Validation (10-fold) = 0.942  ->  how accurate were the model predictions (0 to 1)  
    - ROC Curve = 0.339  ->  horrible  -> how well the model fits the data (0 to 1)

So in trying to figure out if it was just model 1 that was off, I looked at these various statistics. However, I'm not sure about what the proper steps are.

Questions

1. Should I assume that because the win rate and predicted win rates were so different in the first table that the model performed poorly?
2. While the validation stats for model 1 are poor, they are not much better for model 2 or 3. What explains that result?
3. How should one go about comparing two or three different models to one another in terms of overall performance (aka fits the data/predictive quality)?

Answer 1

If you are only concerned about making a decision (did I win or not?), the best model comparison is the misclassification rate. The misclassification rate is a measure of the performance of the model on a validation data set. Simply apply the model to the validation data set (a holdout sample not used to build the model) and count how many times the model got it wrong.

However, if you are more concerned with rankings (determining what to bid on) you should use the Gini Coefficient which is closely related to the ROC index but is used for a binary prediction.