I have trained two classifiers namely a logistic regression and a decision tree on a data set. When evaluating both models on the testing data set, the decision tree has a ROC AUC = 0.62 but the logistic regression has ROC AUC = 0.91. However both models have similar gain chart and the lift for the first decile is almost the same and is 5.7 and this is almost the highest lift I can get considering my response rate (~17%). How come that is possible and what it means?
While the interpretation of the AUC is something you should look up as it's widely available, in cases when you want a deeper analysis of your model, it is a wise choice to plot your ROC curve and not only calculate its area. It can give you insights on how your model is working (for example, your logistic regression model may have a sweet spot where it works well but the other thresholds perform horribly, which would bring the AUC down).
An alternative reason could be underfitting of the data. While it is quite unlikely to happen with decision trees, it could still be the case if some of the parameters were chosen poorly.
Underfitting is the exact opposite of overfitting: the capacity of your model is too low to fit the training data at all and your model under-performs. It generalizes very well and underperforms on the test data as well. When people make statements such as "avoid overfitting", what they mean is actually to try to minimize the gap between the training and test error rates, sometimes referred as "generalization gap". There is a good explanation of this effect in (Goodfellow et al., 2016) see for instance figure 5.3.
If you get a poor fit on the training data you will of course get poor results on the test data too. Before drawing any conclusion about the performance of the test data, remember to always check the fit on the training data too.
(Goodfellow et al., 2016) Ian Goodfellow, Yoshua Bengio and Aaron Courville. Deep Learning. MIT Press, 2016.