Logistic regression is predicting all 1, and no 0 I am running an analysis on the probability of loan default using logistic regression and random forests.  
When I use logistic regression, the prediction is always all '1' (which means good loan).  I have never seen this before, and do not know where to start in terms of trying to sort out the issue.  There are 22 columns with 600K rows.  When I decrease the # of columns I get the same result with logistic regression. 
Why could the logistic regression be so wrong?
**Actual from the data**

0 :   41932

1 :   573426

**Logistic regression output** 

prediction for 1 when actually 0: 41932
prediction for 1 when actually 1:573426

A**s you can see, it always predicts a 1**


**Random forests does better:**

actual 0, pred 0 : 38800 
actual 1, pred 0 : 27 
actual 0, pred 1 : 3132
actual 1, pred 1 : 573399

 A: Well, it does make sense that your model predicts always 1. Have a look at your data set: it is severly imbalanced in favor of your positive class. The negative class makes up only ~7% of your data. Try re-balancing your training set or use a cost-sensitive algorithm.
A: If the problem is indeed the imbalance between the classes, I would simply start by balancing the class weights: 
log_reg = LogisticRegression(class_weight = 'balanced')

This parameter setting means that the penalties for false predictions in the loss function will be weighted with inverse proportions to the frequencies of the classes. This can solve the problem you describe. 
A: When you classify using logit, this is what happens. 
The logit predicts the probability of default (PD) of a loan, which is a number between 0 and 1. Next, you set a threshold D, such that you mark a loan to default if PD>D, and mark it as non-default if PD

Naturally, in a typical loan population PD<<1. So, in your case 7% is rather high probability of it's one year data (PDs are normally reported on annual basis). If this is multi year data, then we're talking about so called cumulative PD, in this case cumPD=7% is not a high number for 10 years of data, for instance. Hence, by any standards, I wouldn't say that your data set is problematic. I'd describe it at least typical for loan default data, if not great (in the sense that you have relative large number of defaults).
Now, suppose that your model predicting the following three levels of PD: 


*

*0.1 (563,426)

*0.5 (20,000)

*0.9 (31,932)


Suppose also that the actual defaults for these groups were: 


*

*0

*10,000

*31,932


Now you can set D to different values and see how the matrix changes. Let's use D = 0.4 first:


*

*Actual default, predict non-default: 0

*Actual default, predict default: 41,932

*Actual non-default, predict non-default: 563,426

*Actual non-default, predict default: 10,000


If you set D = 0.6:


*

*Actual default, predict non-default: 31,932

*Actual default, predict default: 10,000

*Actual non-default, predict non-default: 573,426

*Actual non-default, predict default: 0


If you set D = 0.99:


*

*Actual default, predict non-default: 41,932

*Actual default, predict default: 0

*Actual non-default, predict non-default: 573,426

*Actual non-default, predict default: 0


The last case is what you see in your model results. In this case I'm emphasizing the threshold D for a classifier. A simple in change in D may improve certain characteristics of your forecast. Note, that in all three cases the predicted PD remained the same, only the threshold D has changed.
It is also possible that your logit regression itself is crappy, of course. So, in this case you have at least two variables: the logit spec and the threshold. Both impact your forecast power.
A: The short answer is that logistic regression is for estimating probabilities, nothing more or less.  You can estimate probabilities no matter how imbalanced $Y$ is.  ROC curves and some of the other measures given in the discussion don't help.  If you need to make a decision or take an action you apply the loss/utility/cost function to the predicted risk and choose the action that optimizes the expected utility.  It seems that a lot of machine learning users are not really understanding risks and optimum decisions.
A: Well, without more information its hard to say, but by the definition of logistic regression you are saturating based on the fitted data. So in the equation the e^-t term is going to 0. So the first place to look would be to see what the actual coefficients are. 
This could also be due to poorly scaled variables. There might be an issue where one of the columns is huge in numerical value compared to others that is causing it mess up.
A: You may use SMOTE to balance the unbalanced dataset. A good paper for reference is:
Lifeng Zhou, Hong Wang, Loan Default Prediction on Large Imbalanced Data Using Random Forests, TELKOMNIKA Indonesian Journal of Electrical Engineering, Vol.10, No.6, October 2012, pp. 1519~1525, link.
