Significance of variable but low impact on log likelihood?

Question

I'm currently working on a logistic regression, trying to find out how significant some independent variables considering the dichotomous outcome are.

I did some test runs for the logistic regression and wondering about the results that raise some questions about the validity of the results.

1. Is it possible that a variable has a strong significance in the model but is barely contributing to reducing the log likelihood value? As I got from Fields Discovering Statistics, a log likelihood somewhat indicates how much unexplained information is there. With my very small knowledge in logistic regression and statistics in general, I would assume that a higher significance of a variable would lead to a lower loglikelihood value and therefore less unexplained information. Or is there nothing wrong about having a significant variable but still a high log likelihood value?

2. As I mentioned before, I'm having trouble to judge whether the results are valid or not (with statsmodels in python):

   
   | Dep. Variable: Churner | No. Observations: 107262 |
   | Model: Logit | Df Residuals: 107258 |
   | Method: MLE | Df Model: 3 |
   | Date: Fri, 18 Mar 2016 | Pseudo R-squ.: 0.3798 |
   | Time: 01:12:22 | Log-Likelihood: -41705. |
   | converged: True | LL-Null: -67247. | | LLR p-value: 0.000 |

                          coef     std err     z       P>|z|   [95.0% Conf. Int.]
   const               -0.5324     0.019    -27.886    0.000     -0.570 -0.495
   currentBreak         0.0065     7.42e-05  87.507    0.000      0.006 0.007
   Frequency            0.0019     0.000     19.164    0.000      0.002 0.002
   numActiveCategories -0.1589     0.007   -24.117     0.000     -0.172 -0.146
   

Without having knowledge about the data preprocessing (which is a bit too much to explain it here), could you tell if the results are somewhat usual or are at some point absolutely contradictory?

Answer 1

I think there is nothing wrong.

Statistical significance in large samples can also occur if there is a tiny nonzero effect because the power to detect small differences increases. This does not mean that the effect is "important".

You didn't show by how much your loglikelihood value changes as you drop a variable. One issue is relative versus absolute change in the loglikelihood value. The Likelihood Ratio test for dropping a variable considers the change in the loglikelihood, which has a constant distribution (chisquare(1) for 1 restriction). However, for example, the pseudo rsquared is looking at relative changes in loglikelihood. So a one unit change in the loglikelihood might be relatively small or relatively large. In a large sample with a large negative loglikelihood the contribution of an additional variable might look relatively small.

The analogy to linear regression would be sum of squares versus R-squared.

A better way to look at the importance of explanatory variables is to look at the prediction and the change in prediction that a variable causes. First, the contribution of a variable cannot be seen from the coefficient if we don't know the scale and variance of the explanatory variable. Second, in the case of the logit model and other nonlinear model, the effect of a change in the explanatory variable depends on the curvature of the logit, or nonlinear, function and on the location where we evaluate the function. See margins to calculate the impact in discrete models.

Answer 2

Some points not discussed in the answer by @Josef. There might be nothing wrong, but with such a large sample size (107262) and logistic regression (LR) I am skeptical.

1. With LR there is always the possibility of overdispersion, and with large sample size there will often be a substructure to the data/sampling process, some kind of dependence, that might cause overdispersion. Tell us more about your data context, investigate.

2. You have very few model df. Your variables seem to be treated as numerical. Try some more flexible modeling, represent predictors via [regression splines](Logistic Regression with regression splines in R. Look at the residuals, show us some plots from your residuals analysis.

3. This might be relevant even with a huge sample size: The significance tests you have shown for individual coefficients are Wald tests, which is based on a quadratic approximation to the log likelihood. For LR that might be vary bad! this is known as the Hauck-Donner phenomenon. Compute confidence intervals via profile likelihood (in R the confint function does this, must be possible in Python also.) Maybe not very probable in your case ... but have a look.