I performed a logistic regression on my data where the dependent binary variable $Y$ have $0$ & $1$ values and the independent variables $X$ being binary as well as continuous. The regression results suggest that none of my three $X$ variables have $p<0.05$ which is very disappointing as I strongly believed them to be affecting the outcome of $Y$. Stratifying the data makes no difference as well. I can't help but think there might be some problem with my data. So I noticed that only $13\%$ of the observations $(n = 4080)$ actually have $1$ as $Y$ value. So a huge majority have $0$ and maybe this is causing the regression to show no statistically significant relation? Also all $X$ variables are different from each other so there is no multi colinearity. I'm new to statistics so I just want to make sure the data is not the issue here. Or is it that I was wrong all along and there might not be a relation after all. Any thoughts?
2 Answers
You're right, you should be slightly concerned about the power of your analysis. In general, 50% 1s and 0s will lead to more power.. but that's only half the story. 400 successes and 3600 failures is typically a LOT of power, depending on the distribution of the $X$ values.
Cross tabulate values of each $X$ variable and $Y$ as well (assuming $X$ is categorical as well). ``A chain is only as strong as its weakest link'' comes to mind: the power of the logistic regression model to detect an association will generally be greatly limited by the smallest cell count in the model.
This is because for a log odds ratio (natural parameter for a logistic regression model, in the $2 \times 2$ contingency table for outcomes:
$$ \begin{array}{c|cc} & Y & \bar{Y} \\ \hline X & A & B \\ \bar{X} & C& D \\ \end{array}$$
the log odds ratio $LOR = \log \frac{AD}{BC}$ has standard error:
$\mbox{se} ( LOR)^2 = 1/ A + 1/B + 1/C + 1/D$
So, the larger $A, B, C$ and $D$ are, the closer those fractions are to 0, and the mutual se is closer to 0.
-
$\begingroup$ I think you are correct, crosstabs doesn't show any statistical significance as well. 70% of observations with Y value 0 have X value 0 but also same 70% of observations with Y value 1 have X value 0. So no relation I guess. $\endgroup$– SalmanCommented Aug 3, 2015 at 21:41
-
$\begingroup$ @Salman are there 0 counts in these cells? Or counts of, say, less than 10? $\endgroup$– AdamOCommented Aug 3, 2015 at 21:49
-
$\begingroup$ No there are no such small counts. The minimum count is 127 $\endgroup$– SalmanCommented Aug 3, 2015 at 22:14
-
$\begingroup$ @Salman then you should have high power. That means the results strongly suggest there is no association, unless you've run your model incorrectly. $\endgroup$– AdamOCommented Aug 3, 2015 at 22:23
From my understanding of Logistic regression, you want to check that each category of yes/no's or 1/0's has a count >10*(p-1), where p is the number of covariates + 1 (for the intercept). If this holds true, you should be good as far as your dependent variables go. There is good reasoning to have closer to a 50/50 ratio of 1/0's but you don't have to.
Have you considered checking for Goodness of Fit using a Hosmer-Lemeshow test Statistic to see if logistic regression is appropriate with your covariates? To do in R, you can use this code
library(ResourceSelection)
hoslem.test()$p.value #look up the parameters in R's help file
If you aren't using R, I'm not sure how to do this test.
-
$\begingroup$ Hmmm, my data does not have the 50/50 distribution but it satisfies the condition you mentioned easily. Yes Hosmer Lemershow test also shows a 0.129 which is a good fit i guess? But it doesn't change the fact that there is still no statistical significance shown by logistic regression. $\endgroup$– SalmanCommented Aug 3, 2015 at 21:44
-
$\begingroup$ P value > than alpha means there is no evidence against the usefulness of logistic regression, i.e. your data passes the Homser-Lemeshow test. As far as your covariates go, how are you checking for statistical significance? $\endgroup$– JakeCommented Aug 4, 2015 at 12:51