I wanted to know if we do EDA before logistic regression.

Sure, I will look at the variables and their distributions, but is there anything specific to logistic?

  • $\begingroup$ Box plots of any continuous covariates for the binary outputs has always seemed very helpful to me in identifying which covariates seem relevant to classifying the binary outputs. $\endgroup$ – RustyStatistician Feb 5 '16 at 21:06
  • 1
    $\begingroup$ How do the box plots help you? $\endgroup$ – Frank Harrell Feb 5 '16 at 21:07
  • $\begingroup$ But that would be after building the model, right? I am asking before I build the model $\endgroup$ – Bach Feb 5 '16 at 21:08
  • $\begingroup$ @FrankHarrell hopefully my answer is satisfactory for your question. $\endgroup$ – RustyStatistician Feb 5 '16 at 21:27
  • $\begingroup$ I would say that the 4-plot is a rev0 set of EDA. Trend, lag, histogram and prob-plot. Boxplot (aka JMP variability plot) also go well. I like the multivariate matrix plot because the brain can see more of the dimensions. I might look at the results of "glmulti" to get a sense of important factors. $\endgroup$ – EngrStudent Feb 5 '16 at 21:50

This is just one of the many exploratory data analyses (EDA's) that one could do for logistic regression that may shed some insights to the problem (and hopefully this clarifies my comment above @FrankHarrell).

For clarity sake, let's give an understandable example. Imagine you have data for some people and of interest is trying to predict the individual's gender (male or female, so binary) based on their height (a continuous covariate).

So, in my opinion, a good EDA to conduct would be to create box plots for the continuous covariate, height, based on the binary output, gender. Using some made up data, it would look like the following:

enter image description here

And from this EDA plot I would think it would be reasonable to assume that including height as a covariate in a logistic regression would be a good idea since, visually, it appears that the two genders only take on certain height values with little overlap.

I.e., when there is little to no overlap of the box plots then perhaps we should conclude that the covariate in question would be good for predicting (or in this case classifying) the binary outcome. When the box plots overlap considerably then this may not be case.

In case anyone is interested in the R code:

height = c(sample(40:62,100,replace=TRUE),sample(60:80,100,replace=TRUE))
gender = c(rep("Female",100),rep("Male",100))

        main="Exploratory Data Analysis Plot\n of Gender Versus Height")

The proper role of exploratory data analysis is a subtle issue. I wouldn't say that it is "necessary" to perform EDA before fitting a logistic regression model. You could conduct exploratory analyses if that is the point of your endeavor. On the other hand, if your goal is to test a hypothesis (say a treatment effect), exploring your data first to determine the optimal way to do it puts your thumb on the scale: you could end up with inflated type I error rates and poor out of sample performance.

Under the assumption that you are exploring a dataset because your true purpose is to explore the dataset, there are some plots that are handy. With all discrete variables, any number of simple plots (e.g., bar plots) could be used. The difficulty is when you have a continuous predictor. In that case, an option is to use a spineplot. There, you bin the x-variable using an algorithm for a histogram without looking at Y. Then you compute the proportion of 'successes' within each bin. A different option is to use a conditional density plot. Here is an example, coded in R using the NASA space shuttle o-ring failure data (adapted from the examples in the documentation):

fail = factor(c(2,2,2,2,1,1,1,1,1,1,2,1,2,1,1,1,1,2,1,1,1,1,1), 
              levels=c(1, 2), labels=c("no", "yes"))
temp = c(53,57,58,63,66,67,67,67,68,69,70,70,70,70,72,73,75,75,76,76,78,79,81)

dy = density(temp[fail=="yes"], from=min(temp), to=max(temp))
dn = density(temp[fail=="no"],  from=min(temp), to=max(temp))
sm = dy$y + dn$y
  spineplot(fail~temp, main="Splineplot")
  plot(dy, xlim=range(temp), ylim=c(0, 0.11), col="red", main="Densities",
  lines(dn, col="blue")
  lines(dy$x, sm, col="black")
  legend("topleft", legend=c("Failed","Didn't","Sum"), lty=1, 
  cdplot(fail~temp, main="Conditional density plot")

spineplot; densities; conditional density plot


Depending on the purpose of your models, you may need to examine plots and carry out other EDA techniques to determine if the covariates are correlated, which could lead to multicollinearity and affect parameter estimation and interpretation.

In addition, I'd suggest running covariate balance tests similar to the ones used in propensity score models to determine whether or not your data have common support (but this is a general regression problem and not limited to logistic).

Lastly, I would also create crosstabs of your dependent variable and your independent variables to determine if you have any cells that are perfectly predictive (or nearly so) of your outcome since this will cause "complete seperation" or "quasi-complete separation" problems (see here for example). Generally speaking, zero-valued cells cause problems during model fitting.

I believe you may find some additional methods listed in chapter four of Hosmer, Lemeshow, and Sturdivant's Applied Logisitc Regression, 3rd ed., Wiley, 2013.

  • 1
    $\begingroup$ I can see some value in that but also danger if the plot makes the statistician change the model, resulting in inaccurate statement of the true number of degrees of freedom present and inflation of type I error, lack of coverage of confidence intervals. Moreover, box plots seem to be interchanging what is considered the dependent variable and what is the independent variable. $\endgroup$ – Frank Harrell Feb 6 '16 at 13:22
  • $\begingroup$ @FrankHarrell did you intend to address my post or RustyStatistician's post (or both)? I'm not advocating the creation of any boxplots. $\endgroup$ – StatsStudent Feb 7 '16 at 3:13
  • $\begingroup$ More to @RustyStatistician but just a bit to you. $\endgroup$ – Frank Harrell Feb 7 '16 at 3:35
  • 1
    $\begingroup$ You are showing the distribution of $X|Y$ whereas we want $Y|X$ e.g. $Prob(Y=1|X)$. $\endgroup$ – Frank Harrell Feb 7 '16 at 19:36
  • 1
    $\begingroup$ That doesn't change how the calculations are done. The best way to keep the meaning of independent and dependent variables is to do a loess nonparametric smoother relating continuous x to binary y. But looking at this may possibly bias model specification. $\endgroup$ – Frank Harrell Feb 11 '16 at 22:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.