How to assess predictive power of set of categorical predictors of a binary outcome? Calculate probabilities or logistic regression?

Question

I am trying to determine if simple probabilities will work for my problem or if it will be better to use (and learn about) more sophisticated methods like logistic regression.

The response variable in this problem is a binary response (0, 1). I have a number of predictor variables that are all categorical and unordered. I am trying to determine which combinations of the predictor variables yield the highest proportion of 1's. Do I need logistic regression? How would it be an advantage to just calculating proportions in my sample set for each combination of the categorical predictors?

Answer 1

Logistic regression will, up to numerical imprecision, give exactly the same fits as the tabulated percentages. Therefore, if your independent variables are factor objects factor1, etc., and the dependent results (0 and 1) are x, then you can obtain the effects with an expression like

aggregate(x, list(factor1, <etc>), FUN=mean)

Compare this to

glm(x ~ factor1 * <etc>, family=binomial(link="logit"))

---

As an example, let's generate some random data:

set.seed(17)
n <- 1000
x <- sample(c(0,1), n, replace=TRUE)
factor1 <- as.factor(floor(2*runif(n)))
factor2 <- as.factor(floor(3*runif(n)))
factor3 <- as.factor(floor(4*runif(n)))

The summary is obtained with

aggregate.results <- aggregate(x, list(factor1, factor2, factor3), FUN=mean)
aggregate.results

Its output includes

   Group.1 Group.2 Group.3         x
1        0       0       0 0.5128205
2        1       0       0 0.4210526
3        0       1       0 0.5454545
4        1       1       0 0.6071429
5        0       2       0 0.4736842
6        1       2       0 0.5000000
...
24       1       2       3 0.5227273

For future reference, the estimate for factors at levels (1,2,0) in row 6 of the output is 0.5.

The logistic regression gives up its coefficients this way:

model <- glm(x ~ factor1 * factor2 * factor3, family=binomial(link="logit"))
b <- model$coefficients

To use them, we need the logistic function:

logistic <- function(x) 1 / (1 + exp(-x))

To obtain, e.g., the estimate for factors at levels (1,2,0), compute

logistic (b["(Intercept)"] + b["factor11"] + b["factor22"] + b["factor11:factor22"])

(Notice how all interactions must be included in the model and all associated coefficients have to be applied to obtain a correct estimate.) The output is

(Intercept) 
        0.5

agreeing with the results of aggregate. (The "(Intercept)" heading in the output is a vestige of the input and effectively meaningless for this calculation.)

---

The same information in yet another form appears in the output of table. E.g., the (lengthy) output of

table(x, factor1, factor2, factor3)

includes this panel:

, , factor2 = 2, factor3 = 0

   factor1
x    0  1
  0 20 21
  1 18 21

The column for factor1 = 1 corresponds to the three factors at levels (1,2,0) and shows that $21/(21+21) = 0.5$ of the values of x equal $1$, agreeing with what we read out of aggregate and glm.

---

Finally, a combination of factors yielding the highest proportion in the dataset is conveniently obtained from the output of aggregate:

> aggregate.results[which.max(aggregate.results$x),]
  Group.1 Group.2 Group.3         x
4       1       1       0 0.6071429

Answer 2

For a quick glance at the proportion of binary responses within each category and/or conditional on multiple categories, graphical plots can be of service. In particular, to simultaneously visualize proportions conditioned on many categorical independent variables I would suggest Mosaic Plots.

Below is an example taken from a blog post, Understanding area based plots: Mosaic plots from the Statistical graphics and more blog. This example visualizes the proportion of survivors on the Titanic in blue, conditional on the class of the passenger. One can simultaneously assess the proportion of survivors, while still appreciating the total number of passengers within each of the subgroups (useful information for sure, especially when certain sub-groups are sparse in number and we would expect more random variation).

One can then make subsequent mosaic plots conditional on multiple categorical independent variables. The next example from the same blog post in a quick visual summary demonstrates that all children passengers in the first and second classes survived, while in the third class children did not fare nearly as well. It also clearly shows that female adults had a much higher survival rate compared to males within each class, although the proportion of female survivors between classes diminished appreciably from the first to second to third classes (and then was relatively high again for the crew, although again note not many female crew members exist, given how narrow the bar is).

It is amazing how much information is displayed, this is proportions in four dimensions (Class, Adult/Child, Sex and Proportion of Survivors)!

I agree if you are interested in prediction or more causal explanation in general you will want to turn to more formal modelling. Graphical plots can be very quick visual clues though as to the nature of the data, and can provide other insights often missed when simply estimating regression models (especially when considering interactions between the different categorical variables).

Answer 3

Given your five categorical predictors with let's say 20 outcomes each, then the solution with a different prediction for each configuration of predictors needs $20^5$ parameters. Each of those parameters needs many training examples in order to be learned well. Do you have at least ten million training examples spread over all configurations? If so, go ahead and do it that way.

If you have less data, you want to learn fewer parameters. You can reduce the number of parameters by assuming, for example, that configurations of individual predictors have consistent effects on the response variable.

If you believe that your predictors are independent of each other, then logistic regression is the unique algorithm that does the right thing. (Even if they're not independent, it can still do fairly well.)

In summary, logistic regression makes an assumption about independent influence of predictors, which reduces the number of model parameters, and yields a model that's easy to learn.

Answer 4

Depending on your needs, you might find that recursive partioning provides an easy to interpret method for predicting an outcome variable. For an R introduction to these methods, see Quick-R's Tree-based model page. I generally favour ctree() implementation in R's `party package as one does not have to worry about pruning, and it produces pretty graphics by default.

This would fall into the category of feature selection algorithms suggested in a previous answer, and generally gives as good if not better predictions as logistic regression.

Answer 5

You should look at feature selection algorithms. One that is suitable for your case (binary classification, categorical variables) is the "minimum Redundancy Maximum Relevance" (mRMR) method. You can quickly try it online at http://penglab.janelia.org/proj/mRMR/