10
$\begingroup$

I am trying to determine if simple probabilities will work for my problem or if it will be better to use (and learn about) more sophisticated methods like logistic regression.

The response variable in this problem is a binary response (0, 1). I have a number of predictor variables that are all categorical and unordered. I am trying to determine which combinations of the predictor variables yield the highest proportion of 1's. Do I need logistic regression? How would it be an advantage to just calculating proportions in my sample set for each combination of the categorical predictors?

|cite|improve this question
$\endgroup$
  • $\begingroup$ If you have more than 1 predictor, it could be tricky to do this without some kind of regression model. What did you have in mind? Just a big $k$-dimensional contigency table ($k$ is the number of predictors)? $\endgroup$ – Macro Jun 13 '12 at 19:35
  • $\begingroup$ Are the predictor categories grouped into more than one factor, & if so are they crossed or nested? Also, are you only interested in making a descriptive statement? If your data are complex a LR model may be more convenient, & if you want to make inferences LR is strongly preferable, I think. $\endgroup$ – gung - Reinstate Monica♦ Jun 13 '12 at 19:40
  • $\begingroup$ @Macro - Yes, I was thinking it would essentially be a big table, with one column showing # of sample points that correspond to the scenario, and another column showing the proportion of 1's. I have five categorical predictors, each with 10-30 possible values, so I know the list of scenarios would be high. I was thinking to script a loop in R that goes through each and output significant results (high proportion of 1's plus large # of sample points in the scenario). $\endgroup$ – Rachel Jun 13 '12 at 19:40
  • $\begingroup$ @gung - The factors are only partially crossed. None of the factors would be considered nested. I'm interested in finding the combinations of factors (e.g. State, Customer, Employee) that will likely have a high probability of the response variable being equal to 1. $\endgroup$ – Rachel Jun 13 '12 at 19:42
  • 1
    $\begingroup$ @EmreA - Unfortunately the categorical variables are not entirely independent. Some combinations will be more likely than others... $\endgroup$ – Rachel Jun 13 '12 at 21:30
11
$\begingroup$

Logistic regression will, up to numerical imprecision, give exactly the same fits as the tabulated percentages. Therefore, if your independent variables are factor objects factor1, etc., and the dependent results (0 and 1) are x, then you can obtain the effects with an expression like

aggregate(x, list(factor1, <etc>), FUN=mean)

Compare this to

glm(x ~ factor1 * <etc>, family=binomial(link="logit"))

As an example, let's generate some random data:

set.seed(17)
n <- 1000
x <- sample(c(0,1), n, replace=TRUE)
factor1 <- as.factor(floor(2*runif(n)))
factor2 <- as.factor(floor(3*runif(n)))
factor3 <- as.factor(floor(4*runif(n)))

The summary is obtained with

aggregate.results <- aggregate(x, list(factor1, factor2, factor3), FUN=mean)
aggregate.results

Its output includes

   Group.1 Group.2 Group.3         x
1        0       0       0 0.5128205
2        1       0       0 0.4210526
3        0       1       0 0.5454545
4        1       1       0 0.6071429
5        0       2       0 0.4736842
6        1       2       0 0.5000000
...
24       1       2       3 0.5227273

For future reference, the estimate for factors at levels (1,2,0) in row 6 of the output is 0.5.

The logistic regression gives up its coefficients this way:

model <- glm(x ~ factor1 * factor2 * factor3, family=binomial(link="logit"))
b <- model$coefficients

To use them, we need the logistic function:

logistic <- function(x) 1 / (1 + exp(-x))

To obtain, e.g., the estimate for factors at levels (1,2,0), compute

logistic (b["(Intercept)"] + b["factor11"] + b["factor22"] + b["factor11:factor22"])

(Notice how all interactions must be included in the model and all associated coefficients have to be applied to obtain a correct estimate.) The output is

(Intercept) 
        0.5

agreeing with the results of aggregate. (The "(Intercept)" heading in the output is a vestige of the input and effectively meaningless for this calculation.)

The same information in yet another form appears in the output of table. E.g., the (lengthy) output of

table(x, factor1, factor2, factor3)

includes this panel:

, , factor2 = 2, factor3 = 0

   factor1
x    0  1
  0 20 21
  1 18 21

The column for factor1 = 1 corresponds to the three factors at levels (1,2,0) and shows that $21/(21+21) = 0.5$ of the values of x equal $1$, agreeing with what we read out of aggregate and glm.

Finally, a combination of factors yielding the highest proportion in the dataset is conveniently obtained from the output of aggregate:

> aggregate.results[which.max(aggregate.results$x),]
  Group.1 Group.2 Group.3         x
4       1       1       0 0.6071429
|cite|improve this answer
$\endgroup$
  • 1
    $\begingroup$ This is a lot of information and will take me some time for me to digest, but I'm glad to hear the tabulated percentages and logistic regressions will essentially give me the same results. I ran the aggregate function on two of the predictors and it finished immediately...the results for the logistic fit has been running for a number of minutes and hasn't finished yet. I'll continue to tweak things, but I may end up just using the tabulated percentages. Thank you! $\endgroup$ – Rachel Jun 13 '12 at 21:00
  • $\begingroup$ (+1), Off the cuff I wonder if standard errors in this context from the logistic regression can be incorporated into the graphical mosaic plot summaries I suggested. I also suspect this could be a good way to quickly "filter" the results for interesting interactions (these are just my musings though!) $\endgroup$ – Andy W Jun 13 '12 at 22:41
  • $\begingroup$ "Logistic regression will, up to numerical imprecision, give exactly the same fits as the tabulated percentages": Isn't she tabulating percentages for every configuration of predictors? In which case, logistic regression can't encode all possible relationships. $\endgroup$ – Neil G Jun 13 '12 at 23:03
  • $\begingroup$ @Neil Sure it can encode all the "relationships" (combinations of factors): use all possible interactions. See the R code for an example. When the factors are numerous, there will in practice be many empty combinations, but decent software will have no problem dealing with that. Even better software will handle the combinations with no variation in responses. $\endgroup$ – whuber♦ Jun 14 '12 at 13:02
  • $\begingroup$ @Andy I love your reply about the graphics--we need to find a way to create a thread where it's exactly the right answer! Perhaps your thought about incorporating error estimates into the mosaic plots would be the way to approach it: how would you represent the logistic estimates, the actual data, and/or the error simultaneously in them? $\endgroup$ – whuber♦ Jun 14 '12 at 13:06
6
$\begingroup$

For a quick glance at the proportion of binary responses within each category and/or conditional on multiple categories, graphical plots can be of service. In particular, to simultaneously visualize proportions conditioned on many categorical independent variables I would suggest Mosaic Plots.

Below is an example taken from a blog post, Understanding area based plots: Mosaic plots from the Statistical graphics and more blog. This example visualizes the proportion of survivors on the Titanic in blue, conditional on the class of the passenger. One can simultaneously assess the proportion of survivors, while still appreciating the total number of passengers within each of the subgroups (useful information for sure, especially when certain sub-groups are sparse in number and we would expect more random variation).

Mosaic plot of Titanic
(source: theusrus.de)

One can then make subsequent mosaic plots conditional on multiple categorical independent variables. The next example from the same blog post in a quick visual summary demonstrates that all children passengers in the first and second classes survived, while in the third class children did not fare nearly as well. It also clearly shows that female adults had a much higher survival rate compared to males within each class, although the proportion of female survivors between classes diminished appreciably from the first to second to third classes (and then was relatively high again for the crew, although again note not many female crew members exist, given how narrow the bar is).

Mosaic conditional on third variable
(source: theusrus.de)

It is amazing how much information is displayed, this is proportions in four dimensions (Class, Adult/Child, Sex and Proportion of Survivors)!

I agree if you are interested in prediction or more causal explanation in general you will want to turn to more formal modelling. Graphical plots can be very quick visual clues though as to the nature of the data, and can provide other insights often missed when simply estimating regression models (especially when considering interactions between the different categorical variables).

|cite|improve this answer
$\endgroup$
  • $\begingroup$ +1, this is the point I was trying to make in my comment above regarding whether the goal was simple description or inference. Nb, that the point is clearer & better made w/ figures! $\endgroup$ – gung - Reinstate Monica♦ Jun 13 '12 at 20:41
  • $\begingroup$ @gung thanks, what does Nb stand for? I always say better with figures as well! $\endgroup$ – Andy W Jun 13 '12 at 22:42
  • 2
    $\begingroup$ Not anything related to statistics, "n.b." stands for nota bene, which in turn is Latin for 'note well' (literally), or 'note that' / 'notice' (more colloquially). $\endgroup$ – gung - Reinstate Monica♦ Jun 14 '12 at 1:23
3
$\begingroup$

Depending on your needs, you might find that recursive partioning provides an easy to interpret method for predicting an outcome variable. For an R introduction to these methods, see Quick-R's Tree-based model page. I generally favour ctree() implementation in R's `party package as one does not have to worry about pruning, and it produces pretty graphics by default.

This would fall into the category of feature selection algorithms suggested in a previous answer, and generally gives as good if not better predictions as logistic regression.

|cite|improve this answer
$\endgroup$
2
$\begingroup$

Given your five categorical predictors with let's say 20 outcomes each, then the solution with a different prediction for each configuration of predictors needs $20^5$ parameters. Each of those parameters needs many training examples in order to be learned well. Do you have at least ten million training examples spread over all configurations? If so, go ahead and do it that way.

If you have less data, you want to learn fewer parameters. You can reduce the number of parameters by assuming, for example, that configurations of individual predictors have consistent effects on the response variable.

If you believe that your predictors are independent of each other, then logistic regression is the unique algorithm that does the right thing. (Even if they're not independent, it can still do fairly well.)

In summary, logistic regression makes an assumption about independent influence of predictors, which reduces the number of model parameters, and yields a model that's easy to learn.

|cite|improve this answer
$\endgroup$
1
$\begingroup$

You should look at feature selection algorithms. One that is suitable for your case (binary classification, categorical variables) is the "minimum Redundancy Maximum Relevance" (mRMR) method. You can quickly try it online at http://penglab.janelia.org/proj/mRMR/

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Is it possible to run this program with more than one categorical predictor? On the upload page it looks like only the first column can be the "class" of data... Perhaps I do not understand how the data is supposed to be formatted. $\endgroup$ – Rachel Jun 13 '12 at 21:45
  • $\begingroup$ Or is the "class" supposed to be the output variable, in this case a 0 or 1? If so, is it important to turn the categorical variables into dummy variables to show indicators for each? $\endgroup$ – Rachel Jun 13 '12 at 21:47
  • $\begingroup$ You can have as many predictors as you want. The first row of your data file must be the feature names, and the first column must be the classes (response variable) for samples. So, an example is: response,predictor1,predictor2,predictor3 <line break here> 1,5,4,3 <line break here> 0,5,3,-1 <line break here> 1,1,2,3 $\endgroup$ – emrea Jun 13 '12 at 21:48
1
$\begingroup$

I work in the field of credit scoring, where what here is being presented as a strange case is the norm.

We use logistic regression, and convert both categorical and continuous variables into weights of evidence (WOEs), that are then used as the predictors in the regression. A lot of time is spent grouping the categorical variables, and discretising (binning/classing) the continuous variables.

The weight of evidence is a simple calculation. It is the log of the odds for the class, less the log of odds for the population:
WOE = ln(Good(Class)/Bad(Class)) - ln(Good(ALL)/Bad(ALL)) This is the standard transformation methodology for almost all credit scoring models built using logistic regression. You can use the same numbers in a piecewise approach.

The beauty of it is that you will always know whether the coefficients being assigned to each WOE make sense. Negative coefficients are contrary to the patterns within the data, and usually result from multicollinearity; and coefficients over 1.0 indicate overcompensation. Most coefficients will come out somewhere between zero and one.

|cite|improve this answer
$\endgroup$
  • $\begingroup$ Thank you for an interesting and informative post. I cannot figure out how it responds to the question in this thread, though. What is the "strange case" to which you refer? Did you perhaps intend to use this to reply to another question elsewhere? $\endgroup$ – whuber♦ Jun 16 '17 at 13:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.