# Alternatives to logistic regression in R

I would like as many algorithms that perform the same task as logistic regression. That is algorithms/models that can give a prediction to a binary response (Y) with some explanatory variable (X).

I would be glad if after you name the algorithm, if you would also show how to implement it in R. Here is a code that can be updated with other models:

set.seed(55)
n <- 100
x <- c(rnorm(n), 1+rnorm(n))
y <- c(rep(0,n), rep(1,n))
r <- glm(y~x, family=binomial)
plot(y~x)
abline(lm(y~x),col='red',lty=2)
xx <- seq(min(x), max(x), length=100)
yy <- predict(r, data.frame(x=xx), type='response')
lines(xx,yy, col='blue', lwd=5, lty=2)
title(main='Logistic regression with the "glm" function')

-
 Do we have to assume that you are considering a fixed set of predictors, i.e. you are interested in getting a reliable prediction given $k$ predictors, or are you also interested in some kind of penalization on the $X_j\quad (j=1\dots k)$? – chl♦ Aug 31 '10 at 11:29 I admit that for my personal interest, penalization wouldn't be necessary, and for the sake of knowledge here I would say both are relevant answers :) – Tal Galili Aug 31 '10 at 11:46 For future reference: you might have been able to phrase this question in such a way that we would have allowed it as a non-CW question. See meta.stats.stackexchange.com/questions/290/… – Shane Aug 31 '10 at 13:07 Thank you for the link Shane. A very interesting discussion you opened there. After reading Thomasas answer, I believe this should still be a community wiki, since my intention was to find as many alternatives as possible (something i doubt any one person would be able to supply). Yet, again, thank you for directing me to that thread! – Tal Galili Aug 31 '10 at 13:22

Popular right now are randomForest and gbm (called MART or Gradient Boosting in machine learning literature), rpart for simple trees. Also popular is bayesglm, which uses MAP with priors for regularization.

install.packages(c("randomForest", "gbm", "rpart", "arm"))

library(randomForest)
library(gbm)
library(rpart)
library(arm)

r1 <- randomForest(y~x)
r2 <- gbm(y~x)
r3 <- rpart(y~x)
r4 <- bayesglm(y ~ x, family=binomial)

yy1 <- predict(r1, data.frame(x=xx))
yy2 <- predict(r2, data.frame(x=xx))
yy3 <- predict(r3, data.frame(x=xx))
yy4 <- predict(r4, data.frame(x=xx), type="response")

-

Actually, that depends on what you want to obtain. If you perform logistic regression only for the predictions, you can use any supervised classification method suited for your data. Another possibility : discriminant analysis ( lda() and qda() from package MASS)

r <- lda(y~x) # use qda() for quadratic discriminant analysis

xx <- seq(min(x), max(x), length=100)
pred <- predict(r, data.frame(x=xx), type='response')
yy <- pred$posterior[,2] color <- c("red","blue") plot(y~x,pch=19,col=color[pred$class])
abline(lm(y~x),col='red',lty=2)
lines(xx,yy, col='blue', lwd=5, lty=2)
title(main='lda implementation')


On the other hand, if you need confidence intervals around your predictions or standard errors on your estimates, most classification algorithms ain't going to help you. You could use generalized additive (mixed) models, for which a number of packages are available. I often use the mgcv package of Simon Wood. Generalized additive models allow more flexibility than logistic regression, as you can use splines for modelling your predictors.

set.seed(55)
require(mgcv)
n <- 100
x1 <- c(rnorm(n), 1+rnorm(n))
x2 <- sqrt(c(rnorm(n,4),rnorm(n,6)))
y <- c(rep(0,n), rep(1,n))
r <- gam(y~s(x1)+s(x2),family=binomial)

xx <- seq(min(x1), max(x1), length=100)
xxx <- seq(min(x2), max(x2), length=100)
yy <- predict(r, data.frame(x1=xx,x2=xxx), type='response')

color=c("red","blue")
clustering <- ifelse(r\$fitted.values < 0.5,1,2)

plot(y~x1,pch=19,col=color[clustering])
abline(lm(y~x1),col='red',lty=2)
lines(xx,yy, col='blue', lwd=5, lty=2)
title(main='gam implementation')


There's a whole lot more to do :

op <- par(mfrow=c(2,1))
plot(r,all.terms=T)
par(op)

summary(r)
anova(r)

r2 <- gam(y~s(x1),family=binomial)
anova(r,r2,test="Chisq")


...

I'd recommend the book of Simon Wood about Generalized Additive Models

-

I agree with Joe, and would add:

Any classification method could in principle be used, although it will depend on the data/situation. For instance, you could also use a SVM, possibly with the popular C-SVM model. Here's an example from kernlab using a radial basis kernel function:

library(kernlab)
x <- rbind(matrix(rnorm(120),,2),matrix(rnorm(120,mean=3),,2))
y <- matrix(c(rep(1,60),rep(-1,60)))

svp <- ksvm(x,y,type="C-svc")
plot(svp,data=x)

-
 +1 I was about to post the same answer! – chl♦ Aug 31 '10 at 15:05

There are around 100 classification and regression models which are trainable via the caret package. Any of the classification models will be an option for you (as opposed to regression models, which require a continuous response). For example to train a random forest:

library(caret)
train(response~., data, method="rf")


See the caret model training vignette which comes with the distribution for a full list of the models available. It is split into dual-use and classification models (both of which you can use) and regression-only (which you can't). caret will automatically train the parameters for your chosen model for you.

-

Naive Bayes is a good simple method of training data to find a binary response.

library(e1071)
fitNB <- naiveBayes(y~x)
predict(fitNB, x)

-