Alternatives to logistic regression in R

Question

I would like as many algorithms that perform the same task as logistic regression. That is algorithms/models that can give a prediction to a binary response (Y) with some explanatory variable (X).

I would be glad if after you name the algorithm, if you would also show how to implement it in R. Here is a code that can be updated with other models:

set.seed(55)
n <- 100
x <- c(rnorm(n), 1+rnorm(n))
y <- c(rep(0,n), rep(1,n))
r <- glm(y~x, family=binomial)
plot(y~x)
abline(lm(y~x), col='red', lty=2)
xx <- seq(min(x), max(x), length=100)
yy <- predict(r, data.frame(x=xx), type='response')
lines(xx, yy, col='blue', lwd=5, lty=2)
title(main='Logistic regression with the "glm" function')

Answer 1

Popular right now are randomForest and gbm (called MART or Gradient Boosting in machine learning literature), rpart for simple trees. Also popular is bayesglm, which uses MAP with priors for regularization.

install.packages(c("randomForest", "gbm", "rpart", "arm"))

library(randomForest)
library(gbm)
library(rpart)
library(arm)

r1 <- randomForest(y~x)
r2 <- gbm(y~x)
r3 <- rpart(y~x)
r4 <- bayesglm(y ~ x, family=binomial)

yy1 <- predict(r1, data.frame(x=xx))
yy2 <- predict(r2, data.frame(x=xx))
yy3 <- predict(r3, data.frame(x=xx))
yy4 <- predict(r4, data.frame(x=xx), type="response")

Answer 2

Actually, that depends on what you want to obtain. If you perform logistic regression only for the predictions, you can use any supervised classification method suited for your data. Another possibility : discriminant analysis ( lda() and qda() from package MASS)

r <- lda(y~x) # use qda() for quadratic discriminant analysis

xx <- seq(min(x), max(x), length=100)
pred <- predict(r, data.frame(x=xx), type='response')
yy <- pred$posterior[,2]

color <- c("red","blue")

plot(y~x,pch=19,col=color[pred$class])
abline(lm(y~x),col='red',lty=2)
lines(xx,yy, col='blue', lwd=5, lty=2)
title(main='lda implementation')

On the other hand, if you need confidence intervals around your predictions or standard errors on your estimates, most classification algorithms ain't going to help you. You could use generalized additive (mixed) models, for which a number of packages are available. I often use the mgcv package of Simon Wood. Generalized additive models allow more flexibility than logistic regression, as you can use splines for modelling your predictors.

set.seed(55)
require(mgcv)
n <- 100
x1 <- c(rnorm(n), 1+rnorm(n))
x2 <- sqrt(c(rnorm(n,4),rnorm(n,6)))
y <- c(rep(0,n), rep(1,n))
r <- gam(y~s(x1)+s(x2),family=binomial)

xx <- seq(min(x1), max(x1), length=100)
xxx <- seq(min(x2), max(x2), length=100)
yy <- predict(r, data.frame(x1=xx,x2=xxx), type='response')

color=c("red","blue")
clustering <- ifelse(r$fitted.values < 0.5,1,2)

plot(y~x1,pch=19,col=color[clustering])
abline(lm(y~x1),col='red',lty=2)
lines(xx,yy, col='blue', lwd=5, lty=2)
title(main='gam implementation')

There's a whole lot more to do :

op <- par(mfrow=c(2,1))
plot(r,all.terms=T)
par(op)

summary(r)
anova(r)

r2 <- gam(y~s(x1),family=binomial)
anova(r,r2,test="Chisq")

...

I'd recommend the book of Simon Wood about Generalized Additive Models

Answer 3

I agree with Joe, and would add:

Any classification method could in principle be used, although it will depend on the data/situation. For instance, you could also use a SVM, possibly with the popular C-SVM model. Here's an example from kernlab using a radial basis kernel function:

library(kernlab)
x <- rbind(matrix(rnorm(120),,2),matrix(rnorm(120,mean=3),,2))
y <- matrix(c(rep(1,60),rep(-1,60)))

svp <- ksvm(x,y,type="C-svc")
plot(svp,data=x)

Answer 4

There are around 100 classification and regression models which are trainable via the caret package. Any of the classification models will be an option for you (as opposed to regression models, which require a continuous response). For example to train a random forest:

library(caret)
train(response~., data, method="rf")

See the caret model training vignette which comes with the distribution for a full list of the models available. It is split into dual-use and classification models (both of which you can use) and regression-only (which you can't). caret will automatically train the parameters for your chosen model for you.

Answer 5

Naive Bayes is a good simple method of training data to find a binary response.

library(e1071)
fitNB <- naiveBayes(y~x)
predict(fitNB, x)

Answer 6

There are two variations of the logistic regression which are not yet outlined. Firstly the logistic regression estimates probabilities using a logistic function which is a cumulativ logistic distribution (also known as sigmoid). You can also estimate probabilities using functions derived from other distributions. The most common way apart from the logistic regression is the probit regression which is derived from the normal distribution. For a more detailed discussion between the differences of probit and logit please visit the following site.

Difference between logit and probit models

set.seed(55)
n <- 100
x <- c(rnorm(n), 1+rnorm(n))
y <- c(rep(0,n), rep(1,n))
r <- glm(y~x, family=binomial(link="probit"))
plot(y~x)
abline(lm(y~x),col='red',lty=2)
xx <- seq(min(x), max(x), length=100)
yy <- predict(r, data.frame(x=xx), type='response')
lines(xx,yy, col='red', lwd=5, lty=2)
title(main='Probit regression with the "glm" function')

The second alternative points out a weekness of the logistical function you implemented. If you have a small sample size and/or missing values logistic function is not advisable. Hence an exact logistic regression is a better model. The log odds of the outcome is modeled as a linear combination of the predictor variables.

elrm(formula = y ~ x)

Furthermore there are other alternatives like to be mentioned:

1. Two-way contingency table
2. Two-group discriminant function analysis.
3. Hotelling's T2.

Final remark: A logistic regression is the same as a small neural network without hidden layers and only one point in the final layer. Therefore you can use implementations of neural network packages such as nnet in R.

Edit:

Some weeks later I realized that there is also the Winnow and the Perceptron algorithm. Both are classifiers which work also for classifications into two groups, but both are fallen out of favor in the last 15 years.