3
$\begingroup$

I have binary valued classification variables, and predictors that are not really performing great in GLM with probit/logit model. Some of the predictors are also correlated with each other. I am considering to do a transformation to the parameters like a loess function in R. Loess applies to linear models where dependent variable is continuous, but my dependent variable is binary.

How can this approach extended to GLM probit/logit models? I might need a non-parametric transformation before feeding into GLM. The problem is how to find the non-parametric transform.

Edit 1: Here is an example where loess is applied directly to binary classifier, thus it is two stage. AUC jumps from 0.76 to 0.94. I would be glad to learn if there are any other ways to improve this nonlinear predictor

# nonlinear transformation ------------------------------------------------
set.seed(102)
a  <- runif(1000)
d  <- ifelse((a-0.3)^2 > 0.03, 1, 0)
d[ sample.int(1000, 50)]  <- 1
d[ sample.int(1000, 50)]  <- 0

par(mfrow=c(2,2))


df  <- data.frame(a, d)

glmmod <- glm(d ~ a, df, family=binomial(link = "logit"))
plot(a, glmmod$fitted.values)

lf  <- loess(d ~ a, df, model = T, span = 1)
plot(a, d)
lines(a[order(a)], predict(lf)[order(a)])

df2  <- data.frame(aT = predict(lf), d)
glmmod2 <- glm(d ~ aT, df2, family=binomial(link = "logit"))

plot(a, glmmod2$fitted.values)

require(ROCR)
pred <- prediction(glmmod2$fitted.values, d)
roc.perf = performance(pred, measure = "tpr", x.measure = "fpr")
plot(roc.perf, col="blue")
auc.perf = performance(pred, measure = "auc")
auc.perf@y.values[[1]]

pred <- prediction(glmmod$fitted.values, d)
roc.perf = performance(pred, measure = "tpr", x.measure = "fpr")
plot(roc.perf, add=TRUE, col="red")
auc.perf = performance(pred, measure = "auc")
auc.perf@y.values[[1]]

enter image description here

Improve this question
$\endgroup$
  • 2
    $\begingroup$ How would you use LOESS to transform a variable [N.B. not in statistical terms a parameter!]? I guess you are imagining using predictions of some LOESS smoothing as one variable in a GLM. In essence, GLM neither knows nor cares how a variable fed to it was derived any more than it knows how something was measured. But you would be producing a kind of two-step model dependent on your arbitrary choices within LOESS (choosing defaults is arbitrary too). The implications for inference and repeatability are complicated. (There is no objection, I think, to applying LOESS to a binary response.) $\endgroup$ – Nick Cox Feb 13 '15 at 18:08
  • 1
    $\begingroup$ This does not sound like a solution to your problem. You need to diagnose why the performance is not acceptable. If the problem is nonlinearity, then perhaps what you were thinking of achieving with loess could be accomplished with splines, which are much less unwieldy and natural. $\endgroup$ – whuber♦ Feb 13 '15 at 18:13
5
$\begingroup$

You don't use loess to transform variables.

You may be looking for generalized additive models (GAM), which is an extension of GLMs in the same way that additive models/nonparametric regression (including smoothing splines and local linear or local polynomial regression models) is an extension of linear regression.

https://en.wikipedia.org/wiki/Generalized_additive_model

example in R (picking your code up from df <- ..., using gam: 

df  <- data.frame(a, d)
library(gam) #assuming you already have the package 
gammod <- gam(d ~ s(a,4), df, family=binomial(link = "logit")) #spline model
plot(a,d)
oa=order(a)
lines(a[oa],fitted(gammod)[oa],col=3)

enter image description here

gammod2 <- gam(d ~ lo(a,span=.5), df, family=binomial(link = "logit")) #loess-like 
plot(a,d)
lines(a[oa],fitted(gammod2)[oa],col=4)

enter image description here

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.