3
$\begingroup$

I have binary valued classification variables, and predictors that are not really performing great in GLM with probit/logit model. Some of the predictors are also correlated with each other. I am considering to do a transformation to the parameters like a loess function in R. Loess applies to linear models where dependent variable is continuous, but my dependent variable is binary.

How can this approach extended to GLM probit/logit models? I might need a non-parametric transformation before feeding into GLM. The problem is how to find the non-parametric transform.

Edit 1: Here is an example where loess is applied directly to binary classifier, thus it is two stage. AUC jumps from 0.76 to 0.94. I would be glad to learn if there are any other ways to improve this nonlinear predictor

# nonlinear transformation ------------------------------------------------
set.seed(102)
a  <- runif(1000)
d  <- ifelse((a-0.3)^2 > 0.03, 1, 0)
d[ sample.int(1000, 50)]  <- 1
d[ sample.int(1000, 50)]  <- 0

par(mfrow=c(2,2))


df  <- data.frame(a, d)

glmmod <- glm(d ~ a, df, family=binomial(link = "logit"))
plot(a, glmmod$fitted.values)

lf  <- loess(d ~ a, df, model = T, span = 1)
plot(a, d)
lines(a[order(a)], predict(lf)[order(a)])

df2  <- data.frame(aT = predict(lf), d)
glmmod2 <- glm(d ~ aT, df2, family=binomial(link = "logit"))

plot(a, glmmod2$fitted.values)

require(ROCR)
pred <- prediction(glmmod2$fitted.values, d)
roc.perf = performance(pred, measure = "tpr", x.measure = "fpr")
plot(roc.perf, col="blue")
auc.perf = performance(pred, measure = "auc")
[email protected][[1]]

pred <- prediction(glmmod$fitted.values, d)
roc.perf = performance(pred, measure = "tpr", x.measure = "fpr")
plot(roc.perf, add=TRUE, col="red")
auc.perf = performance(pred, measure = "auc")
[email protected][[1]]

enter image description here

$\endgroup$
2
  • 3
    $\begingroup$ How would you use LOESS to transform a variable [N.B. not in statistical terms a parameter!]? I guess you are imagining using predictions of some LOESS smoothing as one variable in a GLM. In essence, GLM neither knows nor cares how a variable fed to it was derived any more than it knows how something was measured. But you would be producing a kind of two-step model dependent on your arbitrary choices within LOESS (choosing defaults is arbitrary too). The implications for inference and repeatability are complicated. (There is no objection, I think, to applying LOESS to a binary response.) $\endgroup$
    – Nick Cox
    Commented Feb 13, 2015 at 18:08
  • 1
    $\begingroup$ This does not sound like a solution to your problem. You need to diagnose why the performance is not acceptable. If the problem is nonlinearity, then perhaps what you were thinking of achieving with loess could be accomplished with splines, which are much less unwieldy and natural. $\endgroup$
    – whuber
    Commented Feb 13, 2015 at 18:13

1 Answer 1

6
$\begingroup$

You don't use loess to transform variables.

You may be looking for generalized additive models (GAM), which is an extension of GLMs in the same way that additive models/nonparametric regression (including smoothing splines and local linear or local polynomial regression models) is an extension of linear regression.

https://en.wikipedia.org/wiki/Generalized_additive_model

example in R (picking your code up from df <- ..., using gam:

df  <- data.frame(a, d)
library(gam) #assuming you already have the package 
gammod <- gam(d ~ s(a,4), df, family=binomial(link = "logit")) #spline model
plot(a,d)
oa=order(a)
lines(a[oa],fitted(gammod)[oa],col=3)

plot of smooth logistic regression fit based on spline, showing a U-shaped central part, asymptoting to 1 at either end

gammod2 <- gam(d ~ lo(a,span=.5), df, family=binomial(link = "logit")) #loess-like 
plot(a,d)
lines(a[oa],fitted(gammod2)[oa],col=4)

Plot of logistic fit based on locally-weighted smooth function, similar appearance

The 'lo' there stands for a locally weighted glm fit, which is similar to a loess smooth in regression but without some of the additional parts that relate to robustifying the local smooth by downweighting points.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.