# Softmax_Normalization: Using sigmoid transformation on the independent variables to protect against outliers in logistic regression

While logistic regression has a set of easier to meet assumptions than linear regression, logistic regression can still suffer from outliers in the independent input variables. This can manifest itself when one variable can drown out another in certain cases.

It has been suggested that those could be handled by clipping (i.e. Winsoring) or trimming the outlier values. Transformations such as log or inverse can limit the risk from outliers.

However what are the thoughts of using a sigmoid transform, along the lines of a z-score to p-value transformation (cumulative distribution function). In R this can be done via pnorm((x-mean(x))/sd(x)) or approximated by 1.0/(1.0+exp(-1.69897*(x-mean(x))/sd(x))).

In the example code below, the skewness of the predictor variables were dramatically reduced and the KS scores in the fictitious example increased from 60.8 to 61.6. The McFadden pseudo R squared improved from 0.17 to 0.24. While not a huge improvement in the synthetic example, it shows some promise.

Are there any papers that discuss this technique?

References

Chris Busch. 2015. “Need a quick & dirty pnorm, normdist formula in a pinch for #SQL” https://twitter.com/cbuschnotes/status/667505348962418688

DS. 2009. Guide to Credit Scoring in R – CRAN. https://cran.r-project.org/doc/contrib/Sharma-CreditScoring.pdf

Appendix

ks.scoreAP=function(act,prd,yvar='',show=T){
require(ROCR)
pred<-prediction(prd,act)
perf <- performance(pred,"tpr","fpr")
#this code builds on ROCR library by taking the max delt
#between cumulative bad and good rates being plotted by
#ROCR
ks.score=max(attr(perf,'y.values')[[1]]-attr(perf,'x.values')[[1]])
if(show){
plot(perf,main=paste0(yvar,' KS=',round(ks.score*100,1),'%'))
lines(x = c(0,1),y=c(0,1))
}
ks.score;
}

require(e1071)
require(pscl)
set.seed(11)
d=data.frame(x1=rlnorm(1000),x2=rlnorm(1000))
summary(d)
d$y=ifelse(d$x1<1 & d$x2<1,1,0) d$y[1:200]=round(runif(200)) #noise
skewness(d$x1) skewness(d$x2)
m=glm(y ~. ,family=binomial(link='logit'),data=d)
ks.scoreAP(d$y,predict(m,type='response')) require(pscl) pR2(m) d$x1=pnorm((d$x1-mean(d$x1))/sd(d$x1)) d$x2=pnorm((d$x2-mean(d$x2))/sd(d$x2)) summary(d) skewness(d$x1)
skewness(d$x2) m=glm(y ~. ,family=binomial(link='logit'),data=d) ks.scoreAP(d$y,predict(m,type='response'))
pR2(m)


## 1 Answer

I think I found my answer.

What I described is a form of soft max normalization as described in this Wikipedia article.

Softmax_Normalization

The inclusion of 1.69897 softens effect of the tails to a greater extent.