3
$\begingroup$

While logistic regression has a set of easier to meet assumptions than linear regression, logistic regression can still suffer from outliers in the independent input variables. This can manifest itself when one variable can drown out another in certain cases.

It has been suggested that those could be handled by clipping (i.e. Winsoring) or trimming the outlier values. Transformations such as log or inverse can limit the risk from outliers.

However what are the thoughts of using a sigmoid transform, along the lines of a z-score to p-value transformation (cumulative distribution function). In R this can be done via pnorm((x-mean(x))/sd(x)) or approximated by 1.0/(1.0+exp(-1.69897*(x-mean(x))/sd(x))).

In the example code below, the skewness of the predictor variables were dramatically reduced and the KS scores in the fictitious example increased from 60.8 to 61.6. The McFadden pseudo R squared improved from 0.17 to 0.24. While not a huge improvement in the synthetic example, it shows some promise.

result of Sigmoid transformation

Are there any papers that discuss this technique?

References

Chris Busch. 2015. “Need a quick & dirty pnorm, normdist formula in a pinch for #SQL” https://twitter.com/cbuschnotes/status/667505348962418688

DS. 2009. Guide to Credit Scoring in R – CRAN. https://cran.r-project.org/doc/contrib/Sharma-CreditScoring.pdf

Appendix

ks.scoreAP=function(act,prd,yvar='',show=T){  
  require(ROCR)
  pred<-prediction(prd,act)
  perf <- performance(pred,"tpr","fpr")
  #this code builds on ROCR library by taking the max delt
  #between cumulative bad and good rates being plotted by
  #ROCR
  ks.score=max(attr(perf,'y.values')[[1]]-attr(perf,'x.values')[[1]])
  if(show){
    plot(perf,main=paste0(yvar,' KS=',round(ks.score*100,1),'%'))
    lines(x = c(0,1),y=c(0,1))
  }
  ks.score;
}

require(e1071)
require(pscl)
set.seed(11)
d=data.frame(x1=rlnorm(1000),x2=rlnorm(1000))
summary(d)
d$y=ifelse(d$x1<1 & d$x2<1,1,0)
d$y[1:200]=round(runif(200)) #noise
skewness(d$x1)
skewness(d$x2)
m=glm(y ~. ,family=binomial(link='logit'),data=d)
ks.scoreAP(d$y,predict(m,type='response'))
require(pscl)
pR2(m)
d$x1=pnorm((d$x1-mean(d$x1))/sd(d$x1))
d$x2=pnorm((d$x2-mean(d$x2))/sd(d$x2))
summary(d)
skewness(d$x1)
skewness(d$x2)
m=glm(y ~. ,family=binomial(link='logit'),data=d)
ks.scoreAP(d$y,predict(m,type='response'))
pR2(m)
$\endgroup$
1
$\begingroup$

I think I found my answer.

What I described is a form of soft max normalization as described in this Wikipedia article.

Softmax_Normalization

The inclusion of 1.69897 softens effect of the tails to a greater extent.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.