3
$\begingroup$

I have a Bayesian model that tries to predict a binary variable that I am modelling as logistic regression. The training data have lot of wrong labels*. Therefore, I think I might need to introduce some sort of slack (similar to slack variable that SVM). How can I do this?

If I model logistic regression ignoring errors in training data, is logistic regression robust?

*The logistic regression training-data is sampled from another Bayesian model therefore there is large variance in my Gibbs sampling.

$\endgroup$
4
$\begingroup$

Logistic regression already has some slack but if you want even more slack you can use a softer link function. For example, replace the logistic function $\sigma(x)$ with $\sigma(\lambda \tanh(x/\lambda))$ where $\lambda$ controls the amount of slack. Alternatively, you can have an explicit noise model on the labels, i.e. each label gets flipped with probability $p$.

$\endgroup$
2
  • $\begingroup$ Thanks Minka. That's helpful. I think slack in logistic regression is not enough in my case because of lot of mis-labelled data. The side-effect is that $\sigma(x)$ is limited between .01 to .4. Is there are any other modelling possibility to model the binary variable instead of logistic regression? $\endgroup$
    – avi
    Mar 7 '14 at 22:47
  • 1
    $\begingroup$ If you don't like my link function above, another robust link function is $1/2 + arctan(x)/\pi$. $\endgroup$
    – Tom Minka
    Mar 8 '14 at 14:58
2
$\begingroup$

If the labels are wrong it might be worth looking at a model for logistic regression in the presence of label noise, Bootkrajang and Kaban have done some interesting work on this recently (which I suspect is the sort of thing Tom Minka had in mind?).

$\endgroup$
1

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.