10
$\begingroup$

When is weight of evidence (WOE) transformation of categorical variables useful?

The example can be seen in WOE transformation

(So for a response $y$, & a categorical predictor with $k$ categories, & $y_j$ successes out of $n_j$ trials within the $j$th category of this predictor, the WOE for the $j$th category is defined as

$$\log \frac{y_j} {\sum_j^k {y_j}} \frac{\sum_j^k (n_j-y_j)}{n_j-y_j}$$

& the transformation consists of coding each category of the categorical predictor with its WOE to form a new continuous predictor.)

I would like to learn the reason why WOE transformation helps the logistic regression. What is the theory behind this?

$\endgroup$
6
$\begingroup$

In the example you link to, the categorical predictor is represented by a single continuous variable taking a value for each level equal to the observed log odds of the response in that level (plus a constant):

$$\log \frac{y_j} {n_j-y_j} + \log \frac{\sum_j^k (n_j-y_j)}{\sum_j^k {y_j}}$$

This obfuscation doesn't serve any purpose at all that I can think of: you'll get the same predicted response as if you'd used the usual dummy coding; but the degrees of freedom are wrong, invalidating several useful forms of inference about the model.

In multiple regression, with several categorical predictors to transform, I suppose you'd calculate WOEs for each using marginal log odds. That will change the predicted responses; but as confounding isn't taken into account—the conditional log odds aren't a linear function of the marginal log odds—I can't see any reason to suppose it an improvement, & the inferential problems remain.

$\endgroup$
  • $\begingroup$ Can you explain why degrees of freedom is wrong with WOE? It is just a transformation right? Also what if we had several categorical variables, and we got WOE for each of the one by one? In my experience when you have many categorical variables, then some buckets between different variables overlap a lot, and you start to see some coefficients which are insignificant. And also you need to carry around several coefficients. $\endgroup$ – adam Aug 12 '15 at 13:35
  • 1
    $\begingroup$ (1) A transformation that depends on evaluating the relation of predictors to response - something that's supposed to be left to the regression. So e.g. the likelihood ratio test statistic won't have the same distribution as when a transformation is pre-specified. (2) Good point! - a multiple regression on WOEs won't be equivalent to that on dummy variables (unless the models are saturated). (3) So what? (4) Coefficients aren't any heavier than WOEs. $\endgroup$ – Scortchi - Reinstate Monica Aug 12 '15 at 14:56
1
$\begingroup$

Coarse classing using the measure of weight of Evidence (WoE) has the following advantage- WoE displays a linear relationship with the natural logarithm of the odds ratio which is the dependent variable in logistic regression.
Therefore, the question of model misspecification does not arise in logistic regression when we use WoE instead of the actual values of the variable.

$ln(p/1-p)$ = $\alpha$ + $\beta$*$WoE(Var1)$ + $\gamma$*$WoE(Var2)$ + $\eta$*$WoE(Var3 )$

Source: In one of the PPTs my trainer showed me during the company training.

$\endgroup$
  • 1
    $\begingroup$ "model misspecification does not arise in logistic regression when we use WoE instead of the actual values of the variable". Can you explain/prove this mathematically? $\endgroup$ – adam Aug 27 '15 at 10:14
  • $\begingroup$ I'm not from risk analytics background but pg 131,132 of this book seems to say so books.google.co.in/… $\endgroup$ – Srikanth Guhan Aug 27 '15 at 10:19
  • $\begingroup$ Also this link claims the same though no mathematics is explained analyticbridge.com/forum/topics/… $\endgroup$ – Srikanth Guhan Aug 27 '15 at 10:24
  • $\begingroup$ Thanks for the links, but it's clearly untrue that the marginal log odds to which WoE is proportional have a linear relationship with the conditional log odds with which logistic regression concerns itself. Confounding with other predictors can even result in WoE ordering categories differently. $\endgroup$ – Scortchi - Reinstate Monica Sep 1 '15 at 15:12
1
$\begingroup$

WOE transformations help when you have both numeric and categorical data that you need to combine and missing values throughout that you would like to extract information from. Converting everything to WOE helps "standardize" many different types of data (even missing data) onto the the same log odds scale. This blog post explains things reasonably well: http://multithreaded.stitchfix.com/blog/2015/08/13/weight-of-evidence/

The short of the story is that Logistic Regression with WOE, should just be (and is) called a Semi-Naive Bayesian Classifier (SNBC). If you are trying to understand the algorithm, the name SNBC is, to me, far more informative.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.