11
$\begingroup$

This is a question regarding a practice or method followed by some of my colleagues. While making a logistic regression model, I have seen people replace categorical variables (or continuous variables which are binned) with their respective Weight of Evidence (WoE). This is supposedly done to establish a monotonic relation between the regressor and dependent variable. Now as far as I understand, once the model is made, the variables in the equation are NOT the variables in the dataset. Rather, the variables in the equation are now kind of the importance or weight of the variables in segregating the dependent variable!

My question is : how do we now interpret the model or the model coefficients? For example for the following equation : $$ \log\bigg(\frac{p}{1-p}\bigg) = \beta_0 + \beta_1x_1 $$

we can say that $\exp(\beta_1)$ is the relative increase in odd's ratio for 1 unit increase in the variable $x_1$.

But if the variable is replaced by its WoE, then the interpretation will be changed to : relative increase in odd's ratio for 1 unit increase in the IMPORTANCE / WEIGHT of the variable

I have seen this practice in internet, but nowhere I found answer of this question. This link from this community itself is related to somewhat similar query where someone wrote:

WoE displays a linear relationship with the natural logarithm of the odds ratio which is the dependent variable in logistic regression. Therefore, the question of model misspecification does not arise in logistic regression when we use WoE instead of the actual values of the variable.

But I still don't get the explanation. Please help me understand what I am missing.

$\endgroup$
  • $\begingroup$ $\exp(\beta_1)$ is the odds ratio associated w/ a 1 unit increase in $x_1$, not "the relative increase in the odds ratio associated w/ a 1 unit increase in $x_1$". $\endgroup$ – gung Jan 6 '16 at 17:49
  • $\begingroup$ Nope. Clearly in order to get rid of $ \beta_{0} $ you must take ratio of the LHS after exponentiation $\endgroup$ – user3096833 Jan 6 '16 at 18:42
  • $\begingroup$ The odds are p/(1-p), so if p(x) = exp(𝛽0+𝛽1x) and p(x+1) = exp(𝛽0+𝛽1x+𝛽1) note that p(x+1) = exp(𝛽0+𝛽1x)exp(𝛽1) and finally the odds ratio p(x+1)/p(x) = exp(𝛽1) as stated by stats.stackexchange.com/users/7290/gung $\endgroup$ – hwrd Jun 26 at 2:16
9
$\begingroup$

The WoE method consists of two steps:

1 - to split (a continuous) variable into few categories or to group (a discrete) variable into few categories (and in both cases you assume that all observations in one category have "same" effect on dependent variable)
2 - to calculate WoE value for each category (then the original x values are replaced by the WoE values)

The WoE transformation has (at least) three positive effects:
1) It can transform an independent variable so that it establishes monotonic relationship to the dependent variable. Actually it does more than this - to secure monotonic relationship it would be enough to "recode" it to any ordered measure (for example 1,2,3,4...) but the WoE transformation actually orders the categories on a "logistic" scale which is natural for logistic regression
2) For variables with too many (sparsely populated) discrete values, these can be grouped into categories (densely populated) and the WoE can be used to express information for the whole category
3) The (univariate) effect of each category on dependent variable can be simply compared across categories and across variables because WoE is standardized value (for example you can compare WoE of married people to WoE of manual workers)

It also has (at least) three drawbacks:
1) Loss of information (variation) due to binning to few categories
2) It is a "univariate" measure so it does not take into account correlation between independent variables
3) It is easy to manipulate (overfit) the effect of variables according to how categories are created

Conventionally, the betas of the regression (where the x has been replaced by WoE) are not interpreted per se but they are multiplied with WoE to obtain a "score" (for example beta for variable "marital status" can be multiplied with WoE of "married people" group to see the score of married people; beta for variable "occupation" can be multiplied by WoE of "manual workers" to see the score of manual workers. then if you are interested in the score of married manual workers, you sum up these two score and see how much is the effect on outcome). The higher the score is, the greater is probability of an outcome equal to 1.

$\endgroup$
  • 1
    $\begingroup$ (+1) Why's it an advantage to recode a predictor to have a monotonic relation with the response? $\endgroup$ – Scortchi Jun 26 '17 at 13:49
  • 1
    $\begingroup$ @Scortchi I can think of an example - the independent variable is height of people (measured in cm), people are going shopping for nice clothes, the dependent variable would be a binary event - whether they can or cannot buy suitable and comfortable clothes. appareantly the very small and the very tall people will have difficulties buying suitable clothes, while the people in the middle could do it easily. With simple (without interactions and without transformations) regression you could only model that probability of buying suitable clothes either increases or decreases with height of people $\endgroup$ – King Solomon's Horse Jun 28 '17 at 8:36
  • 1
    $\begingroup$ People don't usually use non-monotonic transformations of predictors - not in empirical modelling anyway. Including interactions can remove or introduce conditional non-monotonic relationships, as can including other predictors. But representing a predictor with a polynomial or spline basis function is a straightforward way of allowing for them; & another is binning it & thenceforward treating it as categorical, using e.g. reference-level coding. The last, at least, is considerably simpler than this WoE transformation; none share the detriment to ... $\endgroup$ – Scortchi Jun 29 '17 at 13:12
  • 1
    $\begingroup$ ... inference & interpretability arising from defining a predictor in terms of the response; & all allow a non-monotonic conditional relationship to be modelled even when the marginal relationship is monotonic (or vice versa). I suppose what I'm getting at is that the WoE transformation seems to me to be a solution in search of a problem. Are there a class of situations where it produces better predictions than more widely used methods? - though that's a different question to the one you've answered here (perhaps stats.stackexchange.com/q/166816/17230). $\endgroup$ – Scortchi Jun 29 '17 at 13:12
  • $\begingroup$ What about if you already have categorical data? then is the only advantage "to establish a monotonic relationship" ? It seems like the critical component of WoE is in fact in the binning process $\endgroup$ – information_interchange May 24 at 18:48
6
$\begingroup$

The rational for using WOE in logistic regression is to generate what is sometimes called the Semi-Naive Bayesian Classifier (SNBC). The beginning of this blog post explains things pretty well: http://multithreaded.stitchfix.com/blog/2015/08/13/weight-of-evidence/

The beta parameters in the model are the linear bias of each naive effect (a.k.a. weight-of-evidence) due to the presence of other predictors and they can be interpreted as the linear change in log odds of the particular predictors due to the presence of other predictors.

$\endgroup$
0
$\begingroup$

Weight of Evidence (WoE) is powerful technique to perform variable transformation & selection . It is widely used In credit scoring to measure the separation of good vs bad customers.(variables). Advantages :: - Handles missing values Handles outliers the transformation is based on logrithmic value of distribution. No need for dummy variables by using proper binning technique it can establish monotonic relationship btw the independent & dependent.

mono_bin() = used for numeric variables. char_bin() = used for character variables.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.