Suppose that we are interested in modeling a binary predictor $Y=0,1$ subject to $m$ predictors $x_1,...,x_m$. First, let us examine a simpler model of the impact of $x_j$ on the response $Y$. By the Bayes formula we have $$P(Y=1|x_j) = P(x_j|Y=1)P(Y=1) $$ $$P(Y=0|x_j) = P(x_j|Y=0)P(Y=0). $$ From these two expressions we obtain

$$\log\left(\frac{P(Y=1|x_j)}{P(Y=0|x_j)}\right) = \log\left(\frac{P(Y=1)}{P(Y=0)} \right) + \log\left(\frac{P(x_j|Y=1)}{P(x_j|Y=0)} \right). $$ If we assume now that all predictors $x_1,...,x_m$ are independent, then we find a similar equation: $$\log\left(\frac{P(Y=1|x_1,...x_m)}{P(Y=0|x_1,...,x_m)}\right) = \log\left(\frac{P(Y=1)}{P(Y=0)} \right) + \sum^m_{j=1}\log\left(\frac{P(x_j|Y=1)}{P(x_j|Y=0)} \right). $$

Since this is such a strong assumption to place on our predictors, we assume the Naïve Bayes form: $$\log\left(\frac{P(Y=1|x_1,...x_m)}{P(Y=0|x_1,...,x_m)}\right) = \log\left(\frac{P(Y=1)}{P(Y=0)} \right) + \sum^m_{j=1}\beta_j\log\left(\frac{P(x_j|Y=1)}{P(x_j|Y=0)} \right)$$ where the $\beta_j$ are estimated. The term $\log\left(\frac{P(x_j|Y=1)}{P(x_j|Y=0)} \right)$ is called the weight of evidence ($WOE$), and its predictive power is measured by the so-called information value defined by the following

$$IV_j = \int \log\left(\frac{P(x_j|Y=1)}{P(x_j|Y=0)} \right)\left(P(x_j|Y=1)- P(x_j|Y=0) \right)dx. $$

This method is a standard technique in credit risk modeling. Many sources all over the internet state the criterion
$$\text{if}\ IV< 0.02, \ \text{then useless for prediction } $$ $$\text{if}\ 0.02<IV< 0.1, \ \text{then weak predictive power } $$ $$\text{if}\ 0.1<IV< 0.3, \ \text{then medium predictive power } $$ $$\text{if}\ 0.3<IV< 0.5, \ \text{then strong predictive power } $$ $$\text{if}\ IV> 0.5, \ \text{then suspicious predictive power.} $$

Further, the information value can be written in terms of the Kullback-Leibler divergence $$IV_j = D_{KL}\left(P(x_j|Y=1)||P(x_j|Y=0)\right) + D_{KL}\left(P(x_j|Y=0)||P(x_j|Y=1)\right).$$ This being said, I have the following questions:

  1. What is the Bayesian interpretation of the $\beta_j$? How is $\beta_j$ related to the dependence on the remaining variables? It is not obvious to me how one can obtain the Naïve Bayes log-formula starting from the first principles, i.e., the Bayes formula. Seems like one can't, so is there any connection to Bayesian hierarchies, correlations, ect?
  2. What is the mathematical basis for the bounds on the information value and its predictive power? It is not clear why these bounds arise and thus why should they be followed.
  3. Where does the information value come from?

Thank you for your time.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.