0
$\begingroup$

Reading from Logistic Regression slides by Erdem, before deriving the linear boundary equation from the GNB equation it says "What if we assume variance is independent of class i.e. $\sigma^2_{i,0}=\sigma^2_{i,1}$ and then put

$\prod\limits_{i=1}^{d}P(X_i|Y=0)P(Y=0)=\prod\limits_{i=1}^{d}P(X_i|Y=1)P(Y=1)$

what is that mean and what "variance is independent of class" assumption implies?

$\endgroup$
1
  • $\begingroup$ Suppose you had two points $A$ and $B$ on a plane and you wanted to divide the plane between closest to $A$ and those closest to $B$. There is a dividing line of the perpendicular bisector of $AB$ of points equally distant to them both. Suppose now you want to find points more than twice as far away from $B$ as they are from $A$; you would find the dividing line was a circle including (but not centred on) $A$. In a similar way, coming back to your question, assuming variance is independent of class leads to a linear decision boundary, which you may lose if you do not make that assumption $\endgroup$
    – Henry
    Feb 15, 2020 at 20:23

1 Answer 1

1
$\begingroup$

Let's say we have that variance is dependent of the class. We are talkin about the variance of the indipendent variable $X_i|Y=y_i$, which is a normal conditional PDF equal to enter image description here

You can clearly see that the variance of this variable depends on the class, infact we have the the $y$ index on the variance $\sigma_{i,y}$, meaning that variance could not be the same for all the classes (the classes anyway are two: $y = 0$, $y = 1$).

Practic example like the wikipedia one: We have a dependent variable which follows a Bernoulli rv where the students who pass the exam are the $1s$ the students who fail are the $0s$.

We have two normal random variables to explain the dependent variable: the $X_{1}|Y$ where the $x_{1,i}|y_{i}$ are the hours spent studying. The second random variable is $X_{2}|Y$ where the $x_{2,i}|y_{i}$ are the hours of sleeping. We run the model, we find the estimates and the decision boundary and:

  • we see that when a student fail the exam $y_i = 0$ there are no excuses: he failed because he studied very few hours, there's little randomness and we can assume that $\sigma^2_{i,y=0}=\epsilon$ where $\epsilon$ is a tiny variance.

  • we see that when a student pass the exam $y_i=1$ most of the cases he/she studied, but... there are many cases where he/she was lucky or has cheated, in fact many students who have low $x_{1,i}$ (they studied very few hours) have passed the exam. So in this case there is much randomness meaning that: $\sigma^2_{i,y=1}=\omega$, where $\omega > \epsilon$.

If we do not have the same variance the classification could not be perfect or efficient and we can't derive the linear boundary. If we assume that $\sigma^2_{i,y=0}=\sigma^2_{i,y=1}$ we can see that the conditional PDF loses the $y$ index, because in both cases of failing or passing the exam, we have the same variance.

enter image description here

So now with this assumption we can derive and obtain a decision boundary that is linear. If a distribution has a greater variance than the other one the classification could not be linear.

In your slides you see that you have a multivariate joint conditional PDF $X|Y=(X_{1},X_{2}|Y)$ with mean $M$ and variance $\Sigma$ that follows a normal distribution, where we have 2 information about the hours of study and the hours of sleeping. Like this one:enter image description here

What I do not like in the following picture of your slides is that if we have a different variance of the classes, let's say $\Sigma_{1} < \Sigma_{2} $, we should see a wider circles in the image below on the right, that would mean that the distribution is "wider" or with great variance than when we have an $y_i=0$.

enter image description here

$\Sigma_1$ would be the variance when there is an $y_i=0$ and $\Sigma_2$ would be the variance when there is a $y_i=1$. Hope this helps

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.