2
$\begingroup$

If a feature has zero variance, then it makes sence to drop it as is it has no predictive power. What if a feature has zero variance only for cases that belong to one class but not in the other (binary classification). Does it make sence to drop it?

$\endgroup$
1
  • 1
    $\begingroup$ Even if that feature have zero variance for one of the classes, it might still be very different from the distribution it has in the other class! and so be informative. So you should probably keep it. $\endgroup$ Feb 17, 2020 at 1:00

3 Answers 3

2
$\begingroup$

Let us to use $X$ to represent the feature and $Y$ to represent the label. Essentially, if $P(Y|X)=P(Y)$ or $X$ and $Y$ are independent, we can drop $X$.

What you described

feature has zero variance only for cases that belong to one class but not in the other

Just tells this feature is an important feature that can differentiate different classes, i.e.,

$P(X|Y)$ is different for $Y=0$ and $Y=1$, so we should keep $X$.

Keep in mind that, $P(Y|X) \propto P(X|Y)P(Y)$.


Note that, on the other hand, $X$ can have none-zero variance but still can have nothing to do with predicting $Y$.

$\endgroup$
1
$\begingroup$

Even if that feature have zero variance for one of the classes (or even if it has zero variance for both classes!), it could still have very different values between the classes, so be a good discriminator. You should probably keep it!

$\endgroup$
0
$\begingroup$

Consider trivial example $x = [0, 0, 0, 1, 2, 3]$ and $y = [1, 1, 1, 0, 0, 0]$. If you fit a decision tree to this data, it needs to make single split on $x < 1$ to get perfect fit to the data.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.