In feature selection, what is the reason for considering removing low variance features? I've overheard a few times that when doing feature selection, one should look at features with low variance  and consider removing them.
(My guess is that  if we have a dataset of 100 observations and a feature has low variance, it might mean that the feature might not carry much information as the feature's value is the same for all the observations and thus isn't informative i.e. the feature is nearly constant across observation.)
 A: Imagine the limiting case in which you have a feature $x$ which is constant (no variance) will it have an effect on the output $y$? If $y$ is changing, then $x$ should be irrelevant in the relationship because it is constant.
This is the reason, why people tend to discard variables with low variance. The problem is that there is no rigorous method to determine if a feature has a "low" variance. Comparison with other features is often difficulty because they might have different scales (eg. if you compare the variance of the length of screws with the variance of weights of ships than you will have different scales). A $z$-transform will impose same scale and directly normalize the variance. Hence, $z$-transforms (mean = 0, variance = 1 for all variables) don't really help at this comparison.
One way to investigate this problem would be to look at the distribution of values. If you see a variable that has a very sharp spike (almost only one value is present) then you might consider discarding it.
To be quite honest I seldomly saw a feature in practice that had such a pathological distribution (no one would invest time and money to measure something that is constant all the time). These zero variance variables are often meta informations that were added because they were easy to gather. Hence, in my opinion, this condition is only used to rule out such pathological cases, where the variance is $0$. In all other cases, you must further investigate the variable.
A: Your guess is why I do it. If they have low variance, they likely won't improve your model anyway, so it's safe to remove them. For example, in MNIST, pixels that almost always background. Or some questionnaire items that are nearly always false and so on. Of course, 'variance' is not a good measure for any modality, and it might not be comparable between features.
Another reason is that low/no variance features sometimes make algorithms crash or fail to converge without any meaningful error message. When this happens, my first instinct is to remove invariant features and try again.
Lastly, it can reduce the dimensionality of your problem and make your model fit faster with less memory consumption.
