Almost in every ML models with high dimensional, one of the first things to do is removing features with low variance in order to decrease dimension.
But, when we do this, we don't examine the correlation between target variable and the feature. What if the correlation between target and feature is very high?
The thing we should do is firstly looking the correlation and then variance? Or is there any reason for removing the low variance first or solution related to this issue that I missed?
If variables represent different physical quantities their scaling can be different. By changing units (e.g. from measuring distance in kilometers to measuring distance in nanometers) you can change the scaling of a variable arbitrarily, so why would you even consider removing low-variance variables?
What does make sense though is that if you have a large number of predictors, they are probably highly correlated with each other. So you can use PCA on the training data to collect correlated features into principal components. Again, however, the scaling is relevant here. So unless all your variables are really measured in the same units, I would z-score (remove mean, divide by std) the data first and then apply PCA.
It is not always the case that features with low variance are removed. As @appletree stressed, changing the scale of a feature also change its variance ! So it would be dangerous to discard them.
However, if the variance is zero, it means that the feature is constant and will not improve the performance of the model (or make some matrices singular). In that case, it should be removed. Or if only a handful of observations differ from a constant value, the variance will also be very low.
This situation, where a feature has been poorly evaluated, or brings little information because it is (almost) constant can be a justification to remove a column.
If there is high correlation between 2 features then you would discard one of them. The features that are removed because of low variance have very low variance, that would be near to zero. You should always perform all the tests with existing data before discarding any features. Variables which are all 0's or have near to zero variance can be dropped due to less predictive power. In some cases it might cause a problem as well. Like in Naive Bayes Classifier, if one value is 0, then the entire equation becomes 0. Hence we use Laplace Smoothing where we add 1 to each feature count so that it doesn't come down to zero.
