It reminds me of what is implemented in the caret package for data pre-processing. It is fully described in one of the accompanying vignette, namely Data Sets and Miscellaneous Functions in the caret Package. What is actually done is to identify predictors that have low variance in the full dataset, as you described, whether it be a continuous or a categorical feature. They compute:
- the frequency of the most prevalent value over the second most frequent value (termed "frequency ratio"),
- the proportion of unique values (subject-wise),
considering that
If the frequency ratio is less than a
pre–specified threshold and the unique
value percentage is less than a
threshold, we might consider a
predictor to be near zero–variance. (p. 5, emphasis is mine)
The rationale is that near-zero variance predictors may have exact zero variance when using cross-validation, or induce model instability. They also address the problem of collinearity, but then this really is a matter of statistical modeling (some models, like classical regression models, don't accommodate well correlated predictors because it will inflates standard error of regression coefficients; others don't care about that).
Besides screening those low informative predictors, you can also use a hierarchical clustering method (by variables, not by individuals) to see how it goes. This is often used for studying missing data patterns (i.e., where we are interested in examining which variables are consistently showing an increased number of missing responses across all samples, or a particular subgroup).
