2
$\begingroup$

I have a database with many attributes. I would like to know which attributes has the minimum variation in the data. Is there some standard technique? It should be like clustering without split records in clusters. I would like to know what the records in particular cluster have in common.

I was going to compute the mean ($\bar{x}$) and st.d. ($s$) for each continuous attribute $x$. After computing the coefficient of variation $CV=\frac{s}{\bar{x}}$ I would say that attributes with $CV\leq0.1$ are the similar ones. For categorical ones I would choose attributes with more than $90\%$ relative frequency for the mode.

Is there some standard technique?

$\endgroup$
2
$\begingroup$

It reminds me of what is implemented in the caret package for data pre-processing. It is fully described in one of the accompanying vignette, namely Data Sets and Miscellaneous Functions in the caret Package. What is actually done is to identify predictors that have low variance in the full dataset, as you described, whether it be a continuous or a categorical feature. They compute:

  • the frequency of the most prevalent value over the second most frequent value (termed "frequency ratio"),
  • the proportion of unique values (subject-wise),

considering that

If the frequency ratio is less than a pre–specified threshold and the unique value percentage is less than a threshold, we might consider a predictor to be near zero–variance. (p. 5, emphasis is mine)

The rationale is that near-zero variance predictors may have exact zero variance when using cross-validation, or induce model instability. They also address the problem of collinearity, but then this really is a matter of statistical modeling (some models, like classical regression models, don't accommodate well correlated predictors because it will inflates standard error of regression coefficients; others don't care about that).

Besides screening those low informative predictors, you can also use a hierarchical clustering method (by variables, not by individuals) to see how it goes. This is often used for studying missing data patterns (i.e., where we are interested in examining which variables are consistently showing an increased number of missing responses across all samples, or a particular subgroup).

$\endgroup$
  • $\begingroup$ Nice answer! In the last hint: could it be necessary to transform categorical attributes in numerical ones? I think in missing data patterns analysis the db is a 0/1 db (0 non missing, 1 missing). $\endgroup$ – Simone Mar 24 '11 at 13:10
  • $\begingroup$ @Simone Yes, we can work with a binary indicator for missingness or low variance (with a constant cut-off), but I guess you could also use a numerical summary of "variable sparsity" as defined above (which would work for any kind of variables) $\endgroup$ – chl Mar 24 '11 at 13:15
  • $\begingroup$ Sorry, but maybe I didn't get it. I should transpose my db and then add 1 for each missing value and 0 otherwise. It works for a missing values analysis. In my case, I should compute the mean for each attribute and then for each value put 0 or 1 due to a numerical summary (eg if it is the most frequent value I choose 1 else 0?). Thanks. $\endgroup$ – Simone Mar 24 '11 at 14:18
  • $\begingroup$ @Simone I originally meant recode as 0/1 depending on whether your variables are below or above your fixed threshold for deciding of low-variance, or just use a numerical value (e.g., the frequency ratio cited above, or the CV you proposed) that reflects the amount of variability present in here (a numeric variable with 2 or 3 unique observed values or a categorical variable with only one modality present would be what I call a poorly discriminative variable, it is not necessarily uninformative) -- this is for examining structured patterns of sparsity, if any. $\endgroup$ – chl Mar 24 '11 at 14:49
  • $\begingroup$ let me stop bothering you: each value of an attributes gets 0 if it's below a treshold otherwise 1? I was wondering if I should replace each value for each attribute in a different way. If we moved giving the same value for an attribute it would be nice to substitute them with a frequency ratio to obtain a sort of lattice of cluster, ie attributes with similar frequency ratio. Am I right? Thanks. $\endgroup$ – Simone Mar 24 '11 at 15:10

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.