Which are the most effective methods for selecting independent variables?

Some clustering algorithms require independence of variables but (especially working with real data) variables are often highly correlated.

I have been suggested to apply a Principal Component Analysis to solve the problem (using the resulting linear combination as input for the clustering). But such a solution may affect the resulting clusters since correlated varibles "repeat" the same information in the linear combination.

I.T.Jolliffe (1972; 2002) suggest a method of Variables Reductions (called 'method B4') that select one variable for each Component (the one with highest loading). Even if other method have been considered more effective (specially if the goal is to mantaine as much variance of the original variables as possible), Joliffe's method B4 is the only one -- that I have found -- that ensures the independent of the selected variables.

Question: Are there other more effective methods for selecting independent variables?

• I'm afraid you just need to wait more. I have made an edit to bump it to the top. – user88 Dec 1 '11 at 12:47
• Define "effective." Your goal is to assign cases to clusters, right? What will you do once you've assigned/discovered the clusters? – Michael Bishop Dec 9 '11 at 23:22
• I just wonder if someone can suggest other (more recent?) methods for variables-reduction that ensure the indipendence of the resulting variables. The discovered clusters will be compared with other qualitative classifications of the data. – en. Dec 11 '11 at 21:26
• You state: "But such a solution may affect the resulting clusters since correlated variables 'repeat' the same information in the linear combination." But, in fact, using PCA would result in uncorrelated component scores and would thus eliminate redundancies in the original variables. Using the variable that loads highest is not a bad strategy, but does not ensure that there are not correlations among the variables selected (though, it would likely reduce the intercorrelations among the variables selected). – Brett Dec 13 '11 at 15:14
• yes, PCA guarantees the indipendence of the resulting components, but at the same time these components are linear combination of the orinal variables, therefore the informations 'transformed' but still they could be reduntat. One clarify example: if I am using 3 variables like 'wheight', 'mass' and 'heght' of a sample of -let's say- metallic products. Weight and mass are strongly correlated. If I calculate a PCA the resulting Components will be probably 2: one associatble to mass and weight, one to heght. So if I use the components, I am actually 'repeating' the information weight-mass. – en. Dec 13 '11 at 17:17

There are so many different ways of optimizing (clustering + sparsity + independence)
that you'll have to describe your combination of these in more detail to get better answers.
Can you point to an example of what you want ? Is your input sparse ? What are Nsample, Nfeature, Nclass, roughly ?

There's Sparse PCA, see e.g. scikit-learn SparsePCA.

(By the way, Independent component analysis is something else.)

Added: with only 11 variables, make 11 runs leaving out one at a time and pick the best => e.g. leave out x3; then 10 runs leaving out one at a time => e.g. leave out x8; and so on. (There are much fancier methods.)

• My data are morphological parameters of a big sample of artifacts. (11variables X 50.000 objects) and therefore a lot of variables are strongly correlated (pearson>0.9). I need to run a clustering algorithm that require indipendence of variables but I also want to avoid redundat information (example: Volume and External-Surface would 'repeat' the 'same' information in a linear combination - infact these 2 variables are correlated). So I am wondering about how to reduce the variables and at the same time to have them indipendent and not "redundant". – en. Dec 13 '11 at 17:25
• If you're clustering in order to classify future data, and some of the data is labelled (don't know if that's the case), I'd suggest the combination of SVM and RFE e.g. in scikit-learn. Or how are you doing the clustering ? – denis Dec 13 '11 at 17:40
• 1- by 'labelled' do you mean if I am using cathegorical variables? in that case no, I am actually working only with scalar measurements this time. 2- I am using SPSS two-step clustering algorithm. – en. Dec 14 '11 at 10:55
• Say you have 4 clusters, red yellow green blue (or N E S W or ...), with 4 cluster centres. When a new point comes along, classify it as "red" if nearest the "red" centre etc.. "Labelled" a.k.a. "supervised learning" means you have some points of known class / color, i.e. some expert says "these are red, those yellow ...". Is that the case ? (Also added "leave-one-out" above.) – denis Dec 14 '11 at 13:08
• No, I do not have clusters centres: SPSStwo-step pre-clusters the records in a ﬁrst step of the procedure and then uses an agglomerative hierarchical clustering on the resulting pre-clusters (see BIRCH by Zhang et al.1996) – en. Dec 14 '11 at 17:37