What are the available methods/implementation in R/Python to discard/select unimportant/important features in data? My data does not have labels (unsupervised).

The data has ~100 features with mixed types. Some are numeric while others are binary (0/1).

  • $\begingroup$ What kind of unsupervised learning algorithm are you using? What does your data look like? $\endgroup$ – Max Candocia Jul 21 '14 at 17:55
  • $\begingroup$ @user1362215, Before applying any unsupervised algorithm, I am trying to find a way to perform feature removal. $\endgroup$ – learner Jul 21 '14 at 18:23
  • $\begingroup$ Have you seen this scikit-learn cheatsheet before? It may help you get started... $\endgroup$ – Steve S Jul 21 '14 at 19:15
  • $\begingroup$ Why not use a unsupervised method that perfomes feature selection by itself like random forest in unsupervised mode? $\endgroup$ – JEquihua Jul 21 '14 at 19:20
  • 1
    $\begingroup$ I'm not completely surem, I mean random forest is completely non parametric so don't worry about assumptions. What I'm not sure is if it will serve your purpose. What I CAN say is that there is a version of Random Forest just for "anomaly detection" called isolation forests: cs.nju.edu.cn/zhouzh/zhouzh.files/publication/… there was an implementation in R but i'm not sure if it's up and running as of now. $\endgroup$ – JEquihua Jul 22 '14 at 15:13

It's a year old but I still feel it is relevant, so I just wanted to share my python implementation of Principal Feature Analysis (PFA) as proposed in the paper that Charles linked to in his answer.

from sklearn.decomposition import PCA
from sklearn.cluster import KMeans
from collections import defaultdict
from sklearn.metrics.pairwise import euclidean_distances
from sklearn.preprocessing import StandardScaler

class PFA(object):
    def __init__(self, n_features, q=None):
        self.q = q
        self.n_features = n_features

    def fit(self, X):
        if not self.q:
            self.q = X.shape[1]

        sc = StandardScaler()
        X = sc.fit_transform(X)

        pca = PCA(n_components=self.q).fit(X)
        A_q = pca.components_.T

        kmeans = KMeans(n_clusters=self.n_features).fit(A_q)
        clusters = kmeans.predict(A_q)
        cluster_centers = kmeans.cluster_centers_

        dists = defaultdict(list)
        for i, c in enumerate(clusters):
            dist = euclidean_distances([A_q[i, :]], [cluster_centers[c, :]])[0][0]
            dists[c].append((i, dist))

        self.indices_ = [sorted(f, key=lambda x: x[1])[0][0] for f in dists.values()]
        self.features_ = X[:, self.indices_]

You can use it like this:

import numpy as np
X = np.random.random((1000,1000))

pfa = PFA(n_features=10)

# To get the transformed matrix
X = pfa.features_

# To get the column indices of the kept features
column_indices = pfa.indices_

This is strictly following the described algorithm from the article. I think the method has promise, but honestly I don't think it's the most robust approach to unsupervised feature selection. I'll post an update if I come up with something better.

| cite | improve this answer | |
  • $\begingroup$ In the method described in the paper that you link to, Step 1 is to calculate the covariance matrix and step 2 is to calculate PCA on the covariance matrix from Step 1. I believe your fit function skips Step 1, and performs PCA on the original dataset. $\endgroup$ – user35581 Oct 1 '18 at 15:49
  • $\begingroup$ @user35581 good point. However, what they do is to (1) produce the covariance matrix from the original data and then (2) compute eigenvectors and eigenvalues of the covariance matrix using the SVD method. Those two steps combined are what you call PCA. The Principle Components are the eigenvectors of the covariance matrix of the original data. $\endgroup$ – Ulf Aslak Oct 2 '18 at 7:39
  • $\begingroup$ @Ulf Aslak can you elaborate why you think it is not the most robust approach to unsupervised feature selection? $\endgroup$ – hipoglucido Dec 19 '19 at 9:31
  • 1
    $\begingroup$ @hipoglucido honestly, I can't account for my thoughts when I wrote that. It's three years ago. Reviewing the code again, I'm led to believe it has something to do with the use of KMeans (which is non-deterministic). Also, I would like to see how this compares to just clustering the non-PCA-transformed features. $\endgroup$ – Ulf Aslak Dec 22 '19 at 4:26

The sparcl package in R performs sparse hierarchical and sparse K-means clustering. This may be useful. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2930825/

| cite | improve this answer | |

Principal Feature Analysis looks to be a solution to unsupervised feature selection. It's described in this paper.

| cite | improve this answer | |
  • $\begingroup$ Link is dead. Shared a python implementation of the method proposed in the article stats.stackexchange.com/a/203978/76815. $\endgroup$ – Ulf Aslak Mar 27 '16 at 19:42
  • $\begingroup$ Thanks, I removed the link (it was deleted as off-topic). $\endgroup$ – Max Ghenis Mar 27 '16 at 19:49

I've found a link wich could be useful, those are matlab implementations, they may work out for you http://www.cad.zju.edu.cn/home/dengcai/Data/MCFS.html it's a multicluster feature selection method, you can find strong foundation about it in recent papers Let me know if it works for you

| cite | improve this answer | |

There are many options available in R. A nice place to look is the caret package which provides a nice interface to many other packages and options. You can take a look at the website here. There are many options out there, but I will illustrate one.

Here is an example of using a simple filter using the built into R "mtcars" datasets (shown below).

                   mpg cyl disp  hp drat    wt  qsec vs am gear carb
Mazda RX4         21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
Mazda RX4 Wag     21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
Datsun 710        22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
Hornet 4 Drive    21.4   6  258 110 3.08 3.215 19.44  1  0    3    1

Now some code setup (loading packages, etc.):

# setup a parallel environment
cl <- makeCluster(2) # number of cores to use

And we can fit a simple model to select variables:

fit1 <- sbf(mtcars[, -1], mtcars[, 1],
  sbfControl =
    sbfControl(functions = rfSBF, method = "repeatedcv", repeats = 10)

Viewing the results, we get:

Selection By Filter

Outer resampling method: Cross-Validated (10 fold, repeated 10 times) 

Resampling performance:

  RMSE Rsquared RMSESD RsquaredSD
 2.266   0.9224 0.8666     0.1523

Using the training set, 7 variables were selected:
   cyl, disp, hp, wt, vs...

During resampling, the top 5 selected variables (out of a possible 9):
   am (100%), cyl (100%), disp (100%), gear (100%), vs (100%)

On average, 7 variables were selected (min = 5, max = 9)

Finally we can plot the selected variables (in fit1$optVariables ) against the outcome, mpg:

lapply(fit1$optVariables, function(v) {
  ggplot(mtcars, aes_string(x = v, y = "mpg")) +

Resulting in this graph: scatter plots

| cite | improve this answer | |
  • 1
    $\begingroup$ This isn't unsupervised learning as OP requested, since you're model uses mpg as an outcome. Is there a way to use methods like these in unsupervised models? $\endgroup$ – Max Ghenis Aug 9 '15 at 4:33

The nsprcomp R package provides methods for sparse Principal Component Analysis, which could suit your needs.

For example, if you believe your features are generally correlated linearly, and want to select the top five, you could run sparse PCA with a max of five features, and limit to the first principal component:

m <- nsprcomp(x, scale.=T, k=5, ncomp=1)
m$rotation[, 1]

Alternatively, if you want to capture the orthogonal nature of features, you can select the top feature from each of the top five PCs, limiting each PC to one feature:

m <- nsprcomp(x, scale.=T, k=1, ncomp=5)

An ensemble of these could be useful too; i.e., features that consistently come to the top across different methods are likely to explain a large amount of variance in the feature space. Having played around with nsprcomp a bit, it seems like the first two methods raise ~1/2 of the same features to the top. That said, optimizing this process may be an empirical effort.

| cite | improve this answer | |

Not the answer you're looking for? Browse other questions tagged or ask your own question.