Methods in R or Python to perform feature selection in unsupervised learning [closed]

Question

What are the available methods/implementation in R/Python to discard/select unimportant/important features in data? My data does not have labels (unsupervised).

The data has ~100 features with mixed types. Some are numeric while others are binary (0/1).

Answer 1

It's a year old but I still feel it is relevant, so I just wanted to share my python implementation of Principal Feature Analysis (PFA) as proposed in the paper that Charles linked to in his answer.

from sklearn.decomposition import PCA
from sklearn.cluster import KMeans
from collections import defaultdict
from sklearn.metrics.pairwise import euclidean_distances
from sklearn.preprocessing import StandardScaler

class PFA(object):
    def __init__(self, n_features, q=None):
        self.q = q
        self.n_features = n_features

    def fit(self, X):
        if not self.q:
            self.q = X.shape[1]

        sc = StandardScaler()
        X = sc.fit_transform(X)

        pca = PCA(n_components=self.q).fit(X)
        A_q = pca.components_.T

        kmeans = KMeans(n_clusters=self.n_features).fit(A_q)
        clusters = kmeans.predict(A_q)
        cluster_centers = kmeans.cluster_centers_

        dists = defaultdict(list)
        for i, c in enumerate(clusters):
            dist = euclidean_distances([A_q[i, :]], [cluster_centers[c, :]])[0][0]
            dists[c].append((i, dist))

        self.indices_ = [sorted(f, key=lambda x: x[1])[0][0] for f in dists.values()]
        self.features_ = X[:, self.indices_]

You can use it like this:

import numpy as np
X = np.random.random((1000,1000))

pfa = PFA(n_features=10)
pfa.fit(X)

# To get the transformed matrix
X = pfa.features_

# To get the column indices of the kept features
column_indices = pfa.indices_

This is strictly following the described algorithm from the article. I think the method has promise, but honestly I don't think it's the most robust approach to unsupervised feature selection. I'll post an update if I come up with something better.

Answer 2

The sparcl package in R performs sparse hierarchical and sparse K-means clustering. This may be useful. http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2930825/

Answer 3

Principal Feature Analysis looks to be a solution to unsupervised feature selection. It's described in this paper.

Answer 4

I've found a link wich could be useful, those are matlab implementations, they may work out for you http://www.cad.zju.edu.cn/home/dengcai/Data/MCFS.html it's a multicluster feature selection method, you can find strong foundation about it in recent papers Let me know if it works for you

Answer 5

There are many options available in R. A nice place to look is the caret package which provides a nice interface to many other packages and options. You can take a look at the website here. There are many options out there, but I will illustrate one.

Here is an example of using a simple filter using the built into R "mtcars" datasets (shown below).

                   mpg cyl disp  hp drat    wt  qsec vs am gear carb
Mazda RX4         21.0   6  160 110 3.90 2.620 16.46  0  1    4    4
Mazda RX4 Wag     21.0   6  160 110 3.90 2.875 17.02  0  1    4    4
Datsun 710        22.8   4  108  93 3.85 2.320 18.61  1  1    4    1
Hornet 4 Drive    21.4   6  258 110 3.08 3.215 19.44  1  0    3    1

Now some code setup (loading packages, etc.):

# setup a parallel environment
library(doParallel)
cl <- makeCluster(2) # number of cores to use
registerDoParallel(cl)
library(caret)

And we can fit a simple model to select variables:

fit1 <- sbf(mtcars[, -1], mtcars[, 1],
  sbfControl =
    sbfControl(functions = rfSBF, method = "repeatedcv", repeats = 10)
)

Viewing the results, we get:

fit1
Selection By Filter

Outer resampling method: Cross-Validated (10 fold, repeated 10 times) 

Resampling performance:

  RMSE Rsquared RMSESD RsquaredSD
 2.266   0.9224 0.8666     0.1523

Using the training set, 7 variables were selected:
   cyl, disp, hp, wt, vs...

During resampling, the top 5 selected variables (out of a possible 9):
   am (100%), cyl (100%), disp (100%), gear (100%), vs (100%)

On average, 7 variables were selected (min = 5, max = 9)

Finally we can plot the selected variables (in fit1$optVariables ) against the outcome, mpg:

library(ggplot2)
library(gridExtra)
do.call(grid.arrange, 
lapply(fit1$optVariables, function(v) {
  ggplot(mtcars, aes_string(x = v, y = "mpg")) +
    geom_jitter()
}))

Resulting in this graph:

Answer 6

The nsprcomp R package provides methods for sparse Principal Component Analysis, which could suit your needs.

For example, if you believe your features are generally correlated linearly, and want to select the top five, you could run sparse PCA with a max of five features, and limit to the first principal component:

m <- nsprcomp(x, scale.=T, k=5, ncomp=1)
m$rotation[, 1]

Alternatively, if you want to capture the orthogonal nature of features, you can select the top feature from each of the top five PCs, limiting each PC to one feature:

m <- nsprcomp(x, scale.=T, k=1, ncomp=5)
m$rotation

An ensemble of these could be useful too; i.e., features that consistently come to the top across different methods are likely to explain a large amount of variance in the feature space. Having played around with nsprcomp a bit, it seems like the first two methods raise ~1/2 of the same features to the top. That said, optimizing this process may be an empirical effort.