# Evaluate output of different dimensionality reduction methods

I used PCA, ICA, and FA to perform dimensionality reduction on my data. How can I measure which method performed best? If I reduce my data to 3 dimensions and plot it, what type of trends would represent a good transformation?

• This depends on what you are trying to achieve with dimensionality reduction. Why are you doing it? Jul 22, 2015 at 7:32
• I'm doing it as a preprocessing step to clustering.
– cavs
Jul 22, 2015 at 12:46
• Then perhaps you should use some measure of "clusteredness" as the measure of how well dimensionality reduction worked? Jul 22, 2015 at 12:49

All three methods have different assumptions and way of calculating components. These are data transformation techniques and the actual dimension reduction depends upon the correlation between the variables. For example in PCA, the eigen values represent percentage variance explained by the PCs. These are eigen values of the correlation matrix of the data. Suppose in my data analysis I see my eigen values are not very different. This means my variables are not very highly correlated. If I decide to choose few PCs for the purpose of dimension reductions, I am actually loosing valuable information (I wont do it). On the other hand if the first few eigen values are large compared to the rest, I can happily perform dimension reduction without loosing valuable information. you may look at the scree plot for this pattern. Similarly ICA focuses on Independence and FA assumes N(0,1) factors and the rest as error. To perform FA you need to decide on the number of factors and that may not be easy for all kind of problems. To summarize - you should evaluate the result in context with your data and have a theoretical understanding of these techniques.

• What would you suggest if my data is mostly uncorrelated?
– cavs
Jul 22, 2015 at 14:43
• Could you please elaborate what is the nature of your problem? I mean what is the exact objective you are trying to achieve during/after the dimensionality reduction? Jul 22, 2015 at 16:41
• I'm using it as a pre-processing step to hierarchical clustering and may try to use it as a pre-processing step to some sort of predictive analytics as well.
– cavs
Jul 22, 2015 at 16:48
• I think for pre-processing step to cluster analysis, PCA should be helpful. For predictive analytics, I would recommend first separate the variables as per their predictive power and then apply the transformation (if required). Please note that adding additional technique not necessarily improves the accuracy. You should try multiple combination of models with/without the transformation and evaluate them in out of sample data. After that you will know which is best technique for your data. Jul 22, 2015 at 21:22

I had the same question recently and bump into the nice notebook here https://www.kaggle.com/arthurtok/interactive-intro-to-dimensionality-reduction

In general, the idea is to run your dimensionality reductions, transform data and visualize to see if it is now possible to create clusters on it (like in your case). You can see than using for example PCA and its principal components how realistic is to use them to cluster the data. Here you are definitely limited by the amount of axes.

Alternatively, one can think about the reconstruction of the data (from Hands-on book from Géron)

X_reduced = pca.fit_transform(X_train)
X_recovered = pca.inverse_transform(X_reduced)

And calculate the mean squared distance between the original data and the reconstructed data. In this case the error shows not the best DR technique but what technique reduces dimensions more accurately and/or makes it possible to recover the original data set. It might be useful in some cases as well

Consider using a holdout sample and evaluate MSE there.

Often when dimensionality reducing, the end goal is to get a simpler model in order to predict more accurately in small samples. If that is indeed the purpose, splitting your sample in two, estimating on one, and then assessing fit on the other will tell you a lot. If your sample is too small for this, perhaps K-fold cross validation may be informative as well (essentially, like doing a 10% holdout 10 times and averaging the error on the holdout sample).

Make sure to compare with a reasonable benchmark, e.g. OLS.

• This is being automatically flagged as low quality, probably because it is so short. At present it is more of a comment than an answer by our standards. Can you expand on it? We can also turn it into a comment. Dec 11, 2016 at 16:57