Evaluate output of different dimensionality reduction methods I used PCA, ICA, and FA to perform dimensionality reduction on my data. How can I measure which method performed best? If I reduce my data to 3 dimensions and plot it, what type of trends would represent a good transformation?
 A: All three methods have different assumptions and way of calculating components. These are data transformation techniques and the actual dimension reduction depends upon the correlation between the variables. For example in PCA, the eigen values represent percentage variance explained by the PCs. These are eigen values of the correlation matrix of the data. Suppose in my data analysis I see my eigen values are not very different. This means my variables are not very highly correlated. If I decide to choose few PCs for the purpose of dimension reductions, I am actually loosing valuable information (I wont do it). On the other hand if the first few eigen values are large compared to the rest, I can happily perform dimension reduction without loosing valuable information. you may look at the scree plot for this pattern. Similarly ICA focuses on Independence and FA assumes N(0,1) factors and the rest as error. To perform FA you need to decide on the number of factors and that may not be easy for all kind of problems. To summarize - you should evaluate the result in context with your data and have a theoretical understanding of these techniques. 
A: I had the same question recently and bump into the nice notebook here https://www.kaggle.com/arthurtok/interactive-intro-to-dimensionality-reduction
In general, the idea is to run your dimensionality reductions, transform data and visualize to see if it is now possible to create clusters on it (like in your case). You can see than using for example PCA and its principal components how realistic is to use them to cluster the data. Here you are definitely limited by the amount of axes. 
Alternatively, one can think about the reconstruction of the data (from Hands-on book from Géron) 
X_reduced = pca.fit_transform(X_train)
X_recovered = pca.inverse_transform(X_reduced)

And calculate the mean squared distance between the original data and the reconstructed data. In this case the error shows not the best DR technique but what technique reduces dimensions more accurately and/or makes it possible to recover the original data set. It might be useful in some cases as well
A: Consider using a holdout sample and evaluate MSE there. 
Often when dimensionality reducing, the end goal is to get a simpler model in order to predict more accurately in small samples. If that is indeed the purpose, splitting your sample in two, estimating on one, and then assessing fit on the other will tell you a lot. If your sample is too small for this, perhaps K-fold cross validation may be informative as well (essentially, like doing a 10% holdout 10 times and averaging the error on the holdout sample). 
Make sure to compare with a reasonable benchmark, e.g. OLS. 
