Ways to compare feature selection methods Context: A hyperspectral image is taken (here Indiana Pines) which needs to be reduced to a lower dimension from 200 bands for this GSA is to be used.
What will be possible metrics to grade various dimension reductions?
Work attempted so far:

*

*Using KMeans clustering as a measure for the distribution. Problem is KMeans is highly dependent on the random_state and a simple relabelling would result in poor results;

*Using the inter-point distance matrix to compare results. Problem is there are $\approx 2 * 10^4$ points So the matrix is of size $\approx 2 * 10^8$ which is computationally heavy;

*Using a SVM over the data and grading based on accuracy. Problem is again fitting the SVM and scoring is computationally heavy so is not suitable metric for dimension reduction;

*fraction of variance of the data points preserved. Problem: doesn't  hold all data

*Compute distance to neighbours and compare the original vs reduced dimensional distance matrices. Problem: not a normalized value

Any help will be appreciated.
 A: The reason that you haven't found a canonical answer to this question is that the best measure of efficacy of dimension reduction is the measure you will ultimately use the reduced dimensions for.  When fitting a model in machine learning or statistics, the feature engineering and dimension reduction are part of the model, so their efficacy is judged on the same metrics that a model is judged on (MSE, MAPE, likelihood, etc).  If you are reducing dimensions to plot them and make a decision, then the efficacy of dimension reduction should be judged on how it enables the decision of interest (power, minimum detectable effect, etc.).  I recommend that your next step be to define precisely what the data will ultimately be used for and use that context to determine which method of dimension reduction works the best.
A: Adding to the excellent answer by @RCarnell, I wanted to note that there are different approaches to dimensionality reduction with different levels of generality. The great thing about PCA is that it allows you to get rid of useless information if you know your signal-to-noise ratio. Namely, the eigenvalues significantly smaller than SNR may be discarded, as any information that may have been contained in them is already corrupted beyond recognition. The same philosophy applies to other techniques from this family such as ICA, FA, NMF.
However, if you have a noise-less data structure and you can't guarantee that small changes in the structure won't result in big changes in classification, then there is no way to proceed. One approach could be to try to find precise symmetries in the data. It is important to represent data in a sensible way, namely, split categorical data into separate dimensions, as well as splitting categorical data hidden in floating-point values into separate dimensions. Generally, expanding data into many dimensions all of which are as simple as possible before applying a dimensionality reduction procedure is a good way to go. Further, nonlinear dimensionality reduction techniques can be used to try to hunt for that dimensionality. Thinks like kernel PCA and even total correlation come to mind.
But, as @RCarnell said, the only thing that eventually matters is whether any given feature is actually predictive. Without prior knowledge, the only way to know is to check
