Dimensionality reduction and regression analysis I have what seem like a silly question about dimensionality reduction.
I am just learning about this now on my own, and most of the information I have found is centered around reducing dimension by figuring out whether or not linear dependence is present. Upon learning about dimensionality reduction, what I sort of assumed was that one method would be to run a regression and figure out whether or not your variables were statistically significant or not. If they were not, then they could be eliminated, and this as well would reduce dimension. Is this also a correct method? Am I thinking about this the right way?
Please keep in mind that I am an undergraduate and have only taken introductory courses on regression, linear algebra, statistics, etc but I am interested in these topics and just trying to develop some intuition for different methods and when they are appropriate.
 A: In your post, what you are describing is feature selection. You can find many posts on methods for feature selection. Here are two popular posts


*

*Algorithms for automatic model selection

*A more definitive discussion of variable selection
What most data scientist/statisticians see as dimension reduction is methods of actually condensing the information in your $m$ variables down to $p$ variables where $p < m$. This can be viewed as a method for feature selection as well. Here is a post that briefly talks about that


*

*Using principal component analysis (PCA) for feature selection
Two algorithms/methods that are commonly used for dimension reduction are principal component analysis (PCA) and Gaussian Random Projection.
I assume your goal is to select features for a regression model. Depending on your goal/research question, dimension reduction by PCA, or otherwise, may or may not be necessary. Often, interpretability is lost when using dimension reduction, so sticking to other methods of feature selection may behoove you if you'd like to keep interpretation in your statistical results.
