I am studying a data set with one continuous response variable and 4 continuous predictor variables. The two predictors most correlated with the response also have a high (but not perfect) correlation with each other. A more parsimonious model might rely on 3 predictors, but I don't want to drop any of the variables.
I am considering two approaches. One is to do principal components analysis (PCA) on the space of predictors, and running a principal components regression (PCR), regressing the response on the first 3 principal components of the predictors' space.
The other alternative is to run a factor analysis (FA), attempting to discover three factors underlying the predictors' space that best explain the response; the method of FA would be PCA.
My question is whether these approaches are equivalent, or whether there is a fundamental conceptual difference in the goals of these methods. Thank you!