Which variable do I keep when the variables are linear combinations of others? I am doing an analysis of NBA data that has very typical team stats where I am attempting to find a prediction equation for the percentage of wins per team. Such data includes Two_Points_Made, Two_Point_Attempts, Two_Point_Percentage, Three_Points_Made, Three_Point_Attempts, Three_Point_Percentage, etc...
I understand I will need to remove some of the variables that are simply linear combinations of other variables.  For example, I removed the variable Total_Rebounds because my data set included Offensive_Rebounds and Defensive_Rebounds where Total_Rebounds is simply the addition of offensive and defensive rebounds. 
However, I am not sure how to approach the examples above (Two_Point_Made, Two_Point_Attempts, Two_Point_Percentage).  Is it appropriate to simply explain my reasoning for choosing one to drop between the three and to be consistent with the other metrics in the same manner? (i.e., dropping percentage of shots made for all places where it applies)  Or is there some analysis I can do that will help me choose which is best to drop?  Meaning... should I drop made, attempts, or percentage for two-pointers, three-pointers, and free throws?
 A: In principle, Factor Analysis was devised to deal with exactly your problem. It tries to achieve three aims:


*

*Identify groups of strongly correlated features.

*Identify "irrelevant" features, i.e. features not "contributing" to any of the groups.

*Replace each group by a new value called factor that is a linear combination of the features of the respective group


Beware however, that factor analysis is one of the most controversial stastitical techniques. Here are some quotes:


*

*"Factor analysis should not be used in most practical situations." (Chatfield & Collins, 1980)

*"Factor analysis is not worth the time necessary to understand it and carry it out." (Hills, 1977)

*"It is hard to find examples in the literature for which a factor analysis model fits well." (Venables & Ripley, 2002)


When I was preparing an introduction into factor analysis for a Pattern Recognition course, I was indeed running into the third problem, and it took me considerably time to actually find an example where factor analysis worked. Maybe your use case is such an example. As my brief introduction into factor analysis is in German and not published, it is presumably not of use to you. For a more extensive introduction in English, see

Hoyle, Rick H., and Jamieson L. Duvall. "Determining the number of factors in exploratory and confirmatory factor analysis." Handbook of quantitative methodology for the social sciences (2004): 301-315.

