Determining correlation in certain subsets of a dataset in R I have a large dataset that has many variables.  I'm trying to determine which variables correlate strongly with one specific variable.  When you look at the entire dataset as a whole, the correlation of different variables is pretty weak.
I know, however, that within certain subsets of the data the correlation is strong.  For example:
When variable A is between X1 and Y1 and when variable B is between X2 and Y2, that resulting subset has unusually large instances of the variable I'm trying to optimize.
How can I determine using R which subsets of data have unusually large instances of the optimization variable when there are hundreds of variables to test against?
 A: It is a bit unclear what is your aim behind this, but maybe you just need a feature selection?
Try for instance training a Random Forest predicting the value you optimize from the other ones and extract its importance scores. What it does is almost explicitly a search for hyper-rectangles in your feature space with smallest possible variance and selecting those dimensions on which meaningful intervals are most frequently made.
A: Very Interesting problem. With my limited experience my first comment is that this problem would not be having many shortcuts. However I have done this kind of exercise. I would suggest the following points:
1) make a list of the variables that could be related-- this means that dont try to relate in your mind every x with y. Make a business case and ask yourself as to why could be there a pattern between x and y..example age could be shoowing pattern with salary( generally people who are more aged earn more) but lets say in insurance "my age may not be related with my agents age"( though actually it could be )
2) make a cross tab of these x and y.all possible Xs and Ys
3) read carefully through these cross tabs if there is a trend coming out from population...example has mean of salary gone up with age
4) finally make a category of those X( though this step is not necessary) and compare it against the whole dataset..you might see a different trend of these few rows as compared to the whole dataset.
Hope this helps.
