0
$\begingroup$

Lets say I have an R value of .5 and therefore an $R^2$ value of .25, and a p value < 0.001. (All through the stock cor.test function in R).

Is there a way to isolate those cases/observations whose variation is most explained by my $R^2$? My goal is two fold, find the group for which this correlation is maximized, and take the other group too explore separately for other interesting interactions instead if this one.

Sorry for the imprecise wording here, clearly I'm not quite an expert in this area, and hence dont quite know how or what question to ask, but thank you for any direction you can provide.

Improve this question
$\endgroup$
6
  • $\begingroup$ I'm not sure there's a meaningful general answer to this question. But if you have a few "groups" in mind you can always compute the correlations on those groups and compare them. $\endgroup$
    – shadowtalker
    Jul 20, 2014 at 15:34
  • $\begingroup$ Yes, i have done a bit of that manually, but that is a cumbersom process and has yielded little for the effort... I'm considering linear regression to arrive at a predicted value, and then a clustering method targeting clusters for which the prediction was most and least accurate... does that seem viable, or within the bounds of good-practice? $\endgroup$
    – Odj fourth
    Jul 20, 2014 at 18:16
  • $\begingroup$ I think the problem is that you can't really compute correlation for one observation, so observation-level clustering isn't really possible here. $\endgroup$
    – shadowtalker
    Jul 20, 2014 at 18:19
  • $\begingroup$ Ah... so given that I have validated the correlation, maybe a decision tree would help in exploring appropriate subgroups. $\endgroup$
    – Odj fourth
    Jul 20, 2014 at 18:56
  • $\begingroup$ Why not approach this via regression, in which case the answer is very straight-forward? $\endgroup$
    – jona
    Jul 20, 2014 at 19:13

2 Answers 2

Reset to default
0
$\begingroup$

In theory, this could be easily done by constructing a simple linear model (e.g. using R's lm()) and, for each point, comparing the actual value $y$ with the one predicted by your model. Consider what a linear regression actually is: a simple function,

$y = Intercept + (beta * x) + noise$

So the Intercept is a constant added to each observation, and beta is the slope of the regression line, or, phrased differently, by how much is the outcome/dependent variable y assumed to be higher than Intercept if the predictor/independent variable is x? So if Intercept is 0 and beta is 1.5, your prediction for y = x is y= 1.5 plus random noise, for x = 5 it's y = 7.5 plus random noise, etc.

So you get the slope/beta and intercept from summary(lm(y~x)). Then you go through each pair of points $i$ in your data and calculate the result of $Intercept + (beta*x_i)$, and check the absolute (or possibly squared or log) deviance to $y_i$.

What this would basically be equivalent to is doing a scatterplot including the best-fit linear regression line, and checking how far each point diverges from the line on the y axis. This is I think one of the possible answers to your question.

The question is, how meaningful is this? And that greatly depends on your data and question, and I think in very many contexts it would not be especially meaningful.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Excellent, thanks! I'll give it shot. As for how meaningful it will be, I'm not sure either. I have hunches that I want to confirm or deny though, and this should get me started on that path given that I'm comfortable enough with model validation to check the validity of my results. $\endgroup$
    – Odj fourth
    Jul 21, 2014 at 0:32
1
$\begingroup$

You might be interested into stepwise regression as an approach towards adding predictors to significantly increment the explained variance. Does this method provide what you were looking for?

Improve this answer
$\endgroup$
6
  • $\begingroup$ I'm not sure... I'd like to separate the observations for which this correlation is the highest so that i can focus on the remaining data to pursue other factors. Is there a clustering method that might allow for this? $\endgroup$
    – Odj fourth
    Jul 20, 2014 at 18:11
  • $\begingroup$ Hm, this appears a bit circular: A correlation is calculated over all values, so, each time you re-calculate it on a subsample of the original data, you alter it. This way you might continue removing 'noise' until you end up with two values per subsample. $\endgroup$
    – skip
    Jul 20, 2014 at 18:26
  • $\begingroup$ Well, too small a sample would result in insignificant results, so I wouldn't go that far. So, lets say I wanted to find the 25% of the population for which this correlation was maximized, is there a straight forward way of achieving that? $\endgroup$
    – Odj fourth
    Jul 20, 2014 at 18:43
  • $\begingroup$ The problem remains, essentially. Provided you had a large enough sample of bivariate data you could extract from the very same sample subsamples yielding significant positive or negative correlations, which doesn't mean these correlations are valid. The only thing I can think of, that might help you a bit, is the sliding window technique. $\endgroup$
    – skip
    Jul 20, 2014 at 21:25
  • $\begingroup$ Thanks, I'll look into that as well. I tried to accept both answers, but could only accept one, so I chose the other because it might suite my needs more directly. Stepwise would help explain more variance ( and I can double my R^2 doing so ) but one of the main questions I'm trying to answer is something like, "My single most explanatory attribute is X. But for the population whose variance is least explained by X, 1) What is that group? 2) Let's dig it and see what can explain their variance." $\endgroup$
    – Odj fourth
    Jul 21, 2014 at 0:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.