8
$\begingroup$

One assumption for regression analysis is that $X$ and $Y$ are not intertwined. However when I think about it It seems to me that it makes sense.

Here is an example. If we have a test with 3 sections (A B and C). The overall test score is equal to the sum of individual scores for the 3 sections. Now it makes sense to say that $X$ can be score in section A and $Y$ the overall test score. Then the linear regression can answer this question: what is the variability in overall test score that is attributable to section A? Here, several scenarios are possible:

  1. Section A is the hardest of the 3 sections and students always score lowest on it. In such a case, intuitively $R^2$ would be low. Because most of the overall test score would be determined by B and C.
  2. Section A was very easy for students. In this case also the correlation would not be high. Because students always score 100% of this section and therefore this section tells us nothing about the overall test score.
  3. Section A has intermmediate difficulty. In this case the correlation would be stronger (but this also depends on the other scores (B and C).

Another example is this: we analyze the total content of a trace element in urine. And we analyze independently the individual species (chemical forms) of that trace element in urine. There can be many chemical forms. And if our analyses are correct, the sum of chemical forms should give us the same as the total content of an element (analyzed by a different technique). However, it makes sense to ask whether one chemical form is correlated with the total element content in urine, as this total content is an indicator of the total intake from food of that element. Then, if we say that $X$ is the total element in urine and $Y$ is chemical form A in urine then by studying the correlation we can explore whether this chemical form is the major one that contributes to the overall variablity or not.

it seems to me that it makes sense sometimes even when $X$ and $Y$ are not independent and that this can in some cases help answer scientific questions.

Would you think $R^2$ can be useful or meaningful in the examples above ? If we consider the test score example above, I would already say there would be about 33% contribution of each section had the difficulty been exactly the same for the students. But in practice this is not necessarily true. So I was thinking maybe using regression analysis can help us know the true variability attributed to each section of an exam. So it seems to me that $R^2$ would be meaningful even though we already know the null hypothesis is not true.

Are there alternative modified regression methods to account for such situations and provide us with meaningful parameters?

$\endgroup$
3
  • $\begingroup$ I don't agree with this statement: "1- section A is the hardest of the 3 sections and students always score lowest on it. In such a case, intuitively R squared would be low." If section A is the hardest then there will be more inconsistencies of answers leading to increased variability as a result, R-squared, which measures the proportion of variability explained by section A would be larger. $\endgroup$ Mar 10, 2015 at 15:22
  • $\begingroup$ Thanks for your comment. My thinking was that section A is hard to the extent that all students are scoring very low (and close to zero on a 0-100 scale) on this section. Then variability in the score of this section does not contribute significantly to the overall test score (for example it doesnt matter if it is 15 or 20 or 10%). The overall test score (and its variability) will be determined by the other more variable test scores that play the bigger part of the overall score. $\endgroup$
    – Bassam
    Mar 10, 2015 at 16:34
  • $\begingroup$ Why are you expecting $Y$ and $X$ to be independent in a regression analysis? One is the mean value of the other (at least if we omit other predictors). Also, I don't understand your first sentence, what assumption is that? A regression should involve some kind of intertwining. $\endgroup$
    – swmo
    Mar 13, 2015 at 0:08

4 Answers 4

5
+100
$\begingroup$

You might want to consider an approach outside of the traditional regression approach. This is comparable to the types of problems that psychometrics is designed to resolve (well, actually your first example is precisely that, since it is a test).

In Classical Test Theory, one of the most common metrics is the item-total score correlation, which is essentially the correlation between the item score and the total score. It tells you the item's discrimination - it's ability to discriminate between high and low scoring respondents. This is comparable to explaining the variance, like what you are asking about above with $R^2$. There are two ways to calculate this score, either by using the total test score including the item of interest, or excluding it. When you have a lot of items, these two methods are almost the same, but when you have few items, then they can make a big difference.

Another approach from Item Response Theory (IRT) is to estimate, either via a 2-parameter item response model or via a confirmatory factor analysis (which statistically are the same, but interpretation-wise are different). A 2-parameter model includes a parameter for the item difficulty (the relative difficulty of the item) and one for item discrimination, which is interpreted very similar to the item-total score correlation. High discrimination=the item differentiates between high and low scorers well. If you use confirmatory factor analysis (CFA), you have item loadings, which are essentially your discrimination parameters. They tell you how much of the total score is driven by a particular item.

Using IRT or CFA assumes you have a latent score, not an observed score, that you are trying to estimate. In the examples you give above, you are concerned with an observed score, that isn't latent. So these models wouldn't be what you are after, since they are probabilistic and you kind of have a tautological relationship (your total is by definition made up of the parts, with no error). But I point them out as examples of ways statistics gets at similar answers.

Last thing I want to point out, and this is probably something others would argue with, but while an assumption is that regressors are independent, when we have a categorical variable, and we enter dummies into the model, those dummy variables are, by definition, correlated. So this would seemingly violate assumptions of independence and bring in multicollinearity. If you think of it this way, it would make sense to run your regression of say the elements in urine, and exclude one, the coefficients would be valid just as if it was a single categorical variable. In that sense, you are getting a comparable number to the item-total correlation from Classical Test Theory I pointed out above.

$\endgroup$
4
$\begingroup$

A quick mathematial way of looking at it is to expand out the formulas. Let $Z=X+Y+W$.

$$ R^2 =\left(\frac{Cov(X,Z)}{\sigma_X \sigma_Z}\right)^2 =\left(\frac{Var(X)+Cov(X,Y)+Cov(X,W)}{\sigma_X \sigma_Z}\right)^2 $$

So in a nutshell You're going to get the variance of $X$ plus its relationship with your other two variables, divided by a scaling factor. The scaling factor itself could be expanded, but the numerator is telling the story. In general, things that will affect that number are a) the relative scale of X compared to Y and W, b) the relative variance of X, c) X's "contribution" to the variance of Y and W.

As for whether that's useful or not, that kind of depends on what you're after. It's probably best to think of it as a "percentage of total variation" or something like that, even though the same for Y and W may not all sum to 1 (or maybe it does...not sure).

$\endgroup$
2
$\begingroup$

If X is one of several variables that sum to define Y, then clearly the assumptions of linear regression are broken. The P values won't be useful. The slopes and their confidence intervals can't be interpreted in the usual way. But is $R^2$ still useful? I suppose it is as a descriptive statistics. If you have three $R^2$ values quantifying correlation between Y and each of its three components, I suppose you'd learn something interesting by seeing the relative values of $R^2$.

$\endgroup$
1
  • $\begingroup$ Thank you for your comment. That was exactly my feeling. That R2 values and a comparison between them might provide us with useful information. $\endgroup$
    – Bassam
    Mar 11, 2015 at 10:31
0
$\begingroup$

One assumption for regression analysis is that $X$ and $Y$ are not intertwined.

This is incorrect. One assumption for regression analysis is that the ERRORS are uncorrelated. See the wikipedia entry for the Gauss-Markov theorem.

If $X$ is one of several variables that sum to $Y$, is the $R^2$ between $X$ and $Y$ a useful value?

About the only use I can think of for the $R^2$ between $X$ and $Y$ is to show how much better your model performs when you include other predictors. There's other values that would be much informative. The values of the estimated coefficients and their standard errors in particular.

$\endgroup$
0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.