Summary: Am I correlating two independent variables? Is that the problem?

Let's say I have data point for square footage of a house and the asking price. Now, I can ask "Does square footage (x) determine price (y)?" This is intuituve, and makes sense. R-squared would end up saying "Square footage explains X% of the variation in price". So far, so good.

But, what if I want to predict the square footage from the price? That seems valid. Now, I ask "Does price (x) determine square footage (y)?" So far, it seems either can function as the independent or dependent variable. However, the wording or r-squared seem off. "Price explains X% of the variation in square footage". Huh? Square footage is not some sort of multi-factor variable. It's more static. Nothing "explains" the square footage, it just is. Get what I'm saying? Like if price only explains x% of square footage, what else would explain square footage? Square footage is just square footage. It's not like price, which can be determined by many things (square footage, renovations, size of yard, etc).

Another example can be age (x) and the mileage on a car (y). With a regression equation, I can use one to predict the other. Either order seems to work. However does age "explain" the mileage, or does mileage "explain" the age? In this case, both seem weird. Both are just static independent variables. Neither explains the other, if you ask me.

What am I missing here? Thanks!

  • 2
    $R^2$ can be expressed as the square of the correlation between the observed and the fitted values in a regression model. Correlation is a symmetric operation (e.g. ${\rm cor}(X,Y) = {\rm cor}(Y,X)$ for two random variables $X,Y$) so, no, order does not matter. – Macro Nov 15 '11 at 20:02
  • So, why does the above example sound so off? – JackOfAll Nov 15 '11 at 21:58
  • 2
    Perhaps these observations will help a little: notice that it's the variation in something that is being "explained" here, not the something itself. "Explained" is a particularly poor choice of words, IMHO. "Accounted for" or "accommodated by the fitted model" are closer to what's going on, but they're longer and fussier to say or write. – whuber Nov 15 '11 at 22:54
  • 6
    Incidentally, JackOfAll, are you aware of how the process of upvoting and accepting answers works on this site? It's a valuable aspect of creating great questions and answers, so please spend a few moments reading our FAQ and act accordingly on the questions you have already asked. In particular, any thoughtful, well-crafted useful answer, even if it's not the best in the group or only partially addresses your question, deserves your vote in recognition of the effort on the part of the respondent. – whuber Nov 15 '11 at 23:01
up vote 7 down vote accepted

Your wording is implying causality, which is not what the R^2 represents. "Does price (x) determine square footage (y)?" is implying causality which is not what is captured through a correlation. "Price explains X% of the variation in square footage" describes that there is a relationship between price and square footage, but not a causal one. This only implies that these variables vary together, not that price causes square footage. Its more akin to saying "In general, When price goes up X amount square footage goes up Y amount"

  • You're right, but it's ironic that people usually understand a phrase like "when price goes up, ... square footage goes up..." in the causal sense of changes in price creating changes in square footage. (Realtor, I need more room in my house: please put it on the market and then increase the asking price by 20%!) The commonplace interpretation of R^2 as "explaining" variance suffers from the same problem of colloquially implied causality, which I believe is at the root of the original question. In light of this, it might be better to restate your last sentence in different words. – whuber Nov 15 '11 at 22:51
  • Good point something like "positive relationship between" or something which avoids some kind of underlying mechanism would be better. – Dan Nov 15 '11 at 22:59

Edited with added material in response to comments by @whuber

This is an answer based on probability theory, not statistical estimates, so your mileage may vary.

If random variables $X$ and $Y$ have correlation coefficient $\rho$, then the linear least-mean-square error estimate of $Y$ given the value of $X$ is $$\hat{Y} = \mu_Y + \rho\frac{\sigma_Y}{\sigma_X}(X - \mu_X),$$ and similarly, the linear least-mean-square error estimate of $X$ given the value of $Y$ is $$\hat{X} = \mu_X + \rho\frac{\sigma_X}{\sigma_Y}(Y - \mu_Y).$$ Note that $\hat{Y}$ and $\hat{X}$ are random variables that are linear functions of $X$ and $Y$ respectively. Their means are $$\begin{align*} \mu_{\hat{Y}} &=E[\hat{Y}] = E\left[\mu_Y+\rho\frac{\sigma_Y}{\sigma_X}(X - \mu_X)\right] = \mu_Y+ \rho\frac{\sigma_Y}{\sigma_X}E[X - \mu_X] = \mu_Y\\ \mu_{\hat{X}} &= E[\hat{X}] = E\left[\mu_X+\rho\frac{\sigma_X}{\sigma_Y}(Y - \mu_Y)\right] = \mu_Y+ \rho\frac{\sigma_X}{\sigma_Y}E[Y - \mu_Y] = \mu_X \end{align*}$$ while the variances are $$\begin{align*} \sigma_{\hat{Y}}^2 &= E[(\hat{Y} - \mu_{\hat{Y}})^2] = \frac{\rho^2\sigma_Y^2}{\sigma_X^2}E[(X-\mu_X)^2] = \rho^2\sigma_Y^2\\ \sigma_{\hat{X}}^2 &= E[(\hat{X} - \mu_{\hat{X}})^2] = \frac{\rho^2\sigma_X^2}{\sigma_Y^2}E[(Y-\mu_Y)^2] = \rho^2\sigma_X^2 \end{align*}$$ Finally, the variances of the residual errors $Y - \hat{Y}$ and $X - \hat{X}$ are $\sigma_Y^2(1-\rho^2)$ and $\sigma_X^2(1-\rho^2)$ respectively. One can think of these results as follows.

If we use the mean $\mu_Y$ as an estimate for $Y$, the mean-square error is $\sigma_Y^2$, but if we know the value of $X$ and use $\mu_Y + \rho\frac{\sigma_Y}{\sigma_X}(X - \mu_X)$ as the estimate of $Y$, the mean-square error is reduced to $\sigma_Y^2(1-\rho^2)$.

If we use the mean $\mu_X$ as an estimate for $X$, the mean-square error is $\sigma_X^2$, but if we know the value of $Y$ and use $\mu_X + \rho\frac{\sigma_X}{\sigma_Y}(Y - \mu_Y)$ as the estimate of $X$, the mean-square error is reduced to $\sigma_X^2(1-\rho^2)$.

In both cases, the mean-square error is reduced by the same fraction $(1-\rho^2)$.


In terms of scatter plots (for discrete random variables or data) on a plane with coordinate axes $x$ and $y$, we have two straight lines $$ \begin{align*} y &= \mu_Y + \rho\frac{\sigma_Y}{\sigma_X}(x - \mu_X),\\ x &= \mu_X + \rho\frac{\sigma_X}{\sigma_Y}(y - \mu_Y), \end{align*} $$ of different slopes $\rho\sigma_Y/\sigma_X$ and $\sigma_Y/\rho\sigma_X$ passing through the mean point $(\mu_X,\mu_Y)$. The reason for the different slopes is that we are choosing the slope to minimize the sum of the squares of the vertical distances of the points from the line in the first case, and to minimize the sum of the squares of the horizontal distances of the points from the line in the second case. These sums of squared distances are $\sigma_Y^2(1-\rho^2)$ and $\sigma_X^2(1-\rho^2)$ respectively.

As a simple example, suppose that $(X,Y)$ takes on values $(0,0)$, $(0,1)$ and $(1,1)$ with equal probability $\frac{1}{3}$ each or we have a scatter plot with these three points. One can grind through the calculations if desired, but it should be intuitively obvious that we should estimate $Y$ as $\frac{1}{2}$ if $X = 0$ and as $1$ if $X = 1$, while we should estimate $X$ as $0$ if $Y = 0$ and as $\frac{1}{2}$ if $Y = 1$, that is the two lines have different slopes $\frac{1}{2}$ and $2$ (in fact, reciprocal slopes since $\sigma_X^2 = \sigma_Y^2 = \frac{2}{9}$ in this example). This is what probability theory gives. But if you treat the three points as a small sample from from an unknown population and use estimates of the population means, variances and correlation coefficent, then your results may be different.

  • It would help to be explicit about what this says about $R^2$ and to provide some intuition for that. Formulas are great--they provide rigor that is valued in good replies--but the question begs for explanation. So why, then, should minimizing vertical distances and minimizing horizontal distances result in the same $R^2$? – whuber Nov 16 '11 at 4:20
  • @whuber I am not a statistician and am not too sure that I know what $R^2$ is. Based on a cursory reading of what wikipedia says, it would seem to be similar to $\rho^2$, in which case Macro's comment seems to be pertinent. – Dilip Sarwate Nov 16 '11 at 15:08

Your example can legitimately be run the other way. Why not estimate square footage from price? Suppose price data is publicly available, but square footage is not. Yet you want to estimate square footage (to determine the carpet or furniture market, the likely heating cost, or whatever). It's perfectly valid to model square footage as a function of price.

In my opinion, you are getting hung up in the semantics of "independent" and "dependent" variables. Better to use "predictor" and "predicted".

Your Answer

 
discard

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Not the answer you're looking for? Browse other questions tagged or ask your own question.