7
$\begingroup$

If we have X and Y that are mathematically dependent: X = Y + Z, is it 'forbidden' to use Y as a predictor to X in linear regression? I'm trying to find a concise explanation for why it is, or isn't.

One explanation I've found is that you shouldn't use simple linear regression in this case, but multiple linear regression, with Y and Z as predictors, but this doesn't seem right. The mathematical relationship is exact, and the multiple regression would return coefficients 0, 1, and 1 for X = 0 + 1Y + 1Z. So, there is something more fundamental I'm missing.

The purpose for linear regression is predicting the value of one variable from the other. But we already know that X can be calculated as the sum of Y and Z, so effectively, we are calculating the regression between Y and Y + Z. Is this 'forbidden'?

But, if we can only measure Y, does the regression give us a tool to predict X? Isn't it better to find the regression between Y and Z? I'm confused.

Improve this question
$\endgroup$
13
  • $\begingroup$ I'm not sure I understand what you mean by "forbidden". Why would linear regression forbid the usage of some variables? $\endgroup$
    – gunes
    Commented Jun 9, 2022 at 14:21
  • $\begingroup$ Of course, you can do anaything in mathematics, but maybe the interpretation is problematic, and that is why people don't do it - it is "fobridden". Like using linear regression when the relationship is very nonlinear; or when the variables are not homoscedastic. $\endgroup$
    – amc____
    Commented Jun 9, 2022 at 14:30
  • 1
    $\begingroup$ Regression models do their best to view the response variable (here, $X$) as a linear combination of other variables (such as $Y+Z$). You are, in effect, asking if it's forbidden to use linear regression in precisely the situation it's designed for! $\endgroup$
    – whuber ♦
    Commented Jun 9, 2022 at 14:38
  • $\begingroup$ In this situation it is given that X=0 + 1Y+1Z. For example, gross pay X is the sum of net pay Y and all deductions Z. The question is, Is it ever useful to find a relationship, using linear regression, between gross pay and net pay? Or is this a conceptual error? $\endgroup$
    – amc____
    Commented Jun 9, 2022 at 14:50
  • 2
    $\begingroup$ If Z is zero-centered noise with an unknown value, it certainly makes sense to use Y as a predictor for X, in my opinion. $\endgroup$
    – HelloGoodbye
    Commented Jun 10, 2022 at 13:24

3 Answers 3

Reset to default
20
$\begingroup$

If you know $X = Y + Z$ and you have $Y$ and $Z$ measured, why would you need to run a regression of $X$ on $Y$ and $Z$? It provides no additional information and does not allow you to make "better" predictions about $X$ (since you know $X$ exactly from $Y$ and $Z$). But if you don't have $Z$, don't know the exact relationship between $Y$ and $X$, and want to predict $X$ from $Y$, then you absolutely can (and should!) regress $X$ on $Y$. The true marginal relationship between $X$ and $Y$ depends on the correlation between $Y$ and $Z$, which is not stated in the problem, so you won't automatically know what the marginal relationship between $X$ and $Y$ is from the formula for $X$ alone. Indeed, this is the usual case that we do regression: we assume a (possibly) deterministic relationship between the outcome and some other variables, but many of those variables are unmeasured, so their influence is captured in the error term. In the absence of measured $Z$, the effect of $Z$ will be captured in the error term, and the marginal effect of $Y$ will be estimated by the model.

Improve this answer
$\endgroup$
8
  • $\begingroup$ But if you don't have Z and want to predict X from Y, then you absolutely can (and should!) regress X on Y.-> But couldn't you just calculate Z as X-Y, instead of correlating X and Y? $\endgroup$
    – amc____
    Commented Jun 9, 2022 at 15:01
  • $\begingroup$ To clarify - the important information here is the correlation between Z and Y. If you have it, and you measure Y, you can find Z from the statistical relationship, and X as the sum of Y and Z. Alternatively, if you try first regressing Y on X, you might get a very high correlation, because you are effectively doing regression of Y on Y + Z - this is usually going to be very high - for random variables Y and Z the r will tend to 0.7. $\endgroup$
    – amc____
    Commented Jun 9, 2022 at 15:11
  • 4
    $\begingroup$ What you're saying makes sense. I just mean that even if there is a deterministic relationship between {Y and Z} and X, if you don't know what that relationship is and you don't have Z, then a regression of X on Y is informative. If you know everything, you don't need to fit statistical models. That's the most confusing part about your question; the whole point of doing statistics is to estimate an unknown truth from sample data; in your case you have the data-generating parameters, so there is no reason to do statistics. It's not "forbidden", just useless. $\endgroup$
    – Noah
    Commented Jun 9, 2022 at 16:17
  • $\begingroup$ Thanks, I apprechiate the answers, it does seem like it would be useless if we know all the data. I'm having trouble with the subtle issue of correlating the Y with the sum Y+Z. Would you say that correlating Y and Y+Z is a statistical or conceptual error, (both)? Would it not give spurius correlations, unless for special values of Z? $\endgroup$
    – amc____
    Commented Jun 9, 2022 at 17:06
  • 1
    $\begingroup$ Where what is practically useful? It's not useful to think about purely deterministic relationships in statistics. Understanding the size of bias in the case of omitted variables is a major application and essentially the focus of the entire field of causal inference and a major part of econometrics. Look into omitted variable bias and unmeasured confounding to see this in action and the formulas used to quantify the bias. The covariance between the observed variable and omitted variable is a major component. $\endgroup$
    – Noah
    Commented Jun 9, 2022 at 19:44
6
$\begingroup$

Linear regression is a tool that is used to achieve a goal. So any answer will depend on the goal to be achieved. As said already in the answer of @Noah, if you already know $X$, $Y$, and $Z$, I can't see any goal worth achieving that is achieved by this regression. Why would you want imprecise predictions of $X$ if you can have precise ones?

If however you don't know $Z$, linear regression may work well for predicting $X$ from $Y$. There is nothing that formally forbids you from trying that out (in fact if $Z$ is independent of $Y$ and distributed according to standard regression model assumptions, the standard regression model is just fulfilled). But then, depending on the exact nature of the data (particularly the distribution of $Z$ and how it is related to $Y$), it may not work that well, and/or other techniques may work better. This can be explored for example using bootstrap or cross-validation.

Improve this answer
$\endgroup$
5
$\begingroup$

Two people step on a scale. The scale outputs only the total weight of the couple.
If you know the weight of the first person ($Y$) can you guess what the scale will output ($X$) without knowing anything about the weight of the second person ($Z$)?

Intuitively, there has to be some connection. If $Y$ is very heavy, it is likely that the total weight $X$ will also be high.
Regressing X on Y here is just trying to put a number to that connection by looking at a bunch of real-life examples of total weight $X$ given first person weight $Y$.
It is not forbidden, as you say. It is simply trying to make the best guess given the information you have.

Of course if you know the weight of both people $Y$ and $Z$ and you know how scales work, you don't need to make observations and regression.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.