7
$\begingroup$

Is it possible to perform a regression where you have an unknown / unknowable feature variable?

Say I have $y_n = a_0 + a_1 x_1 + a_2 x_2 + a_3 x_3$ but I do not / cannot measure the value of the feature variable $x_3$. Can I still perform a regression to ascertain the coefficients $a_i$?

How about if I have some knowledge of the statistics of how $x_3$ is distributed? If I know that $x_3$ is drawn from a Gaussian distribution $\mathcal{N}(0, \sigma^2)$, with known $\sigma$ does this allow me to perform the regression to ascertain the values of $a_i$?

$\endgroup$
  • 4
    $\begingroup$ No, how would you know what is attributable to the feature and what is just random? $\endgroup$ – user2974951 Aug 23 at 10:56
  • 7
    $\begingroup$ If it was possible, world would be beautiful! Just imagine: no data collection! no data wrangling! no people sending you data as screenshot embeded in Word document... $\endgroup$ – Łukasz Deryło Aug 23 at 10:58
  • $\begingroup$ Thanks for the answers, just wanted some confirmation that this was the case as I suspected. I guess one could still perform the regression on the other feature variables if the effect of $x_3$ was very weak relative to the other feature variables, and so could be neglected entirely, but in the case where it is a large contribution regression is impossible. $\endgroup$ – SomeRandomPhysicist Aug 23 at 11:15
  • $\begingroup$ @user2974951 Why not turn that into an answer? I don't even think you would need to expand on that. $\endgroup$ – mkt Aug 23 at 11:19
  • $\begingroup$ What about a state space model? $\endgroup$ – Chris Haug Aug 23 at 14:20
12
$\begingroup$

The complete formula for a linear model is (in quasi matrix form)

$$Y=\beta X+\epsilon$$

So we have multiple coefficents for the variables that we are controlling for, and then we have $\epsilon$, which is everything else which we did not explain with our included variables.

In this error term belong all the variables which we did not consider, either because we do not have information for them or because we simply do not know of them (random deviation).

So there is just no way for you to know what in this term belongs to what unknown term.

$\endgroup$
  • 1
    $\begingroup$ Upvoted for the excellence and clarity of the answer. $\endgroup$ – James Phillips Aug 23 at 11:37
3
$\begingroup$

How about if I have some knowledge of the statistics of how x3 is distributed?

If you do the regression of $y$ on $x_1$ and $x_2$, then if you're willing to make educated guesses how $x_3$ correlates with each of these, you can calculate what these guesses would entail for how the coefficients you estimate would change if you could observe $x_3$ and ran the full regression.

Suppose for instance that $x_3$ isn't correlated with $x_1$. Then

$\alpha_{2, \text{your regression}} =\alpha_{2, \text{full regression}} + \alpha_3 \cdot \frac{cov(x_3, x_2)}{var(x_2)}$

So if $x_3$ is likely to be only weakly correlated with $y$ or $x_1$ and $x_2$ not much would change. And if it is, you can use these omitted-variable-bias formulas to predict how things would change.

$\endgroup$
0
$\begingroup$

It is always possible... but your estimates will be biased in many cases. The most favorable case occurs:
(a) When $x_{3n}$ is not correlated with the other regressors, in this case, regress $y_n$ on $(\iota,x_{1},x_{2})$ and you have unbiased estimates of $a_0,a_1,a_2$ (Frish-Waugh-Lovell theorem)
(b) If in addition to (a) you know $\sigma$ and $x_3 \sim \mathcal{N}(0, \sigma^2)$, then you can even identify $a_3$: draw $N$ iid values for $x_{3n} \sim \mathcal{N}(0, \sigma^2)$ and regress $y_n$ on $(\iota,x_{1},x_{2},x_{3})$.

$\endgroup$
  • $\begingroup$ Could you explain further how to perform the regression in situation (b)? Do you generate values for the unknown $x_3$ by drawing them from $\mathcal{N}(0, \sigma^2)$ and then regress on that? $\endgroup$ – SomeRandomPhysicist Aug 27 at 10:52
  • $\begingroup$ Yes, this is the way to do it. It is possible to iterate on random number generation and estimation, but this does not seem interesting in your simple case. There is a burgeoning literature on simulation based methods using a similar approach, but they are a bit more involved than in your example: see Gouriéroux and Monfort, for instance. See also the pedagogical description in the texbook of Cameron and Trivedi (2005). $\endgroup$ – Bertrand Aug 27 at 21:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.