10
$\begingroup$

Suppose we are given a set of data of the form $(y,x_{1},x_{2},\cdots, x_{n})$ and $(y,x_{1},x_{2},\cdots, x_{n-1})$. We are given the task of predicting $y$ based on values of $x$. We estimate two regressions where: $$ \begin{align} y &=f_{1}(x_{1},\cdots, x_{n-1}, x_{n}) \tag{1} \\ y &=f_{2}(x_{1},\cdots, x_{n-1}) \tag{2} \end{align} $$

We also estimate a regression that predicts values of $x_{n}$ based on values of $(x_{1},\cdots, x_{n-1})$, that is: $$ x_{n}=f_{3}(x_{1},\cdots, x_{n-1}) \tag{3} $$

Suppose now we are given values of $(x_{1},\cdots, x_{n-1})$, then we would have two different methods to predict $y$:

$$ \begin{align} y&=f_{1}(x_{1},\cdots, x_{n-1},f_{3}(x_{1},\cdots,x_{n-1})) \tag{4} \\ y&=f_{2}(x_{1},\cdots, x_{n-1}) \tag{5} \end{align} $$

Which one would be better in general?

I am guessing that the first equation would be better because it utilizes information from the two forms of data points whereas the second equation utilizes information from only data points that have $n-1$ predictor values. My training in statistics is limited and thus I would like to seek some professional advice.

Also, in general, what is the best approach toward data that have incomplete information? In other words, how can we extract the most information from data that do not have values in all $n$ dimensions?

$\endgroup$
3
  • $\begingroup$ Estimating with actuals vs estimating with estimates - you decide :) $\endgroup$
    – PhD
    May 21, 2012 at 6:44
  • $\begingroup$ Really it's that simple? $\endgroup$
    – Xiaowen Li
    May 21, 2012 at 7:14
  • $\begingroup$ The answer may be, it depends. How much data is missing? How much data do you have overall? How many predictors do you have? $\endgroup$
    – Joel W.
    May 21, 2012 at 14:12

1 Answer 1

6
$\begingroup$

+1, I think this is a really interesting and clearly stated question. However, more information will help us think through this situation.

For example, what is the relationship between $x_n$ and $y$? It's quite possible that there isn't one, in which case, regression $(1)$ offers no advantage relative to regression $(2)$. (Actually, it is at a very slight disadvantage, in the sense that the standard errors will be slightly larger, and thus betas might be slightly further, on average, from their true values.) If there is a function mapping $x_n$ to $y$, then, by definition, there is real information there, and regression $(1)$ will be better in the initial situation.

Next, what is the nature of the relationship between $(x_1, \cdots, x_{n-1})$ and $x_n$? Is there one? For instance, when we conduct experiments, (usually) we try to assign equal numbers of study units to each combination of values of the explanatory variables. (This approach uses a multiple of the Cartesian product of the levels of the IV's, and is called a 'full factorial' design; there are also cases where levels are intentionally confounded to save data, called 'fractional factorial' designs.) If the explanatory variables are orthogonal, your third regression will yield absolutely, exactly 0. On the other hand, in an observational study the covariates are pretty much always correlated. The stronger that correlation, the less information exists in $x_n$. These facts will modulate the relative merits of regression $(1)$ and regression $(2)$.

However, (unfortunately perhaps) it's more complicated than that. One of the important, but difficult, concepts in multiple regression is multicollinearity. Should you attempt to estimate regression $(4)$, you will find that you have perfect multicollinearity, and your software will tell you that the design matrix is not invertible. Thus, while regression $(1)$ may well offer an advantage relative to regression $(2)$, regression $(4)$ will not.

The more interesting question (and the one you're asking) is what if you use regression $(1)$ to make predictions about $y$ using the estimated $x_n$ values output from the predictions of regression $(3)$? (That is, you're not estimating regression $(4)$—you're plugging the output from the prediction equation estimated in regression $(3)$ into prediction model $(4)$.) The thing is that you aren't actually gaining any new information here. Whatever information exists in the first $n-1$ predictor values for each observation is already being used optimally by regression $(2)$, so there is no gain.

Thus, the answer to your first question is that you might as well go with regression $(2)$ for your predictions to save unnecessary work. Note that I have been addressing this in a fairly abstract way, rather than addressing the concrete situation you describe in which someone hands you two data sets (I just can't imagine this occurring). Instead, I'm thinking of this question as trying to understand something fairly deep about the nature of regression. What does occur on occasion, though, is that some observations have values on all predictors, and some other observations (within the same dataset) are missing some values on some of the predictors. This is particularly common when dealing with longitudinal data. In such a situation, you want to investigate multiple imputation.

$\endgroup$
5
  • $\begingroup$ Thank you Gung for your detailed answer and you help at modifying the wording of my question. I will respond once I fully interpret your answer. For your information, this is a observational study about the price of light bulbs. $x_n$ include life hours, luminosity, and color temperature of the bulb. Information are collected from retailers who usually does not give everything, resulting in missing predictors. Nevertheless we are trying to make the most out of the information we collected. $\endgroup$
    – Xiaowen Li
    May 21, 2012 at 20:50
  • 1
    $\begingroup$ OK, I thought this was just about understanding regression. I would look into multiple imputation. $\endgroup$ May 21, 2012 at 21:18
  • $\begingroup$ Thanks Gung for your insight. You are right that no new information is gained by using equation 4. Imputation turns out to be exactly what I needed. And you are right, I did run into multiple collinearity, giving me a very large p value for the coefficients. Then I was faced with choice of either reducing the number of variables get a smaller p value for the coefficients, or get a larger $r^2$ and larger p. I guess life is filled with trade offs. $\endgroup$
    – Xiaowen Li
    May 27, 2012 at 9:36
  • $\begingroup$ Thanks again for your abstract discussions on regressions. Statistics can be beautifully intriguing if we look at it as a method to find the truth. I will look into it more once I am done with my data set:) $\endgroup$
    – Xiaowen Li
    May 27, 2012 at 9:38
  • $\begingroup$ You should check out parametric fractional imputation. This is work done by Jae Kwang Kim at Iowa State that might be perfect for this situation. See biomet.oxfordjournals.org/content/98/1/119.abstract $\endgroup$ Jan 26, 2015 at 20:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.