How to tell when factors “disagree” in linear regression to produce noisy predictions?

I use a regression as my predictor. Let's say my regression is $y = a_1 x_1 + a_2 x_2 + a_3 x_3$

I realized that in practice, when my prediction is way off, it's usually because one factor significantly skewed the prediction. For example, $x_1, x_2$ are both slightly negative, while $x_3$ is very positive.

In this case (when factors "disagree" with each other), I would rather my predictor not do anything, than reporting a "controversal" prediction.

In my particular application, I don't have to generate a prediction every single time (it can simply returns "no clue"), but when I do, I would rather it be correct.

What is the best way of doing this?

Thanks!

-
It might be a case for non-linearity. Can you show a plot of fitted values vs. residuals? –  Robert Kubrick Mar 26 '12 at 23:40
Its best to avoid using the word skew like that because skewness has a precise and different definition in statistics: en.wikipedia.org/wiki/Skewness –  Michael Bishop Mar 27 '12 at 0:26
What are your goals in creating this model? To minimize the error associated with your predictions? –  Michael Bishop Mar 27 '12 at 0:39
"Confident" is an interesting word in this context, because linear regression provides information for constructing intervals of "confidence" around prediction: prediction intervals‌​. A wide interval reflects lack of "confidence" or trustworthiness in a prediction. You could use these widths to screen predictions for likely correctness. This could work well in conjunction with goodness of fit tests (which would indicate whether to trust the prediction intervals!). –  whuber Mar 27 '12 at 13:46
@rolando Yes, there is a single SE of estimation, but it doesn't tell the whole story. When predicting the value associated with $(x_1,x_2,x_3)$, you also have to account for the (correlated) uncertainties in the parameter estimates $(\hat{a}_1, \hat{a}_2, \hat{a}_3)$. This causes the prediction intervals to spread hyperbolically as the point $(x_1,x_2,x_3)$ moves further from the mean value used in estimation; the widths can even become infinite. –  whuber Apr 26 '12 at 22:54

Based on your description, it sounds like you basically want to be able to tell which predictions are "unstable". As whuber hinted at in the comments, you can get some handle on this by estimating the variance of your predictions and making decisions about their stability of based on that, assuming the model you've fit to the data is the correct model.

To be more concrete suppose you're trying to predict $y$ from a set of predictors $\{x_{1}, ..., x_{p} \}$. These predictors do not need to be independent but there cannot be exact collinearity among them. Suppose the true data generating model is

$$y = \beta_0 + \beta_1 x_{1} + ... + \beta_p x_{p} + \varepsilon$$

where $\varepsilon \sim N(0,\sigma^2)$ and you fit the model by least squares and make predictions with the estimated coefficients. That is, for a new observation $\{x_1, ..., x_p \}$, your prediction is

$$\hat{y} = \hat \beta_0 + \hat \beta_1 x_1 + ... + \hat \beta_p x_p$$

Then

$${\rm var}( \hat{y} ) = \sigma^2 + \sum_{j=0}^{p} \sum_{k=0}^{p} x_j x_k {\rm cov}(\hat \beta_j, \hat \beta_k)$$

where $x_0 = 1$. Also note that ${\rm cov}(\hat \beta_j, \hat \beta_j) = {\rm var}(\hat \beta_j)$. In practice we do not know the variances and covariances that appear in ${\rm var}(\hat y)$ but $\sigma^2$ can be estimated by the residual variance - $\hat \sigma^2$ - and the covariance matrix of the $\hat \beta$s can be estimated by

$$\widehat{ {\rm cov}(\hat \beta_j, \hat \beta_k) } = \hat{\sigma}^2 [({\bf X'X})^{-1}]_{kj}$$

where $${\bf X} = \left( \begin{array}{cccc} 1 & x_{11} & \cdots & x_{1p} \\ 1 & x_{21} & \cdots & x_{2p} \\ \vdots & \vdots & \vdots & \vdots \\ 1 & x_{n1} & \cdots & x_{np} \\ \end{array} \right)$$ is the matrix of predictor values used to fit the model.

From here you can calculate the variance of each prediction and the so-called "controversial" predictions would be indicated by very large variances, which would give you a basis for diagnosing whether or not a prediction will be reliable and for making your algorithm decree "no clue" instead of making a bad prediction, as you put it. Again, this logic is only valid if the linear model is the correct model, so I highly recommend carrying out some goodness of fit diagnostics before putting a lot of faith in this.

-
Thanks Macro, this is super helpful. –  CodeNoob Jun 27 '12 at 15:10

Not sure if this is what you are looking for but: There could be many things that cause your predictions to be off, some are:

1. The values you are predicting are outside the scope of your regression. When predicting values it is good practice to keep the predictions within (or at least around) the values of x that you used in your model.

2. Model specification. Make sure the model is really linear. Use your fitted values and the residuals, plot them on a graph with the line. They should be linear, otherwise you may need to rethink your model.

3. Ttry looking at the correlation between your $x$'s.

4. A dirty way (not too good) is run the regression by omitting one of the variables, take out $x_2$ or $x_3$. See if you get a better fit.

5. Check for unusual data: plot your data and look for outliers, then check for high leverage.

There are tons of others, but this should be a good start.

-
Should have added: check the size values of your x's. Say x1 is measured in inches, but x2 is measured in yards. That will lead to a huge difference in your a's. In this case it would be better to make x1 in yards or x2 in inches. That way your a1 and a2 are closer in magnitude. –  Travis Mar 27 '12 at 3:02