How to tell when factors “disagree” in linear regression to produce noisy predictions?

Question

I use a regression as my predictor. Let's say my regression is $y = a_1 x_1 + a_2 x_2 + a_3 x_3$

I realized that in practice, when my prediction is way off, it's usually because one factor significantly skewed the prediction. For example, $x_1, x_2$ are both slightly negative, while $x_3$ is very positive.

In this case (when factors "disagree" with each other), I would rather my predictor not do anything, than reporting a "controversal" prediction.

In my particular application, I don't have to generate a prediction every single time (it can simply returns "no clue"), but when I do, I would rather it be correct.

What is the best way of doing this?

Thanks!

Answer 1

Based on your description, it sounds like you basically want to be able to tell which predictions are "unstable". As whuber hinted at in the comments, you can get some handle on this by estimating the variance of your predictions and making decisions about their stability of based on that, assuming the model you've fit to the data is the correct model.

To be more concrete suppose you're trying to predict $y$ from a set of predictors $\{x_{1}, ..., x_{p} \}$. These predictors do not need to be independent but there cannot be exact collinearity among them. Suppose the true data generating model is

$$ y = \beta_0 + \beta_1 x_{1} + ... + \beta_p x_{p} + \varepsilon $$

where $\varepsilon \sim N(0,\sigma^2)$ and you fit the model by least squares and make predictions with the estimated coefficients. That is, for a new observation $\{x_1, ..., x_p \}$, your prediction is

$$ \hat{y} = \hat \beta_0 + \hat \beta_1 x_1 + ... + \hat \beta_p x_p $$

Then

$$ {\rm var}( \hat{y} ) = \sigma^2 + \sum_{j=0}^{p} \sum_{k=0}^{p} x_j x_k {\rm cov}(\hat \beta_j, \hat \beta_k)$$

where $x_0 = 1$. Also note that ${\rm cov}(\hat \beta_j, \hat \beta_j) = {\rm var}(\hat \beta_j)$. In practice we do not know the variances and covariances that appear in ${\rm var}(\hat y)$ but $\sigma^2$ can be estimated by the residual variance - $\hat \sigma^2$ - and the covariance matrix of the $\hat \beta$s can be estimated by

$$ \widehat{ {\rm cov}(\hat \beta_j, \hat \beta_k) } = \hat{\sigma}^2 [({\bf X'X})^{-1}]_{kj}$$

where $${\bf X} = \left( \begin{array}{cccc} 1 & x_{11} & \cdots & x_{1p} \\ 1 & x_{21} & \cdots & x_{2p} \\ \vdots & \vdots & \vdots & \vdots \\ 1 & x_{n1} & \cdots & x_{np} \\ \end{array} \right) $$ is the matrix of predictor values used to fit the model.

From here you can calculate the variance of each prediction and the so-called "controversial" predictions would be indicated by very large variances, which would give you a basis for diagnosing whether or not a prediction will be reliable and for making your algorithm decree "no clue" instead of making a bad prediction, as you put it. Again, this logic is only valid if the linear model is the correct model, so I highly recommend carrying out some goodness of fit diagnostics before putting a lot of faith in this.

Answer 2

Not sure if this is what you are looking for but: There could be many things that cause your predictions to be off, some are:

1. The values you are predicting are outside the scope of your regression. When predicting values it is good practice to keep the predictions within (or at least around) the values of x that you used in your model.

2. Model specification. Make sure the model is really linear. Use your fitted values and the residuals, plot them on a graph with the line. They should be linear, otherwise you may need to rethink your model.

3. Ttry looking at the correlation between your $x$'s.

4. A dirty way (not too good) is run the regression by omitting one of the variables, take out $x_2$ or $x_3$. See if you get a better fit.

5. Check for unusual data: plot your data and look for outliers, then check for high leverage.

There are tons of others, but this should be a good start.

Answer 3

You appear to want to know for which cases (perhaps as identified by their predictor variables) your model will be able to predict your outcome highly accurately. Don't assume that this is possible.

You haven't told us anything about the actual data you are modelling (which might get you better answers), but I'm a social scientist and I deal with sets of outcomes and predictors which leave me unable to identify, a priori, which cases I will be able to predict outcomes with a high degree of accuracy.

@whuber suggested you look into prediction intervals. This is a good idea. I'm guessing that what you will find is that you will consider the prediction intervals to be be moderately wide for most of your cases... I'm sure you can improve your model, include interactions and squared terms. Graph residuals against fitted values and against individual predictors, etc.

But wait, you know your model is really accurate for some cases, why can't you just focus on those!? You know the saying, "even a stopped-clock is right twice a day"? Your model is the stopped clock. Don't pay extra attention to the stopped-clock when you notice that the current actual time is getting really close to what the stopped-clock says.

Or, as John Tukey said: The combination of some data and an aching desire for an answer does not ensure that a reasonable answer can be extracted from a given body of data.