Relation between linear regression prediction accuracy and correlation

Question

This question is more of theoretical. I am not sure if this is the right place, but still giving it a try.

I have two variables — direct cost and indirect cost. When sales persons go for a sales pitch to a customer they know about direct cost that they are going to incur for this service, but they don't know much about indirect cost (they will come to know about it in latter stages). An estimate of indirect cost at this stage will be valuable for sales persons.

I am trying to predict indirect cost as a function of direct cost. I am doing this via a simple linear regression. I plotted scatter plot between direct cost and indirect cost and see a good linear relationship between them. I also see that direct cost and indirect cost are highly corelated to each other with correlation coefficient as 0.98, so I expected a very good prediction accuracy. But surprisingly, my prediction accuracy is not so good. I have around 200,000 points in my training data and average prediction error on training data is 17 %. Though adjusted R-Square value is 0.97. I am using lm() function from R.

My question is that in case of simple linear regression, in general, should we expect better prediction accuracy if dependent and independent variables are highly correlated or is it my misconception? If we expect good accuracy, am I missing something here. Please note that I have also tried centering these variables around mean.

Answer 1

In a linear regression you are trying to find the best fit of a set of output $Y=(Y_1,\dots,Y_n)$ as a linear function of input $X=(X_1,\dots,X_n)$, i.e. you want the $a$ and $b$ that gives the smallest residuals:

$$(a,b)= arg \min \sum_i (Y_i-aX_i-b)^2$$

if it is what you did and you formally get (if you are still working with the same $n$ observations)

$$a=arg\min Var(Y-aX), \;\; b=\bar{Y}-a\bar{X},\;\; and \;\; R^2=\frac{var(aX)}{var(Y)}=corr(X,Y)$$ (where for any two vector $U,V$ $\bar{U}$ is the empirical mean $Var(U)$ and $corr(U,V)$ the correlation).

You do not say how you compute your errors but if you are computing $var(Y-ax)$ then it is equal to $(1-R^2)*var(Y)$ so if you normalise the errors by $var(Y)$ then you have

$$Error=\frac{var(Y-ax)}{var(Y)}=1-R^2$$

which in your case should be 0.03 (because $R^2$ is 0.97). But what you did (I guess :) ) is that you computed the root mean error divided by the standard deviation of the output wich is exactly $\sqrt{Error}$ (according to my definition).

Your results are perfectly correct and conform to the theory because

$$ \sqrt{Error}=\sqrt{0.03}=0.1732051$$

nothing surprising !

CQFD

Answer 2

I think I know what's going on. Intuitiviely, your method of calculating prediction error seems OK - You take the residual value (i.e error) and see how big it is relative to your data (i.e., relative to either the predicted value or the value of your dependent variable) by dividing by that value. However, there's at lease one issue with calculating error in this way that I can think of: When you divide by the value of the dependent variable, you are expecting the divisor not be close to zero - when it is, your residual/(actual value) can become huge, simply because the actual value may be a fraction (i.e., close to zero...e.g., .005). I ran the following simulations and confirmed my suspicion:

# we'll run about 150 linear models. The x value will be closely related to the y value (linear relationship), meaning that if x is close to 0, y will be close to 0. The mean of the independent variable (x) is stored in the following vector:
seq(-50, 50, .7) -> mean_vals
# dataframe to save our regression results in:
data.frame(r_sqrd = NA, resid_over_value = NA, mean_vals = NA) -> frame

for(i in 1:length(mean_vals)){
    x <- rnorm(n = 1000, mean = mean_vals[i], sd = 10)
    x + rnorm(n = 1000, mean = 0, sd = 5) -> y
    lm(formula = y ~ x) -> k
    # This - or some close relative of this - is how you are calculating error 
    k$residuals/y -> z
mean(z) -> frame[i,"resid_over_value"]
# THe Rsquared value is a much more typical measure of fit/prediction error for a linear model:
summary(k)$r.squared -> frame[i, "r_sqrd"]
    # Put the mean of the x values in the frame so we can see how your method of prediction error changes as a function of mean
    mean_vals[i] -> frame[i, "mean_val"]
}

As you can see in the plot below, your method of error does not increase as a function of Rsquared (which is an issue in itself, though I'm not 100% sure on why this is happening)...The issue that I focused in on had to do with the outliers in the below graph.

plot(frame$r_sqrd, frame$resid_over_value, xlab = "Rsquared", ylab = "Error according to your method")

...Those points generally occur when the mean of the predictor variable is close to zero, as can be seen in the below graph:

plot(frame$mean_val, frame$resid_over_value, xlab = "Mean of independent variable", ylab = "Error according to your method")

To conclude, I suggest you use a more standard version of model fit/prediction error such as the mean squared error (MSE) or the rquared.