8
$\begingroup$

This question is more of theoretical. I am not sure if this is the right place, but still giving it a try.

I have two variables — direct cost and indirect cost. When sales persons go for a sales pitch to a customer they know about direct cost that they are going to incur for this service, but they don't know much about indirect cost (they will come to know about it in latter stages). An estimate of indirect cost at this stage will be valuable for sales persons.

I am trying to predict indirect cost as a function of direct cost. I am doing this via a simple linear regression. I plotted scatter plot between direct cost and indirect cost and see a good linear relationship between them. I also see that direct cost and indirect cost are highly corelated to each other with correlation coefficient as 0.98, so I expected a very good prediction accuracy. But surprisingly, my prediction accuracy is not so good. I have around 200,000 points in my training data and average prediction error on training data is 17 %. Though adjusted R-Square value is 0.97. I am using lm() function from R.

My question is that in case of simple linear regression, in general, should we expect better prediction accuracy if dependent and independent variables are highly correlated or is it my misconception? If we expect good accuracy, am I missing something here. Please note that I have also tried centering these variables around mean.

$\endgroup$
11
  • 3
    $\begingroup$ This is a good question...for CrossValidated, which is the SE site for statistical analysis. I've flagged your question to be closed and migrated there. $\endgroup$
    – TARehman
    Oct 6, 2015 at 20:51
  • 3
    $\begingroup$ I'm curious to see how a model with 0.97 adjusted R-squared gives you such a poor predictive capability. What happens if you try to predict the training dataset? Do you use the predict function, or you try to manually use the coefficients? Do you have an intercept in your model? If you don't want to post your data would be good to simulate a pair of highly correlated variables and perform a similar analysis and check if you find the same problem if you follow exactly the same process. $\endgroup$
    – AntoniosK
    Oct 6, 2015 at 20:55
  • $\begingroup$ @AntoniosK, sorry, there was a typing mistake from my side! The accuracy that I talk about is on training data itself. I just edited it in my question. I did not use 'predict()' function, lm object provides error/residuals on training data. I am using residuals for my analysis. The suggestion of analysis on simulated variables looks good, I will give it a try. $\endgroup$ Oct 6, 2015 at 21:26
  • $\begingroup$ how are you defining prediction accuracy? typically a measure of accuracy is used for qualitative dependent variables (i.e., for classification problems), not when your dependent var. is continuous. $\endgroup$
    – Chris
    Oct 6, 2015 at 21:29
  • $\begingroup$ Is it possible to post the model output and explain how you found that 17%? @Chris has a point here and I guess we had similar thoughts. Also, I think you can see the predictive capability of your model if you predict (training and test datasets) and then you plot predicted values against the true values. $\endgroup$
    – AntoniosK
    Oct 6, 2015 at 21:30

2 Answers 2

3
$\begingroup$

In a linear regression you are trying to find the best fit of a set of output $Y=(Y_1,\dots,Y_n)$ as a linear function of input $X=(X_1,\dots,X_n)$, i.e. you want the $a$ and $b$ that gives the smallest residuals:

$$(a,b)= arg \min \sum_i (Y_i-aX_i-b)^2$$

if it is what you did and you formally get (if you are still working with the same $n$ observations)

$$a=arg\min Var(Y-aX), \;\; b=\bar{Y}-a\bar{X},\;\; and \;\; R^2=\frac{var(aX)}{var(Y)}=corr(X,Y)$$ (where for any two vector $U,V$ $\bar{U}$ is the empirical mean $Var(U)$ and $corr(U,V)$ the correlation).

You do not say how you compute your errors but if you are computing $var(Y-ax)$ then it is equal to $(1-R^2)*var(Y)$ so if you normalise the errors by $var(Y)$ then you have

$$Error=\frac{var(Y-ax)}{var(Y)}=1-R^2$$

which in your case should be 0.03 (because $R^2$ is 0.97). But what you did (I guess :) ) is that you computed the root mean error divided by the standard deviation of the output wich is exactly $\sqrt{Error}$ (according to my definition).

Your results are perfectly correct and conform to the theory because

$$ \sqrt{Error}=\sqrt{0.03}=0.1732051$$

nothing surprising !

CQFD

$\endgroup$
5
  • $\begingroup$ I recalculated errors as per your equations, and my result align with these euqations. As you say, it does not look so surprising now :-). $\endgroup$ Oct 7, 2015 at 16:55
  • $\begingroup$ I just received enough privilege to do upvote :-) $\endgroup$ Oct 8, 2015 at 16:15
  • $\begingroup$ Sorry, just catching up to this thread now...Can someone explain how the average value of residual/Y is the same as Error according to the equation listed above? Error according to the equation above is a function of variance, so how would this equate to mean(residual/Y)? $\endgroup$
    – Chris
    Oct 9, 2015 at 4:22
  • $\begingroup$ Hi Chris I'm not too sure I get your point. I don't compute the average value of residual/Y ? maybe one of my formulation is unclear and should be rephrased ? $\endgroup$ Oct 9, 2015 at 7:51
  • $\begingroup$ The way that the error was calculated according to the user who posed was the average value of the residual/y. They came up with a prediction error of 17% with this method, which I thought you were saying was a correct way to calculate error? I was claiming that mean(residual/predicted) is not a good way to calculate model fit, and I thought you were saying it was appropriate? Maybe I'm missing something here. $\endgroup$
    – Chris
    Oct 9, 2015 at 15:08
3
$\begingroup$

I think I know what's going on. Intuitiviely, your method of calculating prediction error seems OK - You take the residual value (i.e error) and see how big it is relative to your data (i.e., relative to either the predicted value or the value of your dependent variable) by dividing by that value. However, there's at lease one issue with calculating error in this way that I can think of: When you divide by the value of the dependent variable, you are expecting the divisor not be close to zero - when it is, your residual/(actual value) can become huge, simply because the actual value may be a fraction (i.e., close to zero...e.g., .005). I ran the following simulations and confirmed my suspicion:

# we'll run about 150 linear models. The x value will be closely related to the y value (linear relationship), meaning that if x is close to 0, y will be close to 0. The mean of the independent variable (x) is stored in the following vector:
seq(-50, 50, .7) -> mean_vals
# dataframe to save our regression results in:
data.frame(r_sqrd = NA, resid_over_value = NA, mean_vals = NA) -> frame

for(i in 1:length(mean_vals)){
    x <- rnorm(n = 1000, mean = mean_vals[i], sd = 10)
    x + rnorm(n = 1000, mean = 0, sd = 5) -> y
    lm(formula = y ~ x) -> k
    # This - or some close relative of this - is how you are calculating error 
    k$residuals/y -> z
mean(z) -> frame[i,"resid_over_value"]
# THe Rsquared value is a much more typical measure of fit/prediction error for a linear model:
summary(k)$r.squared -> frame[i, "r_sqrd"]
    # Put the mean of the x values in the frame so we can see how your method of prediction error changes as a function of mean
    mean_vals[i] -> frame[i, "mean_val"]
}

As you can see in the plot below, your method of error does not increase as a function of Rsquared (which is an issue in itself, though I'm not 100% sure on why this is happening)...The issue that I focused in on had to do with the outliers in the below graph.

plot(frame$r_sqrd, frame$resid_over_value, xlab = "Rsquared", ylab = "Error according to your method")

enter image description here

...Those points generally occur when the mean of the predictor variable is close to zero, as can be seen in the below graph:

plot(frame$mean_val, frame$resid_over_value, xlab = "Mean of independent variable", ylab = "Error according to your method")

enter image description here

To conclude, I suggest you use a more standard version of model fit/prediction error such as the mean squared error (MSE) or the rquared.

$\endgroup$
3
  • $\begingroup$ That's great and very useful info for @user3697157 to investigate more. However, he specified he's working with costs, so I don't expect negative values, but definitely some close to zero. This agrees with my assumption that maybe some/few cases generate big errors (in the way he defined them) and increase the mean prediction error. Maybe those few cases are the ones with costs close to zero. $\endgroup$
    – AntoniosK
    Oct 7, 2015 at 11:26
  • $\begingroup$ @Chris, Thanks! While carrying initial analysis, I noticed that percentage error are higher when dependent values are close to zero. This is why, I fit regression model only on the data for which dependend values are more than 10000. 'regData1 <- subset(regData, Direct.Cost> 10000)' Based on your answer, I analyzed error again. I can see that still percentage errors are higher when dependent value is small (between 10,000 to 100,000) and it is within acceptable range for higher dependent values. I need to do rethink on my approach. My dependent value range from zero to millions. $\endgroup$ Oct 7, 2015 at 14:21
  • $\begingroup$ @AntoniosK, Yes. Few of my cases (with smaller dependent values) are generating unusually high errors. I am rethinking on my approach. Any suggestions are welcome! $\endgroup$ Oct 7, 2015 at 14:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.