0
$\begingroup$

I've trained a regression model which predicts the dependent variable from several independent variables. I noticed that there is a strong negative correlation (-0.86) between the dependent variable and the error (prediction - dependent variable). What does this imply?

enter image description here

While it can be caused by all predictions being similar, therefore, creating a linear relationship between the error and the dependent variable, there is also a positive correlation (0.47) between the predictions and the dependent variables.

enter image description here

Is there a way to reduce the errors for larger dependent variable values?

Improve this question
$\endgroup$
1
  • $\begingroup$ It is a necessity that $\hat{Y} - Y$ be correlated with $Y$. What you don't want is a correlation between $\hat{Y} - Y$ and $X$ or $\hat{Y}$ as this indicates systematic lack of fit. $\endgroup$
    – Frank Harrell
    Jan 16 at 13:48

2 Answers 2

Reset to default
-1
$\begingroup$

Look into class imbalance or skewed datasets. Since your dependant variable is imbalanced your accuracy will be biased towards the small values. I'd recommend trying an ensemble algorithm like Bagging. Techniques like oversampling/undersampling might also produce improvements.

Improve this answer
$\endgroup$
-1
$\begingroup$

The fact that the error is correlated with an independent variable may mean that there is an unobserved variable that is both relevant to explain the dependent variable and related to the independent variable - can you maybe figure it out? I.e. try to predict ice cream sales with the amount of snow that falls - they are correlated, but actually both variables relate to temperature.

In the meantime, you could also use transformations of the dependent variable, like its power:

y = beta_1 + beta_2 * x + beta_3 * (x²)

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.