0
$\begingroup$

I'm new to linear regression. I created a simple linear regression model (only one feature). I am checking my assumptions, and I am having difficulty with the Independence assumption. I don't really understand what effect correlated observations has on the model and its results. Can someone explain in layman's terms. I just don't really understand why it matters or why/how we even test for this assumption. I'm just trying to learn more. I know I asked a lot of questions already, but thank you in advance if anyone can help me.

$\endgroup$

1 Answer 1

0
$\begingroup$

Thanks for your questions and for joining the community. The independence assumption you're referring to is the independence in the residuals.

When you apply linear regression ($\hat{f}(x) = ax + b$, where both $a,b$ are real values), you obtain an estimated value $\hat{f}(x)$ given the values $x$. But you also have the true data values (TDV) - these are the data points in your data set. The residuals (or the "noise") are the $TDV - \hat{f}(x)$.

In layman's term, if you have correlation in your noise, it's because there's information in your data that your model is not capturing. In very broad terms, you can think of your data as a source of information that it's not readily available to you. You have to "work your data" to derive meaningful information from it. So, in your case, you applied a linear regression to your data to have a model and make some informed decisions. However, because the residuals are correlated, your model is somehow not capturing all the information your data can provide. Maybe, linear regression is not a suitable approach for your data.

Once you have your sequence of residuals ($= TDV - \hat{f}(x)$), you can can calculate the autocorrelation function for them. In most computer languages, you'll find a function that will do that for you. What you want to see for correlation is a value very close to zero. Visually, you shouldn't be able to spot patterns in the residuals' (again, vaguely) behavior. You don't want residuals with a clear trend (up or down), or with cycles, for instance. These clear patterns would mean the residuals are somehow correlated. It's also beneficial to say that you'll likely find some correlation among residuals, but you have to establish a threshold to define what is acceptable or not.

If you want a more detailed explanation with graphs that will help drive the point home, this link from a Penn State class can help.

$\endgroup$
1
  • 1
    $\begingroup$ WOW! This was extremely helpful. Thank you so much. $\endgroup$ Nov 4, 2022 at 14:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.