0
$\begingroup$

This (trivial) question is about the assumptions of linear regression.

[1] For example, I have a matrix X of size n*p consisting of p explanatory variables and n units, and vector y consisting of n dependent units. I want to perform linear regression on X, y. There are some common assumptions for the model such as linear relationship between $x_1, x_2, ..., x_p$ and y, uncorrelation of error terms,.... Let's name these set of assumptions $\Sigma^1$.

[2] However, let's say the result turns out to be not good (could be due to multicolinearity), so I remove one column from the matrix X to get the new matrix Z, and I perform linear regression again on Z and y. There are some common assumptions for the model such as linear relationship between $z_1, z_2, ..., z_{p-1}$ and y, uncorrelation of error terms,.... Let's name these set of assumptions $\Sigma^2$.

[3] It seems that the model is still not really good. I decide to perform PCA on X to get P, and I perform linear regression on P, y. There are some common assumptions for the model such as linear relationship between $p_1, p_2, ..., _p$ and y, uncorrelation of error terms,.... Let's name these set of assumptions $\Sigma^3$.

Every time I am not happy with my model. I modify the model by removing redundant columns or transforming the data, and I apply the model again. My question is that, for every time I do it, is it true that I have to start with a new set of assumptions (e.g. $\Sigma^2$) and completely forget assumptions that associated with previous models that you design (e.g. $\Sigma^1$)?

$\endgroup$
1
  • 1
    $\begingroup$ Yes! That's right! It might seem unprincipled to be changing our assumptions all the time, but the model building process is exactly the process of figuring out which assumptions to make about the data. So if we're doing things right, and learning about the data, that will change which assumptions are natural and necessary. (This can actually cause some problems for formal Bayesian analysis). $\endgroup$ Aug 3 at 4:17

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.