Sensitivity of regression parameters to noise

Question

How sensitive are the parameters obtained from OLS, logistic or other regression methods to noise ?

By noise, I mean minor changes. For e.g. adding a small noise $-1<\Delta<1$ to $\beta_1$ in $y= \beta_0 + \beta_1 x_1 + \beta_2 x_2$ to get $y= \beta_0 + (\beta_1 +\Delta) x_1 + \beta_2 x_2$

How drastically would the conclusions drawn initially w.r.t the model metrics (AIC, BIC, $R^2$ hypothesis tests associated with the $x_i$ )without the noise added change ?

I understand this may change from case to case depending on the model at hand. I want to know if there is any existing theory about sensitivity.

This is in the context of security where we need to protect our models from being maliciously attacked.

Edit: By noise, I meant a situation where an attacker adds a delta($\Delta$) to the found beta $\beta$.

Answer 1

If the model is correctly specified with, $$y=\beta_0+\beta_1x_1+\beta_2x_2,$$ then $\beta_0, \beta_1,$ and $\beta_2$ are constants that are to be estimated. I think you mean noise in the measurement of $x_1$. Having noise in a regressor is referred to as measurement error and leads to biased parameter estimates. Noise in measurement of the dependent variable only amounts to efficiency losses. There is a large literature on measurement error and instrumental variable methods to deal with the problem.

Answer 2

$\beta$ uncertainty can be quantified in terms of covariance in the sense that slightly different $Y$ samples would lead to very different $\beta$ if the $X$ samples were kept the same.

Technically, $\beta$ is obtained by writing: $$\beta = (X'X)^{-1} X'Y$$ Where $X$ and $Y$ have factors as columns and samples as rows. Then, assuming that $X$ is fixed means that $\beta$ is a linear function of $Y$ and the variance of $Y$ gives us a covariance of $\beta$.

We see that because of the factor inverse covariance matrix term $(X'X)^{-1}$, $\beta$ is uncertain when some factors are linearly dependent or whenever the covariance of factor determinant is too near 0.

The qualitative answer is that the certainty of each $\beta_i$ depends crucially on the orthogonality and relevance of the $x_i$ you chose to regress $y$.

ALso, I don't think you can assume $\beta_i$ noise terms will be independent though, they tend to be correlated when they are noisy.