# Sensitivity of regression parameters to noise

How sensitive are the parameters obtained from OLS, logistic or other regression methods to noise ?

By noise, I mean minor changes. For e.g. adding a small noise $$-1<\Delta<1$$ to $$\beta_1$$ in $$y= \beta_0 + \beta_1 x_1 + \beta_2 x_2$$ to get $$y= \beta_0 + (\beta_1 +\Delta) x_1 + \beta_2 x_2$$

How drastically would the conclusions drawn initially w.r.t the model metrics (AIC, BIC, $$R^2$$ hypothesis tests associated with the $$x_i$$ )without the noise added change ?

I understand this may change from case to case depending on the model at hand. I want to know if there is any existing theory about sensitivity.

This is in the context of security where we need to protect our models from being maliciously attacked.

Edit: By noise, I meant a situation where an attacker adds a delta($$\Delta$$) to the found beta $$\beta$$.

If the model is correctly specified with, $$y=\beta_0+\beta_1x_1+\beta_2x_2,$$ then $$\beta_0, \beta_1,$$ and $$\beta_2$$ are constants that are to be estimated. I think you mean noise in the measurement of $$x_1$$. Having noise in a regressor is referred to as measurement error and leads to biased parameter estimates. Noise in measurement of the dependent variable only amounts to efficiency losses. There is a large literature on measurement error and instrumental variable methods to deal with the problem.

• No. In this case, I have determined my $\beta$. I want to understand what would happen if someone maliciously adds a noise to it. How would my conclusions change. – rgk Mar 13 at 13:33
• $\beta$ is calculated using $x_1, x_2$ and $y$. Where else would the noise come from? – dlnB Mar 13 at 14:03

$$\beta$$ uncertainty can be quantified in terms of covariance in the sense that slightly different $$Y$$ samples would lead to very different $$\beta$$ if the $$X$$ samples were kept the same.

Technically, $$\beta$$ is obtained by writing: $$\beta = (X'X)^{-1} X'Y$$ Where $$X$$ and $$Y$$ have factors as columns and samples as rows. Then, assuming that $$X$$ is fixed means that $$\beta$$ is a linear function of $$Y$$ and the variance of $$Y$$ gives us a covariance of $$\beta$$.

We see that because of the factor inverse covariance matrix term $$(X'X)^{-1}$$, $$\beta$$ is uncertain when some factors are linearly dependent or whenever the covariance of factor determinant is too near 0.

The qualitative answer is that the certainty of each $$\beta_i$$ depends crucially on the orthogonality and relevance of the $$x_i$$ you chose to regress $$y$$.

ALso, I don't think you can assume $$\beta_i$$ noise terms will be independent though, they tend to be correlated when they are noisy.