1
$\begingroup$

How sensitive are the parameters obtained from OLS, logistic or other regression methods to noise ?

By noise, I mean minor changes. For e.g. adding a small noise $-1<\Delta<1$ to $\beta_1$ in $y= \beta_0 + \beta_1 x_1 + \beta_2 x_2$ to get $y= \beta_0 + (\beta_1 +\Delta) x_1 + \beta_2 x_2$

How drastically would the conclusions drawn initially w.r.t the model metrics (AIC, BIC, $R^2$ hypothesis tests associated with the $x_i$ )without the noise added change ?

I understand this may change from case to case depending on the model at hand. I want to know if there is any existing theory about sensitivity.

This is in the context of security where we need to protect our models from being maliciously attacked.

Edit: By noise, I meant a situation where an attacker adds a delta($\Delta$) to the found beta $\beta$.

$\endgroup$
1
$\begingroup$

If the model is correctly specified with, $$y=\beta_0+\beta_1x_1+\beta_2x_2,$$ then $\beta_0, \beta_1,$ and $\beta_2$ are constants that are to be estimated. I think you mean noise in the measurement of $x_1$. Having noise in a regressor is referred to as measurement error and leads to biased parameter estimates. Noise in measurement of the dependent variable only amounts to efficiency losses. There is a large literature on measurement error and instrumental variable methods to deal with the problem.

$\endgroup$
  • $\begingroup$ No. In this case, I have determined my $\beta$. I want to understand what would happen if someone maliciously adds a noise to it. How would my conclusions change. $\endgroup$ – rgk Mar 13 at 13:33
  • $\begingroup$ $\beta$ is calculated using $x_1, x_2$ and $y$. Where else would the noise come from? $\endgroup$ – dlnB Mar 13 at 14:03
0
$\begingroup$

$\beta$ uncertainty can be quantified in terms of covariance in the sense that slightly different $Y$ samples would lead to very different $\beta$ if the $X$ samples were kept the same.

Technically, $\beta$ is obtained by writing: $$\beta = (X'X)^{-1} X'Y$$ Where $X$ and $Y$ have factors as columns and samples as rows. Then, assuming that $X$ is fixed means that $\beta$ is a linear function of $Y$ and the variance of $Y$ gives us a covariance of $\beta$.

We see that because of the factor inverse covariance matrix term $(X'X)^{-1}$, $\beta$ is uncertain when some factors are linearly dependent or whenever the covariance of factor determinant is too near 0.

The qualitative answer is that the certainty of each $\beta_i$ depends crucially on the orthogonality and relevance of the $x_i$ you chose to regress $y$.

ALso, I don't think you can assume $\beta_i$ noise terms will be independent though, they tend to be correlated when they are noisy.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.