4
$\begingroup$

I have tested a regression framework's robustness to noise and I have noticed in some cases that adding noise improves the prediction performance and in other cases the performance degrades.

What could be the reasons for this? If there are multiple reasons, how to I determine which is the cause?

Edit:

Some more details about what I am doing.

The framework uses ridge regression. The inputs are vectors of extracted image features. The outputs are vectors of angles (in degrees, -180 to 180). To test for robustness to noise I am applying 3 levels of noise (white additive Gaussian noise) to the angles (targets) proportional to the individual angle variances (2%, 5%, and 10% of the variance of each angle).

I have noticed that in some observations, adding a small amount of noise (2-5%) leads to a small improvement in performance and in one case, all levels of noise give improvement. In my tests, the regularisation term is fixed across all noise levels, and I have ran each noise level test several times to take into account the fluctuations of the random noise.

Also, I have two broad sets of observation data. The first set was observed relatively accurately, however the second set was more complex (significantly more heterogeneous, leading to notable performance degradation relative to the first set) and exhibited a number of minor errors due to the observation technique being more limited than that which was used with the first set.

In the first set, the phenomena of better performance through adding noise did not occur. However, sometimes more noise was better than less noise.

If more information is required to better answer the question, I'd be happy to provide it.

$\endgroup$
0

2 Answers 2

8
$\begingroup$

Your description is quite sketchy. Adding noise can (seem to) improve prediction if the method of developing the predictions is overfitting. Likewise if you are overfitting you can improve prediction by deleting progressively more of your data. Depending on your sample size, "improvements" are best demonstrated by bootstrapping in 100-fold repeats of 10-fold cross-validation.

$\endgroup$
2
  • $\begingroup$ What further information would assist in answering this question? $\endgroup$
    – Josh
    Jun 11, 2011 at 13:48
  • $\begingroup$ I see your further comments but don't have any insights right now. $\endgroup$ Jun 11, 2011 at 20:53
4
$\begingroup$

If you are already using ridge regression, I think you can get the best performance by finding the correct value of the regularization term though n-fold cross-validation.

To address the reason why adding white-noise will increase the prediction performance, as far as I remember, adding white-noise to the data set is locally equivalent to $l_2$-regularization.

May have a look:

C. M. Bishop. Training with noise is equivalent to Tikhonov regularization. Neural Computation 7(1):108-116, 1995.

Ridge regression is already $l_2$-regularization. Adding noise, from my point of view, is redundant, find the correct value of regularization term by n-fold cross-validation should be better.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.