# Determining the nature of noise

I have a general question. What kind of noise is additive, and what about multiplicative noise? How to determine the nature of noise?

Thanks a lot for your help.

-
just to clarify, since you put regression tag, are you asking how to chose between multiplicative or additive error models? If your question is from signal theory please change the tag. –  mpiktas Mar 8 '11 at 8:43

One thing that you must recognise is that the term "noise" is relative not absolute. Thinking about an obvious example (not anything to do with statistics per se), imagine you are at a night club. What is "noise" in here?

If you trying to have a conversation with someone, then the music is "noise", and so are the other conversations going on inside the night club. The "signal" is the dialogue of the converstion.

If you are dancing, then the music is no longer "noise", but it becomes the "signal" to which you react to. The conversation has changed from "signal" into "noise" merely by a change in your state of mind!

Statistics works in exactly the same way (you could in theory develop a statistical model which describes both these "noise" processes).

In a regression setting, take the simple linear case with 1 covariate, X, and 1 dependent variable Y. What you are effectively saying here is that you want to extract the linear component of X that is related to Y. The general conditions for "additive noise", so that you have:

$$Y_{i}=a+b X_{i} + n_{i}$$

Is "small noise", or more precisely in a mathematical sense, the conditions for Taylor Series linearisation are good enough for your purposes. To show this in the multiplicative case, suppose the actual distribution is:

$$Y_{i}=(a+b X_{i})n_{i}^{(T)}$$

Which we can consider as a function of $n_{i}^{(T)}$, a taylor series expansion about the value 1 gives:

$$(a+b X_{i})n_{i}^{(T)}=a+b X_{i}+(a+b X_{i})(n_{i}^{(T)}-1)$$

If the noise is "small" then it should not differ much from 1, and so the second term will be much small than the first, when the noise is "small" compared to the "signal", which is given by the regression line. So we can make the approximation

$$(a+b X_{i})n_{i}^{(T)} \approx a+b X_{i}+n_{i}^{(A)}$$

Where the approximate noise ignores the dependence on the actual regression line. This dependence will only matter when the noise is large, compared to the slope. If the slope does not vary appreciably over the range of the model, then the "fanning" of the true noise will be indistinguishable from independent noise. This also applies for the general case, for any function g satisfying $g(1)=1$:

$$(a+bX)g(n_{i}^{(T)}) \approx a+b X_{i}+(g(n_{i}^{(T)})-1)(a+bX)g^{(1)}(n_{i}^{(T)})$$ $$\approx a+b X_{i}+n_{i}^{(A)}$$

Where

$$g^{(1)}(x)=\frac{\partial g(x)}{\partial x}$$

But note that this approximation will only apply in the case of "small noise", or that $g(n_{i}^{(T)}) \approx 1$. This "smallness" make all the details of the particular function g irrelevant for all practical purpose. Going through the laborious calculations using g directly will only matter in the decimal places (estimate is 1.0189, using true g it is 1.0233). The more the function g departs from 1, the further up the decimal values will be affected. This is why "small noise" is required

-
adding to my answer, this probably explains why "non-parametric" regression, which consists of simple linear regression on "knots" of the data work so well - the tiny details of the noise matter very little over a "small" region about the regression line. –  probabilityislogic Mar 8 '11 at 13:48
+1 for the night club and demonstration that multiplicative noise can be approximated by additive. –  mpiktas Mar 8 '11 at 13:54
I like your answer very much. +1 –  Qiang Li Mar 8 '11 at 15:36