# Noise in regression data

How can I compare which distribution has more noise than the other. If for example I generate some data, how do I know that it has a large percentage of noise? Here I have a small sample code that generates such data:

set.seed(188)
n=100
p=60
x=matrix(rnorm(n*p, mean=1, sd=1), nrow=n, ncol=p)
beta=matrix(c(rep(1,20), rep(0,40)), nrow=p, ncol=1)
y=x%*%beta+rnorm(n=n, mean=0, sd=2)


How can I know if there is small or high noise in this data? If I change the standard derivation to 10 do I have then high or small noise? Is there any intuitive way to say if the data has high noise or low?

• @Hack-R I can change the example by adding more rows than columns, this is not very important. What I'm more interested to know is in relation to y when can I say that I have added a lot of noise ? When I add rnorm(n=n, mean=0, sd=2) or when I add : rnorm(n=n, mean=0, sd=21)? – Ville Jun 16 '18 at 17:17
• @Hack-R but I add the noise term $\epsilon =rnorm(n=n, mean=0, sd=2)$ to y not to x! Shouldn't there be a relation to y? – Ville Jun 16 '18 at 17:20

My answer is simple and uses code. I hope someone will come and give you a better answer using equations and statistical language to explain it properly.

Noise is variation in Y and X that's unrelated. If Y is perfectly explained by X then there's no noise. Introducing unobserved heterogeneity in Y or unrelated variation in X makes the fit of the model less than perfect, which means there's is noise.

set.seed(188)
n=100
p=60
x=matrix(rnorm(n*p, mean=1, sd=1), nrow=n, ncol=p)
beta=matrix(c(rep(1,20), rep(0,40)), nrow=p, ncol=1)
y=x%*%beta

# 100% of variation in Y is explained by X
summary(lm(y~x))

Residual standard error: 5.334e-15 on 39 degrees of freedom
Multiple R-squared:      1,   Adjusted R-squared:      1
F-statistic: 1.23e+30 on 60 and 39 DF,  p-value: < 2.2e-16

# Now we introduce noise, aka unexplained heterogeneity
y=x%*%beta+rnorm(n=n, mean=0, sd=2)
summary(lm(y~x))

# and there's a lower R-squared and Adjusted R-Squared

Residual standard error: 2.154 on 39 degrees of freedom
Multiple R-squared:  0.9228,  Adjusted R-squared:  0.8041
F-statistic: 7.772 on 60 and 39 DF,  p-value: 4.047e-10

• So this means if I have a low $R^2$ I have more noise in the data, right ? – Ville Jun 16 '18 at 17:28
• @Ville Correct. Where we define noise as random perturbations of the explanatory variables or unobserved heterogeneity in the outcome variable. – Hack-R Jun 16 '18 at 17:31
• Nice practical answer! I think this naturally brings up an interesting fundamental problem in statistical modeling of real-word phenomena - it's very hard to know if the accuracy of a model is limited by fundamental randomness, or by having captured insufficient inputs and using suboptimal modeling structure. – Eduard Gelman Jun 16 '18 at 19:02
• @Ville "So this means if I have a low $R^2$ I have more noise in the data, right?" Wrong. You can have variance in $Y$ that is (a) unexplained by $X$, because $Y$ is a completely deterministic function of some other variable $Z$ (or, for that matter, some nonlinear function of $X$, such a sinusoid), and (b) has Y includes no random variation at all. This would result in a low $R^{2}$, and yet this is not at all because $Y$ is noisy. – Alexis Jun 16 '18 at 20:06

I think, in machine learning it makes more sense to speak about model variance than about noise within variables. We see the effects of noise in the training data as a factor influencing model performance, but in most cases it is not necessary to quantify the noise. I will explain why:

Noise is an expression from physics and describes (in general) a perturbation with a broad and unspecific spectrum. There are many different forms of noise based on its origin or its statistical behavior. For example, white noise has a constant power density spectrum over a defined spectral area. In that sense you can simply measure noise, if you know what you are looking for. To quantify noise you need to know what you are comparing it to. A way to find your signal in the data.

An example: Let's say you have measurements of people heights in [m] and weights in [kg]. You want to build a model to predicts the [kg] from the [m] data. If you only have the raw measurements you don't know what is the signal and what is noise. You can still build a model. It will be influenced by the noise in both the [m] and the [kg] set. With this model you can calculate the training and test error to quantify the variance of you model. Though, that does not tell you much about the noise in each of the data sets. It tells you though how well the variables explain each other.

How to quantify the noise in this example? If you repeat the experiment 100 times within a time frame where you can assume that the peoples heights and weights did not change significantly (maybe within a day). You find that the measurements may still be different from each other (even for a single person) due to various factors such as peoples postures, inexact read-offs etc. Now, you plot a histogram and find that the variation follows a normal distribution. We assume that the true height is close to the mean of all measurements. Now, you know what is your signal e.g. 1.81 for a person that is 1.81m tall. Finally, you calculate the variance or standard error. Now, you quantified the noise height = 1.81 +/- 0.01 normalized by [m]. Reflecting an error of 1cm on average. Yes, a higher deviation means higher noise.

Effectively, you build a noise filter using the mean values. In some cases model performance increases significantly using such a filter. Though with enough training data, the model may learn the about the noise factors implicitly. But that is a different discussion.