Noise in regression data

Question

How can I compare which distribution has more noise than the other. If for example I generate some data, how do I know that it has a large percentage of noise? Here I have a small sample code that generates such data:

set.seed(188)
n=100
p=60
x=matrix(rnorm(n*p, mean=1, sd=1), nrow=n, ncol=p)
beta=matrix(c(rep(1,20), rep(0,40)), nrow=p, ncol=1)
y=x%*%beta+rnorm(n=n, mean=0, sd=2)

How can I know if there is small or high noise in this data? If I change the standard derivation to 10 do I have then high or small noise? Is there any intuitive way to say if the data has high noise or low?

Answer 1

My answer is simple and uses code. I hope someone will come and give you a better answer using equations and statistical language to explain it properly.

Noise is variation in Y and X that's unrelated. If Y is perfectly explained by X then there's no noise. Introducing unobserved heterogeneity in Y or unrelated variation in X makes the fit of the model less than perfect, which means there is noise.

set.seed(188)
n=100
p=60
x=matrix(rnorm(n*p, mean=1, sd=1), nrow=n, ncol=p)
beta=matrix(c(rep(1,20), rep(0,40)), nrow=p, ncol=1)
y=x%*%beta


# 100% of variation in Y is explained by X
summary(lm(y~x))

Residual standard error: 5.334e-15 on 39 degrees of freedom
Multiple R-squared:      1,   Adjusted R-squared:      1 
F-statistic: 1.23e+30 on 60 and 39 DF,  p-value: < 2.2e-16

# Now we introduce noise, aka unexplained heterogeneity
y=x%*%beta+rnorm(n=n, mean=0, sd=2)
summary(lm(y~x))

# and there's a lower R-squared and Adjusted R-Squared

Residual standard error: 2.154 on 39 degrees of freedom
Multiple R-squared:  0.9228,  Adjusted R-squared:  0.8041 
F-statistic: 7.772 on 60 and 39 DF,  p-value: 4.047e-10

Answer 2

I think, in machine learning it makes more sense to speak about model variance than about noise within variables. We see the effects of noise in the training data as a factor influencing model performance, but in most cases it is not necessary to quantify the noise. I will explain why:

Noise is an expression from physics and describes (in general) a perturbation with a broad and unspecific spectrum. There are many different forms of noise based on its origin or its statistical behavior. For example, white noise has a constant power density spectrum over a defined spectral area. In that sense you can simply measure noise, if you know what you are looking for. To quantify noise you need to know what you are comparing it to. A way to find your signal in the data.

An example: Let's say you have measurements of people heights in [m] and weights in [kg]. You want to build a model to predicts the [kg] from the [m] data. If you only have the raw measurements you don't know what is the signal and what is noise. You can still build a model. It will be influenced by the noise in both the [m] and the [kg] set. With this model you can calculate the training and test error to quantify the variance of you model. Though, that does not tell you much about the noise in each of the data sets. It tells you though how well the variables explain each other.

How to quantify the noise in this example? If you repeat the experiment 100 times within a time frame where you can assume that the peoples heights and weights did not change significantly (maybe within a day). You find that the measurements may still be different from each other (even for a single person) due to various factors such as peoples postures, inexact read-offs etc. Now, you plot a histogram and find that the variation follows a normal distribution. We assume that the true height is close to the mean of all measurements. Now, you know what is your signal e.g. 1.81 for a person that is 1.81m tall. Finally, you calculate the variance or standard error. Now, you quantified the noise height = 1.81 +/- 0.01 normalized by [m]. Reflecting an error of 1cm on average. Yes, a higher deviation means higher noise.

Effectively, you build a noise filter using the mean values. In some cases model performance increases significantly using such a filter. Though with enough training data, the model may learn the about the noise factors implicitly. But that is a different discussion.