19
votes
Principal Component Analysis Eliminate Noise In The Data
Principal Component Analysis (PCA) is used to a) denoise and to b) reduce dimensionality.
It does not eliminate noise, but it can reduce noise.
Basically an orthogonal linear transformation is used ...
- 3,221
15
votes
Accepted
How is adding noise to training data equivalent to regularization?
Adding noise to the regressors in the training data is similar to regularization because it leads to similar results to shrinkage.
The linear regression is an interesting example. Suppose $(Y_i,X_i)_{...
- 645
14
votes
How is adding noise to training data equivalent to regularization?
Overview: For linear regression, I'll show that $\ell_2$ regularization (a.k.a. ridge regression) arises from minimizing the expected squared error over random perturbations of the regressors. The ...
- 30k
11
votes
Accepted
Classification with noisy labels?
The right thing to do here is to change the model, not the loss. Your goal is still to correctly classify as many data points as possible (which determines the loss), but your assumptions about the ...
- 5,812
11
votes
How to add noise to a random variable whose range is the unit interval?
A traditional way to handle constrained variables is to transform them into unconstrained variables, apply the jittering, and turn them back into the original scale.
For instance, if $d_i\in(0,1)$, ...
- 93.4k
10
votes
Accepted
Probability distribution for a noisy sine wave
It depends on how the noise process is structured.
Assuming that I've understood your situation correctly, if the noise is additive, independent and identically distributed, you would just take the ...
- 262k
8
votes
How to add noise to a random variable whose range is the unit interval?
It appears there are many ways to accomplish what you are looking for. Here's one suggestion.
Treat each $d$ as if it were the value of some function $\Phi$ from the real number line to the unit ...
- 1,006
8
votes
What is the Fourier Transform of a brownian motion?
Sorry, I know this thread is old, but I feel like some statements are not very clear and/or misleading, and also I would like to add a more mathematically sound perspective on the matter.
As was ...
- 267
7
votes
Accepted
How to detect noisy datasets (bias and variance trade-off)
When noise is "large" then learning is not pointless, but it's "expensive" in some sense. For instance, you know the expression "house always wins". It means that the odds favor the casino against the ...
- 56.8k
7
votes
Accepted
XGBoost when P>>N
Data is king, so if it works in real life we can't argue with it.
Having said that, I agree with you it's bad practice and will usually not end well.
I can design a dataset where it would work though. ...
- 559
6
votes
The algorithm behind pcl::StatisticalOutlierRemoval
Surprisingly the algorithm isn't very complicated and the explanation on the pcl website is actually almost all there is to it.
There's however some subtleties that don't appear in the above link, so ...
- 161
6
votes
How i add uniformly distributed noisy attributes to data set?
One way would be to train a model that learns the distribution of each feature separately; it could be a KDE for each feature.
Then you could use this model to generate outliers for the data. I'd ...
- 346
5
votes
Accepted
Can white noise be (losslessly) compressed?
You may need to think carefully about definitions here.
First question: when quantifying "compression", are you counting in the length of the decompression algorithm itself?
If not, then the answer ...
- 590
5
votes
Deal with noise data
Let me save you a lot of money. There is a lot of information that you do not have, but I will give you a solution to a similar problem.
For starters, let us assume that the chart you have drawn was,...
- 7,142
5
votes
Accepted
Noise in regression data
My answer is simple and uses code. I hope someone will come and give you a better answer using equations and statistical language to explain it properly.
Noise is variation in Y and X that's unrelated....
- 978
5
votes
Accepted
Generate uniform noise from a p-norm ball ($||x||_p \leq r$)
I found the full solution in a paper as suggested by kjetil b halvorsen (https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=758215). I honestly have trouble understanding the math behind it, but the ...
- 321
5
votes
Adding noise to non-negative imputed data
Note that the variation of the imputed unobserved values should be higher than of the really observed ones, because in addition to the variation the values would have even if you had measured them, ...
- 23.4k
4
votes
Fit exponential distribution with noise
In the absence of a response to my questions relating to the variation about the signal, I'll explain a little about nonlinear least squares.
You can fit a model of the following form:
$y_i = c + \...
- 262k
4
votes
Accepted
Comparison of distribution mean or median
The two experiments are mostly unpaired, so the answer applies to that situation. The information regarding outliers defined as >5 IQR is not given. Regarding outliers, when you have nonsense ...
- 11.9k
4
votes
Accepted
Correcting Kullback-Leibler divergence for size of datasets
The fundamental issue is that the KL divergence between the true underlying distributions is zero, as they are the same in your code ($U(0,1)$,) but sampling variation (almost) ensures that in finite ...
- 33.7k
4
votes
Accepted
Sum of Bernoulli random variables with Gaussian noise
Here I simply show the results of a sum of Bernoulli random variables where there is noise added to the probability parameter that follows a truncated Gaussian distribution, restricted to valid values ...
- 2,428
4
votes
Calibration of correlation
Given the intended application, you might be interested in creating realistic modifications of the series of data.
Overview
This is easier to do than you might think. (See the three-line function <...
- 293k
4
votes
Accepted
Are colored noises correlated / uncorrelated?
Question1: If the power spectrum is not flat, then does that mean the
colored noises are correlated?
One way to construct the power spectral density is to take the Fourier transform of the ...
- 126
4
votes
Accepted
Does Gaussian Noise tend to move the Pearson Correlation to zero?
Pearson correlation is, a bit loosely speaking (but not that loosely), measuring how well the points fit a diagonal line. When the points are far from the line, as the noisy setting will show, they ...
- 33.4k
4
votes
What might be the simplest (least flexible, least expressive) model to avoid over-fit?
Use ridge regression. The ridge parameter is a control on the complexity of the model class. Ridge regression is equivalent to fitting a regression model with a constraint on the norm of the weight ...
- 49.3k
3
votes
Accepted
Calculating the noise on data fitting an exponential decay
The usual measure of the size of noise about a nonlinear least squares regression fit is the standard error of the residuals, which is the square root of the variance estimate $$s^2=\hat{\sigma}^2=\...
- 262k
3
votes
Drawing independent Random Variables out of a Probability Distribution
Drawing a random variable $X$ with realisation $x$ is like picking a point $\omega$ in a certain space $\Omega$ (this can be formalised as being equivalent). On that space $\Omega$, a collection of ...
- 93.4k
3
votes
Accepted
I get a different accuracy each time I run a classifier on my data
If you split your sample in different ways, then your classifier will get different training and validation data in each case and therefore classify the test data in different ways. (Plus, it will get ...
- 99.5k
3
votes
What is the difference between "random noise" and "statistical noise"?
Noise (n) is the component of a signal (s) that is not information (i).
$ s = i + n $
Any signal has one or more sources. Typically, each of these sources can theoretically be approximated by some ...
- 529
Only top scored, non community-wiki answers of a minimum length are eligible