Hot answers tagged

19 votes

Principal Component Analysis Eliminate Noise In The Data

Principal Component Analysis (PCA) is used to a) denoise and to b) reduce dimensionality. It does not eliminate noise, but it can reduce noise. Basically an orthogonal linear transformation is used ...
user avatar
15 votes
Accepted

How is adding noise to training data equivalent to regularization?

Adding noise to the regressors in the training data is similar to regularization because it leads to similar results to shrinkage. The linear regression is an interesting example. Suppose $(Y_i,X_i)_{...
user avatar
14 votes

How is adding noise to training data equivalent to regularization?

Overview: For linear regression, I'll show that $\ell_2$ regularization (a.k.a. ridge regression) arises from minimizing the expected squared error over random perturbations of the regressors. The ...
user avatar
  • 30k
11 votes
Accepted

Classification with noisy labels?

The right thing to do here is to change the model, not the loss. Your goal is still to correctly classify as many data points as possible (which determines the loss), but your assumptions about the ...
user avatar
  • 5,812
11 votes

How to add noise to a random variable whose range is the unit interval?

A traditional way to handle constrained variables is to transform them into unconstrained variables, apply the jittering, and turn them back into the original scale. For instance, if $d_i\in(0,1)$, ...
user avatar
  • 93.4k
10 votes
Accepted

Probability distribution for a noisy sine wave

It depends on how the noise process is structured. Assuming that I've understood your situation correctly, if the noise is additive, independent and identically distributed, you would just take the ...
user avatar
  • 262k
8 votes

How to add noise to a random variable whose range is the unit interval?

It appears there are many ways to accomplish what you are looking for. Here's one suggestion. Treat each $d$ as if it were the value of some function $\Phi$ from the real number line to the unit ...
user avatar
  • 1,006
8 votes

What is the Fourier Transform of a brownian motion?

Sorry, I know this thread is old, but I feel like some statements are not very clear and/or misleading, and also I would like to add a more mathematically sound perspective on the matter. As was ...
user avatar
  • 267
7 votes
Accepted

How to detect noisy datasets (bias and variance trade-off)

When noise is "large" then learning is not pointless, but it's "expensive" in some sense. For instance, you know the expression "house always wins". It means that the odds favor the casino against the ...
user avatar
  • 56.8k
7 votes
Accepted

XGBoost when P>>N

Data is king, so if it works in real life we can't argue with it. Having said that, I agree with you it's bad practice and will usually not end well. I can design a dataset where it would work though. ...
user avatar
  • 559
6 votes

The algorithm behind pcl::StatisticalOutlierRemoval

Surprisingly the algorithm isn't very complicated and the explanation on the pcl website is actually almost all there is to it. There's however some subtleties that don't appear in the above link, so ...
user avatar
6 votes

How i add uniformly distributed noisy attributes to data set?

One way would be to train a model that learns the distribution of each feature separately; it could be a KDE for each feature. Then you could use this model to generate outliers for the data. I'd ...
user avatar
  • 346
5 votes
Accepted

Can white noise be (losslessly) compressed?

You may need to think carefully about definitions here. First question: when quantifying "compression", are you counting in the length of the decompression algorithm itself? If not, then the answer ...
user avatar
5 votes

Deal with noise data

Let me save you a lot of money. There is a lot of information that you do not have, but I will give you a solution to a similar problem. For starters, let us assume that the chart you have drawn was,...
user avatar
  • 7,142
5 votes
Accepted

Noise in regression data

My answer is simple and uses code. I hope someone will come and give you a better answer using equations and statistical language to explain it properly. Noise is variation in Y and X that's unrelated....
user avatar
  • 978
5 votes
Accepted

Generate uniform noise from a p-norm ball ($||x||_p \leq r$)

I found the full solution in a paper as suggested by kjetil b halvorsen (https://ieeexplore.ieee.org/stamp/stamp.jsp?arnumber=758215). I honestly have trouble understanding the math behind it, but the ...
user avatar
5 votes

Adding noise to non-negative imputed data

Note that the variation of the imputed unobserved values should be higher than of the really observed ones, because in addition to the variation the values would have even if you had measured them, ...
user avatar
  • 23.4k
4 votes

Fit exponential distribution with noise

In the absence of a response to my questions relating to the variation about the signal, I'll explain a little about nonlinear least squares. You can fit a model of the following form: $y_i = c + \...
user avatar
  • 262k
4 votes
Accepted

Comparison of distribution mean or median

The two experiments are mostly unpaired, so the answer applies to that situation. The information regarding outliers defined as >5 IQR is not given. Regarding outliers, when you have nonsense ...
user avatar
  • 11.9k
4 votes
Accepted

Correcting Kullback-Leibler divergence for size of datasets

The fundamental issue is that the KL divergence between the true underlying distributions is zero, as they are the same in your code ($U(0,1)$,) but sampling variation (almost) ensures that in finite ...
user avatar
  • 33.7k
4 votes
Accepted

Sum of Bernoulli random variables with Gaussian noise

Here I simply show the results of a sum of Bernoulli random variables where there is noise added to the probability parameter that follows a truncated Gaussian distribution, restricted to valid values ...
user avatar
  • 2,428
4 votes

Calibration of correlation

Given the intended application, you might be interested in creating realistic modifications of the series of data. Overview This is easier to do than you might think. (See the three-line function <...
user avatar
  • 293k
4 votes
Accepted

Are colored noises correlated / uncorrelated?

Question1: If the power spectrum is not flat, then does that mean the colored noises are correlated? One way to construct the power spectral density is to take the Fourier transform of the ...
user avatar
  • 126
4 votes
Accepted

Does Gaussian Noise tend to move the Pearson Correlation to zero?

Pearson correlation is, a bit loosely speaking (but not that loosely), measuring how well the points fit a diagonal line. When the points are far from the line, as the noisy setting will show, they ...
user avatar
  • 33.4k
4 votes

What might be the simplest (least flexible, least expressive) model to avoid over-fit?

Use ridge regression. The ridge parameter is a control on the complexity of the model class. Ridge regression is equivalent to fitting a regression model with a constraint on the norm of the weight ...
user avatar
3 votes
Accepted

Calculating the noise on data fitting an exponential decay

The usual measure of the size of noise about a nonlinear least squares regression fit is the standard error of the residuals, which is the square root of the variance estimate $$s^2=\hat{\sigma}^2=\...
user avatar
  • 262k
3 votes

Drawing independent Random Variables out of a Probability Distribution

Drawing a random variable $X$ with realisation $x$ is like picking a point $\omega$ in a certain space $\Omega$ (this can be formalised as being equivalent). On that space $\Omega$, a collection of ...
user avatar
  • 93.4k
3 votes
Accepted

I get a different accuracy each time I run a classifier on my data

If you split your sample in different ways, then your classifier will get different training and validation data in each case and therefore classify the test data in different ways. (Plus, it will get ...
user avatar
3 votes

What is the difference between "random noise" and "statistical noise"?

Noise (n) is the component of a signal (s) that is not information (i). $ s = i + n $ Any signal has one or more sources. Typically, each of these sources can theoretically be approximated by some ...
user avatar

Only top scored, non community-wiki answers of a minimum length are eligible