Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [noise]

noise is a term used for the error term in statistical models and in signal processing. It could be white noise, colored noise or otherwise.

11
votes
2answers
6k views

Probability distribution for a noisy sine wave

I'm looking to analytically calculate a probability distribution of sampling points from an oscillating function when there is some measurement error. I have already calculated the probability ...
10
votes
1answer
7k views

Classification with noisy labels?

I'm trying to train a neural network for classification, but the labels I have are rather noisy (around 30% of the labels are wrong). The cross-entropy loss indeed works, but I was wondering are ...
9
votes
2answers
657 views

Generate uniform noise from a p-norm ball ($||x||_p \leq r$)

I am trying to write a function which generates uniformly distributed noise which comes from a p-norm ball of $n$ dimensions: \begin{equation} ||x||_p \leq r \end{equation} I found possible ...
7
votes
1answer
6k views

Principal Component Analysis Eliminate Noise In The Data

Do Principal Component Analysis (PCA) eliminate noise in the data set? If PCA do not eliminate noise in the data set, what actually does PCA do to the data set? Can somebody help me regarding this ...
6
votes
2answers
934 views

Fit exponential distribution with noise

I'm trying to fit an exponential with noise (which in this case is a constant $c$) like this one $$ y(x) = \alpha e^{- \alpha x} + c \text{ ,}$$ having $(x_i, y_i)$ values (So $\alpha$ and $c$ are ...
4
votes
2answers
2k views

How to add noise to a random variable whose range is the unit interval? [closed]

I have a list of values sampled from a beta distribution that therefore lie in the interval [0,1]. I would like to add (e.g. Gaussian) noise to these values, but of course there is the problem of the ...
4
votes
2answers
191 views

Estimate the variance of Gaussian distribution from noisy sample

I have measured a large data sample from an underlying Gaussian distribution and want to estimate the variance and its error. However, the measured values are noisy with some Gaussian noise with a ...
4
votes
1answer
861 views

How to detect noisy datasets (bias and variance trade-off)

Studying the bias-variance trade-off: expected loss = bias + variance + noise I understand that we minimize this quantity by finding the "best" balance between ...
4
votes
2answers
2k views

Noise in regression data

How can I compare which distribution has more noise than the other. If for example I generate some data, how do I know that it has a large percentage of noise? Here I have a small sample code that ...
4
votes
1answer
773 views

Sparse representations for denoising problems

I have read in a huge number of papers that sparse models (sparse coding, dictionary learning, sparse matrix factorization, ...) are good solutions for image denoising problems. I know that ...
4
votes
1answer
170 views

Why noisy data will benefit Bayesian?

Recently I am reading a paper in 2001, Michael D. Ernst, Jake Cockrell, William G. Griswold, David Notkin Dynamically Discovering Likely Program Invariants to Support Program Evolution TSE 2001, in ...
4
votes
1answer
543 views

High noise high non stationary time series forecasting

I am trying to make a prediction on a time series that look like this (the trend, seasonality and residuals are extracted with a frequency of 12 months: It is a time series of approx 3500 values (10 ...
3
votes
2answers
386 views
3
votes
1answer
5k views

Comparison of distribution mean or median

I am working with very noisy biological data for which I will compare two experimental settings. For each setting I will get a set of measure with a huge variance, sometimes with a skewed distribution,...
3
votes
2answers
144 views

Regression algorithm on [0,1] with lots of mislabeled data

I have a training set mapping some Likert-scale variables (integers between 1 and 7, rescaled to real numbers between 0 and 1) to predict a continuous variable between 0 and 1. The data set is ...
3
votes
1answer
1k views

Calculating the noise on data fitting an exponential decay

I'm trying to calculate the amount of noise in data that fits to an exponential decay function. I'm trying to calculate signal-to-noise at different times of the data. Here is the code for how I ...
3
votes
1answer
413 views

How can I measure noise-reduction after applying an autoencoder?

In my setting I have 21 numerical features (and ten thousands of rows) and I assume that they are noisy measurements of the real features. In order to learn the real features I think about applying ...
3
votes
1answer
585 views

Training on a noisy dataset

I have a dataset of about 300,000 images, of which about 1000 are labeled as containing the salient feature. Unfortunately the labeling is conservative: while almost all (99%) of those labeled have ...
3
votes
2answers
100 views

ARMA models and residual series

Assuming that model is correct, why does the residual series of an ARMA model resemble a white noise process?
3
votes
2answers
29 views

Formula to detect non uniformity noise

I have this line profile, how is the best way to detect this kind of non uniformity? I must detect a sudden change, as opposed to the last section that decreases slowly.. But the sudden difference ...
3
votes
0answers
45 views

Get noise model from true and distorted data

I have been given two data sets: Set A: a small data set containing data randomly drawn form an underlying distribution Set B: a very large data set containing data randomly drawn from the same ...
3
votes
0answers
115 views

Correcting for noise in gene expression data

I have a training set of RT-qPCR gene expression data (not run in triplicate) for a batch of samples with two phenotypes $A$ and $B$ on which I've trained a logistic regression classifier. I also ...
3
votes
0answers
76 views

Expressing Confidence in Conclusions from Noisy Data

I’m working on improving the robustness of a software engineering process that measures performance of programming language compiler and standard library, a.k.a. benchmarks — in the computing sense of ...
3
votes
0answers
72 views

What is known about 2nd order estimation biases due to correlated noise between response variables?

I have recently run into a statistical bias in a type of analysis that I believe is somewhat common in my field, and not typically corrected for. I would like to know if this is more well-known in ...
3
votes
0answers
48 views

Representation of Noise in Fourier transform

I perform an experiment where I sample $M(k)$ which is in theory related to $|f(x)|^2$ via $M(k)=\int e^{ikx}|f(x)|^2\,dx$. I perform the discrete FT on my data in order to obtain $|f(x)|^2$. Without ...
3
votes
0answers
75 views

Who says trading data are noisy?

We try to denoise our time-series and model inputs with a plethora of methods like Kalman filters, EMA, Kernel filters, Splines, Beziers, etc. But who came up with a theory that trading data is noisy ...
3
votes
1answer
37 views

Regression with review scores containing excess zeroes

Here is an imaginary problem, representing something I am dealing with right now. We have a set of movies with averages scores ranging from 0 to 10, such that the target is a continuous variable. By ...
3
votes
0answers
163 views

Detection of noise and outliers

I am measuring the number of cells with a mutation in a series of 106 subjects. For each position of the genome, the method will output the total number of cells analysed and the number of cells with ...
3
votes
1answer
29 views

How to simulate noisy respondent entry of short textual responses? [closed]

I have a list of company name as following: ...
3
votes
0answers
246 views

How to determine rise time of a signal from its noisy background timeseries?

I have temperature vs. time data from a thermometer. The data was recorded using a DAQ system, has a stable background level, and some random noise. At a certain time, the temperature begins to rise ...
2
votes
3answers
2k views

What is the difference between “random noise” and “statistical noise”?

According to English wikipedia, there are Statistical noise and Random noise (e.g., white noise). However, I've never seen definitions of what statistical noise is. So, what is the difference if any? ...
2
votes
1answer
144 views

Sum of Bernoulli random variables with Gaussian noise

This relates to a question asked recently where (one of the edits of) the question asked what happens when a sum of Bernoulli random variables has some form of noise on the probability parameter. ...
2
votes
1answer
76 views

Ridge Regression as Robust Optimization

We were told to assume in class that the below optimization formulations are equivalent- $$\min_w\max_{\delta:||\delta||_F\leq\epsilon}||(X+\delta)w-y||_2^2$$ $$\min_{w}||Xw-y||_2^2+\lambda||w||_2^2 ...
2
votes
1answer
430 views

Arbitrary noise generation

lets say I have some noise time series and I want to sample random numbers from the same underlying distribution as this time series. An easy algorithm I have found online is basically you take the ...
2
votes
1answer
1k views

What is uncorrelated noise and its significance

In many applications such as estimation theory, when we need to estimate a parameter then we usually consider in presence of white gaussian noise of zero mean and some standard deviation. During ...
2
votes
1answer
34 views

Apply 3 sigma formula in gamma distribution?

Let say i have some data that follows gamma distribution, and i calculated the Mean and Standard deviation of the gamma distribution. I also know that there are some outliers(Noise) in the data i ...
2
votes
2answers
53 views

What is the probability that sample variance decreases by adding random Gaussian noise to the variable?

If we assume WLOG that our variable X has mean zero (mean-centered), then this can be stated $Pr \bigg(\sum x^2 > \sum (x-n)^2 \bigg)$ for some random variable $n$ distributed under $N \sim N(0, \...
2
votes
1answer
75 views

Why does “sticky noise” defy averaging attack?

I have read an interesting paper (pdf) describing how a privacy preserving technique might be breached, but I am having trouble understanding the following paragraph describing one of several layers ...
2
votes
2answers
135 views

Why must Gaussian random noise have 0 Covariance?

In Linear Regression, when we model $Y$'s dependence on input $X$, we say that $$Y = X\beta +\epsilon $$ where $\epsilon \sim \mathcal{N}(0, \sigma ^2 I)$ where $I$ is the identity matrix. I ...
2
votes
1answer
173 views

Estimate gaussian background noise

Let's say I have 3 Random Variables $X_1, X_2, X_b$ where "$b$" stands for "background". Each one of them is Gaussian with $N(\mu_i, \sigma^2_i)$ for $ i\in\{1,2,b\}$. I will assume $\mu_b=0$. Now I ...
2
votes
1answer
45 views

5 low signal-to-noise measurements of same signal, know noise distribution and signal distribution. Recover?

I have 5 measurements of $x(t)$ at each instance in time, all contaminated with Gaussian noise. My signal-to-noise ratio is bad. \begin{align} y_1(t)=x(t)+n_1(t)\\ y_2(t)=x(t)+n_2(t)\\ y_3(t)=x(t)+...
2
votes
1answer
116 views

Learn a distribution from distributions on samples [closed]

There's many good ways to learn a distribution $p_X$ of an r.v. $X$ over $k$ symbols given many i.i.d. samples $X_1,\ldots, X_n$. The simplest is to use the sample relative frequencies $\hat{f}_X$ as ...
2
votes
1answer
2k views

Bayesian Weighted Linear regression

I am currently reading the following paper which formulates the weighted linear regression in a Bayesian setting. In classic weighted LS, we minimise the following: $$ \sum_{i=1}^{N} w_i (\beta^Tx_i -...
2
votes
0answers
69 views

What is the difference between Noise, error and residuals?

I was reading about Kalman filter. http://web.mit.edu/kirtley/kirtley/binlustuff/literature/control/Kalman%20filter.pdf They talk about additive noise and error. I need to understand difference ...
2
votes
0answers
215 views

Awful performance of LSTM on noisy time series after stationarisation

Note. The post is quite long because I added some thought process for the sake of seeing the big picture. So grab a coffee and indulge yourself. For tldr the actual question on the bottom. I put my ...
2
votes
0answers
27 views

Noisy conditional simulation

A conditional random field $Z_C(x)$ is a random field whose realisations $z_C(x)$ always take the same values $z_C(x_a)$ at locations $x_a$. Realisations of $Z_C(x)$ can be produced as follows (...
2
votes
1answer
59 views

Noisy observations of independent signals are independent?

Assume 10 signals are transmitted each day derived from the same distribution randomly, and as such are considered independent from each other. Each signal is measured by 10 different monitors each ...
2
votes
0answers
75 views

How to properly treat feature (attribute) data from multiple sensors with different measurement noise (for classification)?

I have a classification problem in which the input feature data are derived from multiple sensors. If the quality of the feature attributes as measured by each sensor varies (for example, because some ...
2
votes
0answers
67 views

Noise estimation in LTE using bandpass filter

Can noise estimation in LTE be done using bandpass filter? As per my study in wireless systems to estimate noise power, if pilot sequence is known is done as |y(k)-p(k)h(k)|^2, where p(k) is pilot ...
2
votes
0answers
74 views

Cleaning the signal from noise

I have a following signal, and I want to correct the part which has a sharp jump, based on the value of the signal sometimes before the jump and its future values. Instead of that jump I want ...