Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [noise]

noise is a term used for the error term in statistical models and in signal processing. It could be white noise, colored noise or otherwise.

3
votes
0answers
45 views

Get noise model from true and distorted data

I have been given two data sets: Set A: a small data set containing data randomly drawn form an underlying distribution Set B: a very large data set containing data randomly drawn from the same ...
3
votes
0answers
115 views

Correcting for noise in gene expression data

I have a training set of RT-qPCR gene expression data (not run in triplicate) for a batch of samples with two phenotypes $A$ and $B$ on which I've trained a logistic regression classifier. I also ...
3
votes
0answers
76 views

Expressing Confidence in Conclusions from Noisy Data

I’m working on improving the robustness of a software engineering process that measures performance of programming language compiler and standard library, a.k.a. benchmarks — in the computing sense of ...
3
votes
0answers
72 views

What is known about 2nd order estimation biases due to correlated noise between response variables?

I have recently run into a statistical bias in a type of analysis that I believe is somewhat common in my field, and not typically corrected for. I would like to know if this is more well-known in ...
3
votes
0answers
48 views

Representation of Noise in Fourier transform

I perform an experiment where I sample $M(k)$ which is in theory related to $|f(x)|^2$ via $M(k)=\int e^{ikx}|f(x)|^2\,dx$. I perform the discrete FT on my data in order to obtain $|f(x)|^2$. Without ...
3
votes
0answers
75 views

Who says trading data are noisy?

We try to denoise our time-series and model inputs with a plethora of methods like Kalman filters, EMA, Kernel filters, Splines, Beziers, etc. But who came up with a theory that trading data is noisy ...
3
votes
0answers
162 views

Detection of noise and outliers

I am measuring the number of cells with a mutation in a series of 106 subjects. For each position of the genome, the method will output the total number of cells analysed and the number of cells with ...
3
votes
0answers
246 views

How to determine rise time of a signal from its noisy background timeseries?

I have temperature vs. time data from a thermometer. The data was recorded using a DAQ system, has a stable background level, and some random noise. At a certain time, the temperature begins to rise ...
2
votes
0answers
69 views

What is the difference between Noise, error and residuals?

I was reading about Kalman filter. http://web.mit.edu/kirtley/kirtley/binlustuff/literature/control/Kalman%20filter.pdf They talk about additive noise and error. I need to understand difference ...
2
votes
0answers
214 views

Awful performance of LSTM on noisy time series after stationarisation

Note. The post is quite long because I added some thought process for the sake of seeing the big picture. So grab a coffee and indulge yourself. For tldr the actual question on the bottom. I put my ...
2
votes
0answers
27 views

Noisy conditional simulation

A conditional random field $Z_C(x)$ is a random field whose realisations $z_C(x)$ always take the same values $z_C(x_a)$ at locations $x_a$. Realisations of $Z_C(x)$ can be produced as follows (...
2
votes
0answers
75 views

How to properly treat feature (attribute) data from multiple sensors with different measurement noise (for classification)?

I have a classification problem in which the input feature data are derived from multiple sensors. If the quality of the feature attributes as measured by each sensor varies (for example, because some ...
2
votes
0answers
67 views

Noise estimation in LTE using bandpass filter

Can noise estimation in LTE be done using bandpass filter? As per my study in wireless systems to estimate noise power, if pilot sequence is known is done as |y(k)-p(k)h(k)|^2, where p(k) is pilot ...
2
votes
0answers
74 views

Cleaning the signal from noise

I have a following signal, and I want to correct the part which has a sharp jump, based on the value of the signal sometimes before the jump and its future values. Instead of that jump I want ...
2
votes
0answers
976 views

Deriving mean and variance of the posterior distribution

I have a simple linear model: $y_{i}=\mu+e_{i}$ for $i=1,...,n$, where $P(e_{i})=w\mathcal{N}(0,\sigma^2) + (1-w)\mathcal{N}(0,k^2\sigma^2)$ with $w=0.9$, $k=10$ and $\sigma=0.1$. It can be understood ...
2
votes
0answers
20 views

Finding the spectral density of a fraction of two noisy variables

I have a certain expression $\Phi=C \frac{V}{R}$ where $C$ is a constant, and both $V$ and $R$ are noisy variables; they are set to some working point value, but they have some small noise component (...
2
votes
0answers
598 views

Dealing with high noise data?

I am working with sensors on trucks which give me data with a lot of noise. The goal is to make a prediction to prevent failure in the system. I am having trouble with this for several reasons: The ...
2
votes
0answers
51 views

For each x, I observe A and know P(C). What can I say about E(A|C)?

For each subject x in a population, I observe x's age, A(x). I can calculate the probability that x has some property of interest c, $P[C(x) = 1]$, where C(x) is a binary variable indicating whether ...
2
votes
0answers
294 views

R: Is it possible to estimate the poisson noise?

I have a dataset of many discrete counts (RNAseq read counts per base), which contain both real signals and background noise. The noise is random, and should be poisson distributed. What I would ...
1
vote
0answers
36 views

Techniques to apply Discrete Wavelet Transform (DWT) to denoise and predict time series

I just started playing with wavelets and have been using this library (https://github.com/rafat/wavelib) to further my understanding and see if 'denoising' the series at all possible levels is ...
1
vote
0answers
157 views

Adding noise to time series data to increase training data

I am dealing with a weekly time series forecasting problem and I am currently investigating the use of an LSTM to make a multi-step forecast for a univariate time series. I actually have a ...
1
vote
0answers
25 views

Using a neural network for a regression problem, where the model to be learned suffers from awgn

I have currently a neural network to learn a (relatively non complex) system model (vector regression). Its problem is that the outputs of the system suffer from arbitry additional white gaussian ...
1
vote
0answers
31 views

Sum of correlated non-Gaussian random variables with same distribution as individual terms

Let's say there is a sum $s$ of $N$ zero-mean correlated random variables $\{x_i\}$: $$ s = x_1 + x_2 + \ldots +x_N, $$ where the correlations $C_{ij} = \mathbb E[ x_i x_j]$ are known. Assume that ...
1
vote
0answers
43 views

How do you use the predictive distribution with noise in Bayesian Optimization?

I have been reading a paper on Bayesian Optimization, and I was reading the section on adding Gaussian noise to your Gaussian process. The article is: Brochu, Cora and de Freitas (2010). A Tutorial ...
1
vote
0answers
24 views

MIxture model in R to generate noise in data

I have a bit of code in R that adds noise to an harmonic series according to a normal distribution: ...
1
vote
0answers
32 views

How does Noise affect the results of Transfer Entropy?

I was reading about Transfer Entropy and came across this package: https://cran.r-project.org/web/packages/TransferEntropy/TransferEntropy.pdf The code in the package: ...
1
vote
0answers
38 views

How to add noise to obfuscate patterns in data

I have a program that generates output data depending on the inputs it is given. Lets say the data generated is a list of n items where each item is a natural number between 1 and k. I need to release ...
1
vote
0answers
33 views

Time series clustering/segmentation based on pattern

I'm currently working with a database which contains several large PPG (pulse oximetry) and ECG time series. These series, however, contain segments within them which are highly contaminated by noise, ...
1
vote
0answers
28 views

Evaluation of Jacobian for Extended Kalman Filter

For the non-additive noise case, \begin{equation} x_k = f(x_{k-1}, u_{k-1}, \xi_{k-1}) \\ y_k = h(x_k, \nu_k) \end{equation} the EKF takes into account the jacobian wrt to the noise terms $ L_{k-1} =...
1
vote
0answers
108 views

Estimating signal-to-noise ratio after smoothing data and covariance estimates

I have data for two variables, let's say $A$ and $B$, which have distributions that are roughly centered at 0. I am computing a value, let's say $C = \sqrt{A^2 + B^2}$, from this data. I also have a ...
1
vote
0answers
83 views

How to decompose the bias-variance-noise?

Suppose I have a dataset, and I want to do a bias-variance-noise decomposition of a particular learning algorithm? What is the practical approach to do this? I think I understand how to calculate bias ...
1
vote
0answers
13 views

Intensity deconvolution

We have a set of intensities (measured) $I_{j} = \cos(\theta_{j}) + N_j$ where $\theta_{j}$ is distributed according to some distribution between 0 and 180 degrees (well, in reality between 0 and ...
1
vote
0answers
16 views

How to assess the true significance of predictors in the face of collinearity and noisy features?

In most practical cases, models are burdened with collinear and noisy features. while regularization helps avoid overfitting by effectively simplifying the model, the question of interpretation of ...
1
vote
0answers
51 views

How to separate two classes when the features values predicting them are so similar ?

What should be my approach. I got 13 principal components from 21 numerical features. The 13 features have a gaussian distribution. The plot below is between the top two components. Should I clean the ...
1
vote
0answers
21 views

Finding the noise spectral density of $\frac{a+\delta a(t)}{b + \delta b(t)}$

I have two quantities $a(t)$ and $b(t)$ that have a constant mean ($a$ and $b$) and some small fluctuating noise part with vanishing mean $\delta a(t)$ and $\delta b(t)$. I'll write them as $a(t) = a +...
1
vote
0answers
24 views

Where can I find good references regarding to noise filtering and prediction in time series?

I want to model the error structure of every certain time period obtained from the past errors produced by the predictions of nonlinear time series. I would like to know if someone knows specialized ...
1
vote
0answers
142 views

Non-Causal time-series filtering techniques for standard noise with unkown variance. (EM vs. weiner vs. kalman)

This is a quick question about filtering stored time-series data using kalman/weiner filtering techniques or expectation maximization. I'm just hoping to fix some confusion about questioning ...
1
vote
0answers
138 views

Lagrange Multipliers in practice

Say we want to minimize the function $f^2({\bf{x}})$, under the constraint $g({\bf{x}})=0$. The classic solution (Method I) is to introduce a Lagrange Multiplier, and solve: $$\frac{\partial f^2({\bf{...
1
vote
0answers
46 views

Signal processing techniques for unevenly spaced and repeated measures series

I am considering using signal processing techniques to find the minimum on a noisy 1D response line. More specifically I have a simulation that requires one parameter, but also includes randomness, ...
0
votes
0answers
5 views

power spectrum density (V**2/Hz) for nonuniform logarithmic array?

I have nonuniform logarithmic time data array which have 1000pts/decade. that means sampling rate is changing in each intervals or decade. How can I calculate and plot power spectrum density (V**2/Hz)...
0
votes
0answers
9 views

RNN outputs noisy predictions

I have an RNN that I've trained and I'm now using to generate new sequences. These sequences are basically discrete state time courses for K different states. The ...
0
votes
0answers
14 views

What is the effect of noisy labels in distant-supervision?

I am just learning about distant supervision. I read the paper of Mintz et al. and trying to get some intuition of how the noise influences the classification. My general assumption is, that having ...
0
votes
0answers
17 views

Bayesian vs Frequentist inference in the presence of noisy data

I'm wondering how Bayesian inference and Classical/Frequentist inference fair towards noisy data. I can't seem to find too much literature addressing this issue and it seems the conclusion is usually ...
0
votes
0answers
17 views

How to perform feature scaling on noise removel process?

i'm working on dataset contain machinery sensor data. each column(feature) represent different sensor data(pressure, temperature, speed, etc) of the machine part. here task is to predict normal ...
0
votes
0answers
32 views

Noise reduction with known noise distribution

I have a time signal with a known noise distribution parameters (gaussian, sd is known). I would like to estimate the true value statistically and in the best case obtain a confidence interval. As I ...
0
votes
0answers
29 views

Prediction of noisy target variable

The target variable for my regression problem has a very high noise, so measurement error is very high and trends can only be seen in longer time periods. What are good approaches to address this ...
0
votes
0answers
27 views

How sensitive are Neural Networks to weight changes?

Let's consider the space of feedforward neural networks with a given structure: $L$ layers, $m$ neurones per layer, ReLu activation, input dimension $d$, output dimension $k$. Which means I'm ...
0
votes
0answers
14 views

What determines whether relative or absolute Gaussian noise should be added to data?

Suppose I have some data vector, $\mathbf{d}$ with length $N$. I want to add "Gaussian noise" to the data. My understanding is that there are two ways to do this. 1) Relative noise percentage $$\...
0
votes
0answers
486 views

“Add White Gaussian Noise with SNR” vs. “Add 5% Gaussian Noise”

I have a noise-free dataset which is a vector of numbers, $\mathbf{d}$, with length $N$. I want to "add noise" to this data. My understanding is that there are two ways to do this. 1) Add some ...
0
votes
0answers
26 views

Optimization of a noisy function with binary output

What frameworks exist to optimize the following problem: $\theta^\ast = \operatorname*{argmin}\limits_{\theta \in \Theta} f(\theta)$ with the characteristic that $f$ is noisy and has a binary output ...