Questions tagged [noise]
noise is a term used for the error term in statistical models and in signal processing. It could be white noise, colored noise or otherwise.
236
questions
4
votes
2answers
94 views
Does a targets-permutation test prove that regression find a real pattern?
I need to solve a standard ("vanilla") regression problem meaning that I have a 2D array of real-valued features (X) and 1D array of real-valued targets (<...
1
vote
4answers
141 views
What might be the simplest (least flexible, least expressive) model to avoid over-fit?
I need to perform a regression on a data set with a huge noise-to-signal ratio. I am not even sure if there is any "signal" in the data (maybe there is only "noise" in the data), ...
0
votes
0answers
23 views
How to categorise data bumps/valleys into actual features or noise?
Here's some data$^1$ denoting the variation of the mean of samples (y-axis) with the no. of samples (x-axis). The uncertainties are $1\sigma$ standard deviation of those samples.
Fig1. Graph of the ...
0
votes
0answers
17 views
Correlation in the presence of noise
I have what I hope is a simple problem. I have two time series over which I wish to calculate a moving correlation with a fixed window of length 50. Visually the two series are closely correlated, ...
0
votes
0answers
87 views
Wick's theorem - Find the right expression for shot noise on each $a_{\ell m}^{2}$ in spherical harmonics
In an astrophysics context, if I take as definition of $a_{l m}$ following a normal distribution with mean equal to zero and $C_{\ell}=\dfrac{1}{2\ell+1}\sum_{m=-\ell}^{\ell}a_{l m}^{2}=\left\langle ...
6
votes
0answers
78 views
What is the difference between a non-zero nugget and a noise term in Kriging/GPR?
With some Gaussian Process Regression/Kriging models, it's possible to specify both a non-zero nugget, and a noise term. For example, in Scikit-learn's GPR model, there is an ...
2
votes
1answer
66 views
HDBSCAN: most data clustered as noise (-1)
I am trying to perform topic modeling on text data, ie. cluster the text messages by topic. I am approaching this by using a BERT model to get sentence embeddings, then use T-sne to reduce the ...
1
vote
0answers
19 views
Mutual Information in the presence of noise
Let Y=X+N, where N is the noise distribution.
a. A user can observe only Y. (I(X; Y)=?)
b. A user can estimate the mean of N and he can observe Y only. (I(X;Y|mean(N))) He has no information regarding ...
0
votes
0answers
17 views
How to calculate Maximum Likelihood Estimator
I have samples of a noisy real vector with constant phase y¯(2) = a¯ . e^jθ + w
where θ is a real scalar, and the entries of w¯ are complex normal i.i.d, where
W i,real, W i,image ∼ N(0,σ^2) for i = 1,...
1
vote
1answer
31 views
Zero mutual information
I have two random variables X and Y=X+N. What should be the condition such that I(X; Y)=0? Can anyone direct me towards the relevant references?
16
votes
3answers
2k views
How is adding noise to training data equivalent to regularization?
I've noticed that some people argue that adding noise to training data equivalent to regularizing our predictor parameters. How is this the case?
Some of the examples listed on SE discussing this ...
0
votes
1answer
303 views
How to add and vary Gaussian noise to input data
I have a time-series data and I would like to add an additive Gaussian Noise to the input of the data. What I am trying to do is that I want to test my ML predictive model against different level of ...
0
votes
1answer
16 views
Rates of convergence with asymptotically negligibly noisy observations
Apologies in advance if this question is not completely well defined. Suppose that I am estimating a nonparametric model for a conditional expectation function $\mathbb E[Y_i | X_i]$ using some i.i.d. ...
1
vote
0answers
23 views
repeated measurements needed for certainty [closed]
A bit of a silly question.
We have a model, which can be described as y = M(x) + some significant levels of noise.
We see that if we repeat the measurements of y N times, we can see a clear ...
1
vote
0answers
49 views
Appropriate noisy optimization method
Say we want to minimize a function $f(x)$ for $x$ in the interval $(0, 1)$. We can't query $f(x)$, and the gradient is not available; instead we can only sample from $f(x) + \epsilon$ where $\epsilon$...
0
votes
1answer
43 views
Can you recommend neural network suitable for finding minimum and maximum of numbers in list with some noisy data?
Let's have a list of integer numbers. These are generated about some "center" value with some tolerance +/-. I call this the "range". Also - there is some noise - the numbers are ...
0
votes
0answers
31 views
Determine if a spike in the data is signal or noise
I have a capacitive touch sensor that briefly engages every 15 seconds. The spikes sometimes happen when other parts of the device are touched, but sometimes happen on their own. The grey bar in the ...
0
votes
0answers
23 views
Dealing with noise when using the matrix profile
We are using the matrix profile for anomaly detection in time-series. It works well when there is less noise in the time-series. However, we face several false positives in the presence of noise. In ...
6
votes
2answers
132 views
Estimating growth rate from noisy data?
Let's say we want to estimate growth rate from noisy data. Due to noise, simple calculation will result in very poor estimates (calculating growth rates just exacerbates noise), so smoothing is needed....
0
votes
0answers
21 views
What is the resulting function to describe a log-normal function with Gaussian noise added?
I have a log-normal distribution:
$\frac{1}{x} Ae^{-\frac{(ln(x)-\mu)^2}{2\sigma^2}}$
And I draw from it (1000 times) and plot the draws in a histogram:
The log-normal function from which the draws ...
0
votes
0answers
19 views
Improving clusters with little observations
I am trying to cluster a small dataset of nearly 400 observations.
For this, I first tried kmeans. After tunning the number of clusters I got:
Then, in order to account for noise, I tried DBSCAN. ...
2
votes
1answer
50 views
Dealing with label noise (Regression, NLP)
For my school project, my group is tackling this Kaggle challenge (assign reading level based on passage).
commonlitreadabilityprize
However, it seems there is some label noise (examples below, lower ...
0
votes
0answers
14 views
Reducing noise in transactional data
I have a dataset sent to me weekly with data in daily format for 3-4 large retailers. Metrics I can see are number of customers, transaction counts, and total spend in dollars. The sample size is ...
1
vote
0answers
19 views
Make use of wrong label in learning?
I have a dataset. Each sample has two labels. The labels in the first set are mostly correct (>90%). The other labels, say annotated by an inexperienced annotator, are mostly incorrect, but they ...
0
votes
1answer
33 views
How to account for known bias in classification data
I apologize for the vagueness beforehand. Here's my experimental setup. I am trying to see if a data point has a property p. For example, in an image classification ...
3
votes
1answer
63 views
Why isn't every nonparametric model with random model design an additive noise model?
Let $Y$ be a real random variable and $X$ be a real random vector. In a nonparametric model with additive noise, we assume the relationship
$$Y = f(X) + \epsilon$$
for some unknown regression function ...
1
vote
0answers
20 views
best loss function to fit model if observations contain montecarlo noise?
I have observations on the sphere and I'm trying to fit spherical-harmonic coefficients to best approximate and interpolate the observations.
I'm using a solver library for non-linear least squares ...
3
votes
1answer
48 views
Does Gaussian Noise tend to move the Pearson Correlation to zero?
I'm wondering if adding Gaussian Noise to two waveforms tends to decrease their Pearson correlation. Below is a simulation of adding noise to two waveforms (...
0
votes
0answers
10 views
Measuring consistent misclassifications from confusion matrix
I'm doing semantic segmentation of images. I want to find to what extent certain classes are misclassified for certain other classes. I compute a confusion matrix for each image in the training set, ...
0
votes
0answers
104 views
Pre Image Problem in Kernel PCA: Understanding a Paper
Synopsis
I want to implement the method for finding pre-images of a feature vector in the feature space after applying kernel methods described in
https://www.aaai.org/Papers/ICML/2003/ICML03-055.pdf
...
0
votes
0answers
71 views
Maximum noise fraction (mnf)
There is a technique in remote sensing based on PCA which is called MNF-Maximum Noise Fraction(sometimes called also Minimum Noise Fraction), which from some reason is not known among other science ...
0
votes
0answers
10 views
Variance of a sample mean with stochastically evolving samples
I have $N$ time dependent measurements, $x_i(t)$, where $N > 100$ and $x_i(t)$ are bounded in the range $[-1,1]$. These measurements evolve in time randomly (stochastically) with some kind of ...
0
votes
1answer
24 views
Given a dataset, how does one know whether there’s (statistical) noise on a set of measurements or not?
Just out of curiosity, I’m looking at linear regression and there are of course cases where the measurements of the response variable are not noisy(say in the case of a simulator). But in general how ...
0
votes
0answers
66 views
How to group every data point with HDBSCAN to some group to have no noise?
TASK
I am clustering products with about 70 dimensions ex.: price, rating 5/5, product tag(cleaning, toy, food, fruits)
I use HDBSCAN to do it
GOAL
The goal is when users come on our site and I can ...
0
votes
0answers
15 views
Variance of DFT of filtered noise
I am struggling with the following question:
Let v(t) be a stationary stochastic process with Gaussian probability distribution and power spectral density $S(\omega)$. Let the DFT of $v(t)$ be $V(k)=\...
0
votes
0answers
14 views
Binary dependent variable with different accuracy of the two levels
My data is of dyadic nature (observation unit is the relationship between two actors) and my dependent variable is dichotomous; taking the value "1" if one of the two actors indicates an ...
1
vote
0answers
9 views
Classification where only a small subset of all samples are actually predictive
Hello datascience community
I have a classification task at hand where only a small subset (<1%) of all samples are actually predictive and even then the signal (i.e. correlation between input ...
0
votes
0answers
13 views
Does Basin Hopping Optimize On Noisy Functions?
Basin Hopping (thence Simulated Annealing) looks for gradients of the unknown objective function to locate its minima through a noisy process of visiting various points in the function's domain. ...
0
votes
1answer
60 views
Why is the average noise generated by the two-sided geometric distribution not null?
I am implementing the distributed differential privacy scheme proposed in this paper http://www.elaineshi.com/docs/ndss2011.pdf. Page 13 they represent a graph with the error added by a naive scheme ...
1
vote
0answers
12 views
Decoupling noises
This is an experimental physics problem:
Say I have 3 random variables $P_1, P_2, P_3 $ such as :
$\Delta P_i=\Delta_Q P_i +\Delta_0P_i \ \ \forall i$, where $\Delta$ is the variance, this equations ...
0
votes
0answers
8 views
Eliminate signal caused by panel changes
I have a timeseries that I am trying to eliminate signals caused by a changing panel size. Essentially I am trying to get the real predicted values from somewhat noisy reported values with the noise ...
2
votes
0answers
26 views
Training dataset from analytical solution
I am currently redesigning an inverse problem on an experimental technique, but I am having doubts about how to create a training dataset. Here is the problem I am trying to solve:
I have already ...
1
vote
0answers
153 views
Using eigenvalues of the covariance matrix to reduce noise in my data
I have an idea to help reduce the noise in my signal but am stuck with a significant problem.
I have a very noisy data set $y_n[t]; n\in\{0, N_{\text{samples}}-1\}; t\in\{0, T-1\}$ I am fitting this ...
1
vote
0answers
9 views
Noise removal from a matrix when the noise covariance structure is known
I have a matrix made by the addition of the signal and the noise. I know the covariance structure of the noise but don't have the actual noise matrix. Is there a way to remove the noise/obtain an ...
1
vote
0answers
63 views
linear regression using other norms
Let us consider the linear regression model in finite dimensions given by $Y = X \beta + \epsilon$ where $Y \in \mathbb{R}^n, X \in \mathbb{R}^{n \times m}, \beta \in \mathbb{R}^m$, and $ \epsilon \in ...
2
votes
3answers
152 views
What is the equivalent of linear Gaussian noise but for discrete data
A common framework in stats and ML is that our data is noisy observation of an underlying true value, e.g.
$$
\begin{aligned}
\mathbf{x} &= \mathbf{z} + \boldsymbol{\varepsilon},
\\
\boldsymbol{\...
1
vote
0answers
36 views
Bias in estimation of a latent / hidden variable drawn from a skewed distribution: what is it called?
I observe a bias effect in my measurement system that I can explain and correct using a simple latent or hidden variable model. I am sure this kind of effect has been described earlier in other fields ...
0
votes
1answer
28 views
Remove subtractive noise
I have a dataset that I'd like to remove noise from. All the noise is subtractive, so the ideal value would always be greater than or equal to the measured value.
The following chart shows the raw ...
5
votes
2answers
546 views
Noise in regression problems and ways to reduce it
In the theory of bias-variance decomposition for regression problems (this page is a very nice reference on this theory) noise is defined as
$$\mathrm{Noise} = \mathrm{E}_{X,Y}[(Y - \mathrm{E}[Y|X])^2]...
2
votes
1answer
66 views
Autoregressive model with observable noise
The classical autoregressive model is a linear model for the dynamic variable $x$, where the added noise $\epsilon$ is directly affecting the dynamics of the model
$$x_{t} = \sum_i \alpha_i x_{t-i} + \...
