Questions tagged [noise]
noise is a term used for the error term in statistical models and in signal processing. It could be white noise, colored noise or otherwise.
270
questions
0
votes
0
answers
17
views
Remove non repeatable stochastic noise from an image
Assume that you have these coordinates inside an image. The algorithm for creating these crosses comes from FAST-algorithm for corner detection.
But the problem with FAST-algorithm is that some of ...
- 381
1
vote
0
answers
21
views
Can I use percentile relative to a null distribution as a dependent variable?
A common practice in my field is to test that the relationship between Time Series A and Time Series B is statistically significant by re-coupling Time Series B against a randomly permuted version of ...
- 65
1
vote
0
answers
58
views
Regression ML model: Data Augmentation [closed]
I'm currently working on data augmentation to my regression problem, and a (possible) solution that came to my mind was to add a perturbed dataset to the original dataset, and hence double the ...
- 111
0
votes
0
answers
29
views
How can I create realistic noisy data from distributions?
I want to create synthetic data from stitched distributions in order to test some models on them (for example Gaussian stitched with a GPD at quantile q).
I'm currently simply sampling N*q points from ...
0
votes
1
answer
22
views
Root Mean Square Error of the addition of two measurements whose RMS Error is known
I am working on a measurement system which tries to measure the distance between two values i.e $\Delta F=F_1-F_2$. Where $F_1, F_2$ are the values I actually measure. I have set up a Monte Carlo ...
1
vote
0
answers
56
views
Estimating a secret with before/after interchanging noises
$\newcommand{\Var}{\mathrm{Var}}\newcommand{\E}{\mathrm{E}}$
For $n+1$ iid "noise" variables $X_0,\dots,X_n$ from the normal distribution $\mathcal{N}(0,1)$ and a "secret" $s$ ...
- 11
0
votes
0
answers
24
views
KDE-like technique to learn a continuous distribution from samples subject to specific noise
There's a continuous-valued random variable $X$ with distribution $f_X$. Normally, we're given a bunch of i.i.d. samples $X_1, \ldots, X_n$, and we try to give an estimate $\hat{f}_X$ of the ...
- 421
1
vote
0
answers
32
views
Can I fit a Poisson distribution to a continuous variable to apply event detection algorithm?
Preprocessing:
I have a time series of number of tweets per 10 minutes time interval that are all taken from a given discussion on a specific topic in a specific region. I preprocessed the data by ...
- 41
0
votes
0
answers
21
views
Deriving parameters of gaussian noise that will lead to bounded random walk
Given a bound on random walk, I am trying to derive the parameters of a normal distribution of noise, which when added to a signal and integrated will lead to random walk within specified bounds.
My ...
- 380
1
vote
0
answers
75
views
How to find the curvature of a noisy function with few evaluations?
I'm working on a project in computational solid state physics where I have to find the curvature of a smooth function $f \ : [0,\infty] \rightarrow \mathbb{R}$ around the minimum point $x_0 = 0$.
The ...
- 111
0
votes
0
answers
28
views
How to treat non-normal measurement noise in a Kalman filter? Plus, how to treat non-zero mean of noise in Kalman filter?
Typically in textbooks, it is assumed that measurement noise $\nu_{t}$ is normally distributed. Suppose that $S_{t}$ is a signal. Then a measurement equation is
$$
S_{t} = x_{t} +\nu_{t} $$
where $$ \...
- 11
0
votes
0
answers
70
views
What is sigma for a time series and how to calculate it
Trying to apply changepoint detection. Many algorithms such as Binary segmentation and PELT use sigma ( std deviation ) as a parameter if the number of changepoints are unknown.
Confused if sigma is ...
- 193
12
votes
7
answers
2k
views
Noise cancels but variance sums - contradiction?
I have been told both things with regard to e.g. summing noisy time series, to justify opposing expectations.
On the one hand, I have been told to expect that summing multiple noisy inputs should lead ...
- 323
0
votes
0
answers
24
views
blue noise error distribution in (MC)MC estimation
In computer graphics, you have an (MC)MC estimate $Q_i$ of the color value of the $i$th pixel and a true value $I_i$. Now you take $\epsilon_i:=Q_i-I_i$, which can be thought of as an "error ...
- 141
2
votes
1
answer
355
views
HDBSCAN on UMAP output
My question is about using UMAP as a dimensional reduction technique before HDBSCAN clustering. I have a dataset of ~5000 observations each with ~20 descriptors. According to HDBSCAN guidelines, ...
- 23
0
votes
1
answer
41
views
How should I understand the noise variance parameter(s) in multifidelity modeling using Emukit
I am learning the multi-fidelity modeling and have a question about Emukit's mixed_noise parameters, or more general, how should we determine the noise when the ...
- 43
0
votes
0
answers
19
views
How to quantify prediction power?
I have data that is the result of measurements $f(x)$ at points $x$. These measurements fluctuate (call it noise), also differently for each $x$ so we have that $\sigma(x)$ are the fluctuations. By ...
- 229
8
votes
2
answers
551
views
Standard Error of Noise Variance in Least Squares
Suppose I'm doing ordinary least squares with homoskedastic errors, so something like:
$$
y = X \beta + \epsilon
$$
where $\epsilon \sim \mathcal{N}(0,\sigma^2)$.
I know how to estimate the expected ...
- 235
0
votes
0
answers
16
views
In supervised learning, how to deal with noisy labels if I care little about recall and a lot about precision
I am trying to solve a binary classification problem with supervised learning. Tabular dataset. I have many labels, however I know they are very noisy. So noisy that it is not realistic to get a good ...
- 346
1
vote
2
answers
58
views
What is the relationship between effect size and noise?
I understand that noise in measurement can affect the size of an observed effect. That noise can result in larger measurement error, which can reduce the size of the observed effect. However, I do not ...
0
votes
0
answers
134
views
How to reduce noise in the time-series data in order to prevent overfitting?
I'm working on a time-series dataset for pain levels classification.
Data: 130 patients, each has 5 high frequency (1000hz) signals and corresponding categorical labels at each timestep $t$.
I tried ...
- 1
0
votes
0
answers
67
views
How to add noise into a standard distribution without increasing its variance?
Suppose I have a standard distribution dataset X with a mean 0 and std 1.
Now I want to create slight variations of this data by injecting some noise.
I could make ...
- 181
2
votes
0
answers
32
views
How do professional statisticians evaluate or mathematically justify choice of one smoothing method from another? [closed]
I would like to fit some economic data and evaluate it. Usually I used only regression techniques, but recently I found out about smoothing techniques which could be used to diminish noise in dataset ...
- 21
0
votes
0
answers
30
views
the signal (numerically generated) is polluted by white noise with amplitude of 10% -- what does that mean?
In a paper, I read that "the values are polluted by white noise of amplitude up to 10%".
Say, the referred values are stored in a vector x, what is meant ...
- 123
2
votes
1
answer
64
views
Prove that Residuals/Errors are not Predictable [duplicate]
Let's say I have a set of data and I modelled it using an ML algorithm. After I fit multiple models I achieve a certain level of accuracy. It doesn't get better beyond that point.
I want to prove ...
- 147
1
vote
1
answer
79
views
Are ARMA models used to model the noise in a time series?
If we have a time series $y_t$ and we want to model it, is the following intuition about the ARMA model correct?
$$Y_t=f(\beta,t) + e_t $$ where $e_t \sim$ ARMA. So there is a decomposition of a ...
- 225
0
votes
0
answers
54
views
How to mitigate spatially correlated noise in image
I was wondering how can I mitigate spatially correlated noise following power law with filtering or other techniques?
For instance for astrophsics you can simulate it
...
0
votes
0
answers
19
views
Information gained from stream of correlated, normally-distributed symbols through a noisy channel
I have the following problem:
I want to transmit a certain amount of information through a channel with Gaussian noise (random variable Z). The input distribution is likewise Gaussian (random variable ...
0
votes
0
answers
1k
views
Silhouette Score with Noise (from DBSCAN)
I stumbled across this example on scikit-learn (1.2.0), where the silhouette score alongside some other metrics is computed for DBSCAN cluster assignments. These assignments include some Noise ...
- 419
2
votes
1
answer
828
views
Is it possible to regularize a covariance matrix?
I have many "parallel" time-series (about 100). They have relatively short history. I calculate a covariance matrix between these time-series.
Now, I believe that the observed covariance ...
- 554
1
vote
0
answers
259
views
What is a mathematic rigorous definition of "blue noise"?
Let $d\in\mathbb N$, $I$ be a finite nonempty set, $(x_i)_{i\in I}\subseteq[0,1)^d$, $(w_i)_{i\in I}\subseteq[0,\infty)$ with $\sum_{i\in I}w_i=1$ and $$\sigma:=\sum_{i\in I}w_i\delta_{x_i}.$$
I ...
- 141
1
vote
1
answer
85
views
Why can't I detect concept drift with linear regression?
When I was first trying to detect concept drift, it seemed naively to me to be a problem of detecting whether noisy data was veering off horizontal trajectory (non-trivial slope above some arbitrary ...
- 33
0
votes
0
answers
28
views
Prooving the convergence rate of noisy estimators (machine Learning)
I want to estimate a quantity and have two choices for estimators (they both sample from the same distribution). I suspect one of them is nosier and has a slower convergence rate. I want to ...
3
votes
0
answers
52
views
Remove Additive Zero Mean Gaussian Noise
I have six values, and each value is corrupted by Additive-Zero-Mean-Gaussian Noise with var = 0.05).
Each value is range from 0 to 1. Is there anyway for me to remove these additive noise?
- 315
4
votes
1
answer
507
views
Detecting peaks in noisy quasi-periodic time series
I have the following acceleration data taken from my phone when rowing
The negative peak is the point at which the oars go in the water. I would like to be able to identify this peak in real time ...
- 43
5
votes
1
answer
515
views
XGBoost when P>>N
Someone built an XGBoost classification model using each pixel in an image (256*256) as a separate feature, plus a few other features. However they only have 500 data points. The target classes were ...
- 165
1
vote
1
answer
315
views
GAN artifacts on borders
not quite a math-question, but I have a doubt.
I'm trying to build from scratch the Pix2pix network, on the facades dataset, and I think I finally got a good model (from the paper I borrowed just the ...
- 859
3
votes
2
answers
206
views
What is the relationship between noise reduction and dimension reduction?
My understanding is that unsupervised methods like PCA, autoencoders and K-means shape a data space such that the modified representation of the data either nicely separates different families of data ...
- 53
0
votes
1
answer
61
views
Ways to detect noise in multi-class classification training data using text embeddings (BERT)
So I have a dataset with a column of text and and labels (5 different labels) associated with it. The labels describe the potential answer to the type of question being asked in the text column. For ...
1
vote
1
answer
100
views
Means to estimate the decay rate of a noisy decay process?
I have a decay process which appears essentially like
$$ f(t) = \xi(t)\exp[-t/\tau],$$
where $\xi(t)$ is a stationary Gaussian noise with some mean, variance, and correlation function.
Given a ...
- 276
3
votes
1
answer
286
views
Adding noise to non-negative imputed data
I have an hourly time series of wind speed data that spans 8 years. Wind speed must be non-negative but there are only ~80 out of ~70000 values that are exactly 0; the remainder are either positive or ...
- 215
0
votes
1
answer
154
views
What is Hamming Window in Audio Analysis?
I am reading this paper. The paper writes
... an input waveform of t seconds is converted into a sequence of 128-D log mel filter bank features computed with 25 ms Hamming window every 10 s. The ...
- 21
1
vote
0
answers
30
views
How to infer noise map from noisy data?
I am given noisy measurements of the solution of a PDE at different times, each measurement is drawn from a Gaussian distribution having as mean the solution of the PDE at that time and a standard ...
- 11
0
votes
0
answers
80
views
Adding noise to a dataset to "hide" its distribution
Given a dataset that follows "some"[1] distribution, how can we add noise to the dataset, such that when the original dataset plus the noise is shown to someone, it looks uniformly random[2]?...
- 1
1
vote
1
answer
64
views
Estimating the standard deviation
I have data of a function $f(t)$, where I always have just a single data point per $t$. It is known that the data is affected by noise, where the noise itself is also a function of $t$. Now I want to ...
- 35
2
votes
1
answer
291
views
What jitter value should I use for this scatter plot?
I plotted a scatterplot between humidity and temperature (air) in centigrade. I got the following graph;
It is evident that the points fall in discrete columns. This might be because of the rounding ...
- 163
1
vote
1
answer
23
views
Training twice on non-injective data
I have a large dataset of 30000 points, but most my Ys are the same while all Xs are different. Ys are from different samples, so I had means of Y for each sample and I used means alongside Xs to ...
0
votes
0
answers
286
views
What is the combined standard deviation of Poisson photon shot noise and Gaussian read noise?
On various image sensor web sites (e.g., https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=10773, https://www.microscopyu.com/tutorials/ccd-signal-to-noise-ratio you'll see something to the ...
1
vote
0
answers
54
views
Additive error model for non-linear case
I have checked other questions regarding additive noise model, but I could not convince myself for non-linear case. Assume the data vector $\mathbf{d}$ is described by a possibly non-linear ...
- 9
4
votes
2
answers
245
views
Does a targets-permutation test prove that regression find a real pattern?
I need to solve a standard ("vanilla") regression problem meaning that I have a 2D array of real-valued features (X) and 1D array of real-valued targets (<...
- 554