Skip to main content

Questions tagged [noise]

noise is a term used for the error term in statistical models and in signal processing. It could be white noise, colored noise or otherwise.

Filter by
Sorted by
Tagged with
0 votes
0 answers
24 views

Regression with noises in X. Should I use the unbiased estimator or the OLS estimator for forecasting?

I am working with a dataset that includes variables $Y$ and $X$. I assume that $$ Y = \beta X + \epsilon $$ satisfies all the assumptions of OLS. Based on industry knowledge, I know that theoretically ...
The One's user avatar
  • 225
0 votes
0 answers
11 views

What if there is only one measurement equation containing two (or more) state variables while there are two unobservable state variables in a model?

I am learning Kalman Filter and ran into a question about the case in which only one signal is available. It is commonly assumed that the number of states equals the number of observations (signals) ...
user14261785's user avatar
1 vote
0 answers
63 views

Calculate inter-rater noise using Kahnemans (2021) approach

I need help calculating signal and noise based on the method described by Kahneman et al. (2021) in their book "Noise." They provide a technique for quantifying noise between raters ...
Magnus Nordmo's user avatar
6 votes
2 answers
259 views

Logistic regression with labels corrupted by known noise model

I am interested in knowing the "right way" to fit a binary logistic regression where the labels have been flipped with instance-specific noise probabilities that are known. For the scenario ...
ted's user avatar
  • 741
1 vote
1 answer
76 views

Non-linear regression with very noisy data with nls() in R

I am trying to fit noisy data to a specific model with two parameters which I would like to estimate. Unfortunately, the model fit is just terrible with added noise. Is there anything I can do to ...
leze's user avatar
  • 11
0 votes
0 answers
14 views

Predicting spelling errors: how should the noisy channel be modeled?

I'm interested in using the noisy channel approach to do spelling correction. I'm particularly interested in using this technique when a spelling mistake results in another existing word. E.g. ...
Seán Healy's user avatar
0 votes
0 answers
41 views

Should I interprete data as noise or not

I am tackling a classification problem with 3 classes. Here is what those classes look like on the Two first principal axes. I fine-tuned a SVM model and the best performance achievable was 50%. By ...
Yann's user avatar
  • 43
0 votes
1 answer
182 views

What is the variance of convolution of two random variables?

Consider two random variables $Z$ and $W$. Given the variances of $Z$ and $W$, how can we compute the variance of their convolution $Z \circledast W $? As an example, please consider the case of noise ...
user409495's user avatar
0 votes
0 answers
20 views

How to interpret Allan Variance curves that do not follow the canonical shape

I am currently working on characterizing the noise sources of a Global Navigation Satellite System (GNSS) sensor using an Allan Variance plot, which is commonly employed to analyze frequency stability ...
RoninAmibo's user avatar
3 votes
1 answer
129 views

Gaussian noise added in social sciences data

In a simulation study (number of simulation $n=200$), there is this quadratic/parabolic function simulated with Gaussian noise added: ...
varin sacha's user avatar
0 votes
0 answers
17 views

Remove non repeatable stochastic noise from an image

Assume that you have these coordinates inside an image. The algorithm for creating these crosses comes from FAST-algorithm for corner detection. But the problem with FAST-algorithm is that some of ...
euraad's user avatar
  • 425
1 vote
0 answers
178 views

Regression ML model: Data Augmentation [closed]

I'm currently working on data augmentation to my regression problem, and a (possible) solution that came to my mind was to add a perturbed dataset to the original dataset, and hence double the ...
Pedro Italo's user avatar
0 votes
0 answers
32 views

How can I create realistic noisy data from distributions?

I want to create synthetic data from stitched distributions in order to test some models on them (for example Gaussian stitched with a GPD at quantile q). I'm currently simply sampling N*q points from ...
Philippe Ear's user avatar
0 votes
1 answer
51 views

Root Mean Square Error of the addition of two measurements whose RMS Error is known

I am working on a measurement system which tries to measure the distance between two values i.e $\Delta F=F_1-F_2$. Where $F_1, F_2$ are the values I actually measure. I have set up a Monte Carlo ...
bad_at_stats's user avatar
1 vote
0 answers
56 views

Estimating a secret with before/after interchanging noises

$\newcommand{\Var}{\mathrm{Var}}\newcommand{\E}{\mathrm{E}}$ For $n+1$ iid "noise" variables $X_0,\dots,X_n$ from the normal distribution $\mathcal{N}(0,1)$ and a "secret" $s$ ...
Nathan's user avatar
  • 11
0 votes
0 answers
31 views

KDE-like technique to learn a continuous distribution from samples subject to specific noise

There's a continuous-valued random variable $X$ with distribution $f_X$. Normally, we're given a bunch of i.i.d. samples $X_1, \ldots, X_n$, and we try to give an estimate $\hat{f}_X$ of the ...
chausies's user avatar
  • 421
1 vote
0 answers
61 views

Can I fit a Poisson distribution to a continuous variable to apply event detection algorithm?

Preprocessing: I have a time series of number of tweets per 10 minutes time interval that are all taken from a given discussion on a specific topic in a specific region. I preprocessed the data by ...
Mim_Tauch's user avatar
0 votes
0 answers
31 views

Deriving parameters of gaussian noise that will lead to bounded random walk

Given a bound on random walk, I am trying to derive the parameters of a normal distribution of noise, which when added to a signal and integrated will lead to random walk within specified bounds. My ...
rocksNwaves's user avatar
1 vote
0 answers
80 views

How to find the curvature of a noisy function with few evaluations?

I'm working on a project in computational solid state physics where I have to find the curvature of a smooth function $f \ : [0,\infty] \rightarrow \mathbb{R}$ around the minimum point $x_0 = 0$. The ...
Jakob KS's user avatar
  • 111
12 votes
7 answers
2k views

Noise cancels but variance sums - contradiction?

I have been told both things with regard to e.g. summing noisy time series, to justify opposing expectations. On the one hand, I have been told to expect that summing multiple noisy inputs should lead ...
benxyzzy's user avatar
  • 333
4 votes
1 answer
877 views

HDBSCAN on UMAP output

My question is about using UMAP as a dimensional reduction technique before HDBSCAN clustering. I have a dataset of ~5000 observations each with ~20 descriptors. According to HDBSCAN guidelines, ...
IvyBlue's user avatar
  • 43
0 votes
1 answer
57 views

How should I understand the noise variance parameter(s) in multifidelity modeling using Emukit

I am learning the multi-fidelity modeling and have a question about Emukit's mixed_noise parameters, or more general, how should we determine the noise when the ...
Ann's user avatar
  • 43
0 votes
0 answers
19 views

How to quantify prediction power?

I have data that is the result of measurements $f(x)$ at points $x$. These measurements fluctuate (call it noise), also differently for each $x$ so we have that $\sigma(x)$ are the fluctuations. By ...
user171780's user avatar
9 votes
2 answers
785 views

Standard Error of Noise Variance in Least Squares

Suppose I'm doing ordinary least squares with homoskedastic errors, so something like: $$ y = X \beta + \epsilon $$ where $\epsilon \sim \mathcal{N}(0,\sigma^2)$. I know how to estimate the expected ...
Ben's user avatar
  • 255
2 votes
2 answers
81 views

What is the relationship between effect size and noise?

I understand that noise in measurement can affect the size of an observed effect. That noise can result in larger measurement error, which can reduce the size of the observed effect. However, I do not ...
psych_student's user avatar
0 votes
0 answers
93 views

How to add noise into a standard distribution without increasing its variance?

Suppose I have a standard distribution dataset X with a mean 0 and std 1. Now I want to create slight variations of this data by injecting some noise. I could make ...
Anonymous's user avatar
  • 181
2 votes
0 answers
34 views

How do professional statisticians evaluate or mathematically justify choice of one smoothing method from another? [closed]

I would like to fit some economic data and evaluate it. Usually I used only regression techniques, but recently I found out about smoothing techniques which could be used to diminish noise in dataset ...
SCP_org's user avatar
  • 21
0 votes
0 answers
31 views

the signal (numerically generated) is polluted by white noise with amplitude of 10% -- what does that mean?

In a paper, I read that "the values are polluted by white noise of amplitude up to 10%". Say, the referred values are stored in a vector x, what is meant ...
Simon's user avatar
  • 133
2 votes
1 answer
69 views

Prove that Residuals/Errors are not Predictable [duplicate]

Let's say I have a set of data and I modelled it using an ML algorithm. After I fit multiple models I achieve a certain level of accuracy. It doesn't get better beyond that point. I want to prove ...
Lopez's user avatar
  • 147
1 vote
1 answer
93 views

Are ARMA models used to model the noise in a time series?

If we have a time series $y_t$ and we want to model it, is the following intuition about the ARMA model correct? $$Y_t=f(\beta,t) + e_t $$ where $e_t \sim$ ARMA. So there is a decomposition of a ...
Keep_On_Cruising's user avatar
0 votes
0 answers
2k views

Silhouette Score with Noise (from DBSCAN)

I stumbled across this example on scikit-learn (1.2.0), where the silhouette score alongside some other metrics is computed for DBSCAN cluster assignments. These assignments include some Noise ...
MERose's user avatar
  • 419
3 votes
1 answer
1k views

Is it possible to regularize a covariance matrix?

I have many "parallel" time-series (about 100). They have relatively short history. I calculate a covariance matrix between these time-series. Now, I believe that the observed covariance ...
Roman's user avatar
  • 622
1 vote
0 answers
312 views

What is a mathematic rigorous definition of "blue noise"?

Let $d\in\mathbb N$, $I$ be a finite nonempty set, $(x_i)_{i\in I}\subseteq[0,1)^d$, $(w_i)_{i\in I}\subseteq[0,\infty)$ with $\sum_{i\in I}w_i=1$ and $$\sigma:=\sum_{i\in I}w_i\delta_{x_i}.$$ I ...
0xbadf00d's user avatar
  • 303
1 vote
1 answer
119 views

Why can't I detect concept drift with linear regression?

When I was first trying to detect concept drift, it seemed naively to me to be a problem of detecting whether noisy data was veering off horizontal trajectory (non-trivial slope above some arbitrary ...
Sanger Steel's user avatar
0 votes
0 answers
28 views

Prooving the convergence rate of noisy estimators (machine Learning)

I want to estimate a quantity and have two choices for estimators (they both sample from the same distribution). I suspect one of them is nosier and has a slower convergence rate. I want to ...
postnubilaphoebus's user avatar
3 votes
0 answers
53 views

Remove Additive Zero Mean Gaussian Noise

I have six values, and each value is corrupted by Additive-Zero-Mean-Gaussian Noise with var = 0.05). Each value is range from 0 to 1. Is there anyway for me to remove these additive noise?
wrek's user avatar
  • 195
4 votes
1 answer
681 views

Detecting peaks in noisy quasi-periodic time series

I have the following acceleration data taken from my phone when rowing The negative peak is the point at which the oars go in the water. I would like to be able to identify this peak in real time ...
Will P's user avatar
  • 43
6 votes
1 answer
619 views

XGBoost when P>>N

Someone built an XGBoost classification model using each pixel in an image (256*256) as a separate feature, plus a few other features. However they only have 500 data points. The target classes were ...
Alex's user avatar
  • 185
1 vote
1 answer
397 views

GAN artifacts on borders

not quite a math-question, but I have a doubt. I'm trying to build from scratch the Pix2pix network, on the facades dataset, and I think I finally got a good model (from the paper I borrowed just the ...
Alberto's user avatar
  • 1,217
3 votes
2 answers
330 views

What is the relationship between noise reduction and dimension reduction?

My understanding is that unsupervised methods like PCA, autoencoders and K-means shape a data space such that the modified representation of the data either nicely separates different families of data ...
Douw Marx's user avatar
0 votes
1 answer
65 views

Ways to detect noise in multi-class classification training data using text embeddings (BERT)

So I have a dataset with a column of text and and labels (5 different labels) associated with it. The labels describe the potential answer to the type of question being asked in the text column. For ...
user11715878's user avatar
2 votes
1 answer
131 views

Means to estimate the decay rate of a noisy decay process?

I have a decay process which appears essentially like $$ f(t) = \xi(t)\exp[-t/\tau],$$ where $\xi(t)$ is a stationary Gaussian noise with some mean, variance, and correlation function. Given a ...
kevinkayaks's user avatar
3 votes
1 answer
352 views

Adding noise to non-negative imputed data

I have an hourly time series of wind speed data that spans 8 years. Wind speed must be non-negative but there are only ~80 out of ~70000 values that are exactly 0; the remainder are either positive or ...
qdread's user avatar
  • 295
0 votes
1 answer
294 views

What is Hamming Window in Audio Analysis?

I am reading this paper. The paper writes ... an input waveform of t seconds is converted into a sequence of 128-D log mel filter bank features computed with 25 ms Hamming window every 10 s. The ...
onexpeters's user avatar
1 vote
0 answers
34 views

How to infer noise map from noisy data?

I am given noisy measurements of the solution of a PDE at different times, each measurement is drawn from a Gaussian distribution having as mean the solution of the PDE at that time and a standard ...
user360500's user avatar
0 votes
0 answers
100 views

Adding noise to a dataset to "hide" its distribution

Given a dataset that follows "some"[1] distribution, how can we add noise to the dataset, such that when the original dataset plus the noise is shown to someone, it looks uniformly random[2]?...
pranavJ's user avatar
1 vote
1 answer
68 views

Estimating the standard deviation

I have data of a function $f(t)$, where I always have just a single data point per $t$. It is known that the data is affected by noise, where the noise itself is also a function of $t$. Now I want to ...
Johny Dow's user avatar
3 votes
1 answer
349 views

What jitter value should I use for this scatter plot?

I plotted a scatterplot between humidity and temperature (air) in centigrade. I got the following graph; It is evident that the points fall in discrete columns. This might be because of the rounding ...
Ritik P. Nayak's user avatar
1 vote
1 answer
23 views

Training twice on non-injective data

I have a large dataset of 30000 points, but most my Ys are the same while all Xs are different. Ys are from different samples, so I had means of Y for each sample and I used means alongside Xs to ...
Shayan Kabiri's user avatar
0 votes
0 answers
322 views

What is the combined standard deviation of Poisson photon shot noise and Gaussian read noise?

On various image sensor web sites (e.g., https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=10773, https://www.microscopyu.com/tutorials/ccd-signal-to-noise-ratio you'll see something to the ...
planetfolly's user avatar

1
2 3 4 5 6