Questions tagged [noise]

noise is a term used for the error term in statistical models and in signal processing. It could be white noise, colored noise or otherwise.

Filter by
Sorted by
Tagged with
0 votes
0 answers
17 views

Remove non repeatable stochastic noise from an image

Assume that you have these coordinates inside an image. The algorithm for creating these crosses comes from FAST-algorithm for corner detection. But the problem with FAST-algorithm is that some of ...
euraad's user avatar
  • 381
1 vote
0 answers
21 views

Can I use percentile relative to a null distribution as a dependent variable?

A common practice in my field is to test that the relationship between Time Series A and Time Series B is statistically significant by re-coupling Time Series B against a randomly permuted version of ...
stck8888's user avatar
1 vote
0 answers
58 views

Regression ML model: Data Augmentation [closed]

I'm currently working on data augmentation to my regression problem, and a (possible) solution that came to my mind was to add a perturbed dataset to the original dataset, and hence double the ...
Pedro Italo's user avatar
0 votes
0 answers
29 views

How can I create realistic noisy data from distributions?

I want to create synthetic data from stitched distributions in order to test some models on them (for example Gaussian stitched with a GPD at quantile q). I'm currently simply sampling N*q points from ...
Philippe Ear's user avatar
0 votes
1 answer
22 views

Root Mean Square Error of the addition of two measurements whose RMS Error is known

I am working on a measurement system which tries to measure the distance between two values i.e $\Delta F=F_1-F_2$. Where $F_1, F_2$ are the values I actually measure. I have set up a Monte Carlo ...
bad_at_stats's user avatar
1 vote
0 answers
56 views

Estimating a secret with before/after interchanging noises

$\newcommand{\Var}{\mathrm{Var}}\newcommand{\E}{\mathrm{E}}$ For $n+1$ iid "noise" variables $X_0,\dots,X_n$ from the normal distribution $\mathcal{N}(0,1)$ and a "secret" $s$ ...
Nathan's user avatar
  • 11
0 votes
0 answers
24 views

KDE-like technique to learn a continuous distribution from samples subject to specific noise

There's a continuous-valued random variable $X$ with distribution $f_X$. Normally, we're given a bunch of i.i.d. samples $X_1, \ldots, X_n$, and we try to give an estimate $\hat{f}_X$ of the ...
chausies's user avatar
  • 421
1 vote
0 answers
32 views

Can I fit a Poisson distribution to a continuous variable to apply event detection algorithm?

Preprocessing: I have a time series of number of tweets per 10 minutes time interval that are all taken from a given discussion on a specific topic in a specific region. I preprocessed the data by ...
Mim_Tauch's user avatar
0 votes
0 answers
21 views

Deriving parameters of gaussian noise that will lead to bounded random walk

Given a bound on random walk, I am trying to derive the parameters of a normal distribution of noise, which when added to a signal and integrated will lead to random walk within specified bounds. My ...
rocksNwaves's user avatar
1 vote
0 answers
75 views

How to find the curvature of a noisy function with few evaluations?

I'm working on a project in computational solid state physics where I have to find the curvature of a smooth function $f \ : [0,\infty] \rightarrow \mathbb{R}$ around the minimum point $x_0 = 0$. The ...
Jakob KS's user avatar
  • 111
0 votes
0 answers
28 views

How to treat non-normal measurement noise in a Kalman filter? Plus, how to treat non-zero mean of noise in Kalman filter?

Typically in textbooks, it is assumed that measurement noise $\nu_{t}$ is normally distributed. Suppose that $S_{t}$ is a signal. Then a measurement equation is $$ S_{t} = x_{t} +\nu_{t} $$ where $$ \...
user14261785's user avatar
0 votes
0 answers
70 views

What is sigma for a time series and how to calculate it

Trying to apply changepoint detection. Many algorithms such as Binary segmentation and PELT use sigma ( std deviation ) as a parameter if the number of changepoints are unknown. Confused if sigma is ...
Sarang Manjrekar's user avatar
12 votes
7 answers
2k views

Noise cancels but variance sums - contradiction?

I have been told both things with regard to e.g. summing noisy time series, to justify opposing expectations. On the one hand, I have been told to expect that summing multiple noisy inputs should lead ...
benxyzzy's user avatar
  • 323
0 votes
0 answers
24 views

blue noise error distribution in (MC)MC estimation

In computer graphics, you have an (MC)MC estimate $Q_i$ of the color value of the $i$th pixel and a true value $I_i$. Now you take $\epsilon_i:=Q_i-I_i$, which can be thought of as an "error ...
0xbadf00d's user avatar
  • 141
2 votes
1 answer
355 views

HDBSCAN on UMAP output

My question is about using UMAP as a dimensional reduction technique before HDBSCAN clustering. I have a dataset of ~5000 observations each with ~20 descriptors. According to HDBSCAN guidelines, ...
IvyBlue's user avatar
  • 23
0 votes
1 answer
41 views

How should I understand the noise variance parameter(s) in multifidelity modeling using Emukit

I am learning the multi-fidelity modeling and have a question about Emukit's mixed_noise parameters, or more general, how should we determine the noise when the ...
Ann's user avatar
  • 43
0 votes
0 answers
19 views

How to quantify prediction power?

I have data that is the result of measurements $f(x)$ at points $x$. These measurements fluctuate (call it noise), also differently for each $x$ so we have that $\sigma(x)$ are the fluctuations. By ...
user171780's user avatar
8 votes
2 answers
551 views

Standard Error of Noise Variance in Least Squares

Suppose I'm doing ordinary least squares with homoskedastic errors, so something like: $$ y = X \beta + \epsilon $$ where $\epsilon \sim \mathcal{N}(0,\sigma^2)$. I know how to estimate the expected ...
Ben's user avatar
  • 235
0 votes
0 answers
16 views

In supervised learning, how to deal with noisy labels if I care little about recall and a lot about precision

I am trying to solve a binary classification problem with supervised learning. Tabular dataset. I have many labels, however I know they are very noisy. So noisy that it is not realistic to get a good ...
hipoglucido's user avatar
1 vote
2 answers
58 views

What is the relationship between effect size and noise?

I understand that noise in measurement can affect the size of an observed effect. That noise can result in larger measurement error, which can reduce the size of the observed effect. However, I do not ...
psych_student's user avatar
0 votes
0 answers
134 views

How to reduce noise in the time-series data in order to prevent overfitting?

I'm working on a time-series dataset for pain levels classification. Data: 130 patients, each has 5 high frequency (1000hz) signals and corresponding categorical labels at each timestep $t$. I tried ...
debrises's user avatar
0 votes
0 answers
67 views

How to add noise into a standard distribution without increasing its variance?

Suppose I have a standard distribution dataset X with a mean 0 and std 1. Now I want to create slight variations of this data by injecting some noise. I could make ...
Anonymous's user avatar
  • 181
2 votes
0 answers
32 views

How do professional statisticians evaluate or mathematically justify choice of one smoothing method from another? [closed]

I would like to fit some economic data and evaluate it. Usually I used only regression techniques, but recently I found out about smoothing techniques which could be used to diminish noise in dataset ...
SCP_org's user avatar
  • 21
0 votes
0 answers
30 views

the signal (numerically generated) is polluted by white noise with amplitude of 10% -- what does that mean?

In a paper, I read that "the values are polluted by white noise of amplitude up to 10%". Say, the referred values are stored in a vector x, what is meant ...
Simon's user avatar
  • 123
2 votes
1 answer
64 views

Prove that Residuals/Errors are not Predictable [duplicate]

Let's say I have a set of data and I modelled it using an ML algorithm. After I fit multiple models I achieve a certain level of accuracy. It doesn't get better beyond that point. I want to prove ...
Lopez's user avatar
  • 147
1 vote
1 answer
79 views

Are ARMA models used to model the noise in a time series?

If we have a time series $y_t$ and we want to model it, is the following intuition about the ARMA model correct? $$Y_t=f(\beta,t) + e_t $$ where $e_t \sim$ ARMA. So there is a decomposition of a ...
Keep_On_Cruising's user avatar
0 votes
0 answers
54 views

How to mitigate spatially correlated noise in image

I was wondering how can I mitigate spatially correlated noise following power law with filtering or other techniques? For instance for astrophsics you can simulate it ...
Mohammad Mohseni Aref's user avatar
0 votes
0 answers
19 views

Information gained from stream of correlated, normally-distributed symbols through a noisy channel

I have the following problem: I want to transmit a certain amount of information through a channel with Gaussian noise (random variable Z). The input distribution is likewise Gaussian (random variable ...
Christian W's user avatar
0 votes
0 answers
1k views

Silhouette Score with Noise (from DBSCAN)

I stumbled across this example on scikit-learn (1.2.0), where the silhouette score alongside some other metrics is computed for DBSCAN cluster assignments. These assignments include some Noise ...
MERose's user avatar
  • 419
2 votes
1 answer
828 views

Is it possible to regularize a covariance matrix?

I have many "parallel" time-series (about 100). They have relatively short history. I calculate a covariance matrix between these time-series. Now, I believe that the observed covariance ...
Roman's user avatar
  • 554
1 vote
0 answers
259 views

What is a mathematic rigorous definition of "blue noise"?

Let $d\in\mathbb N$, $I$ be a finite nonempty set, $(x_i)_{i\in I}\subseteq[0,1)^d$, $(w_i)_{i\in I}\subseteq[0,\infty)$ with $\sum_{i\in I}w_i=1$ and $$\sigma:=\sum_{i\in I}w_i\delta_{x_i}.$$ I ...
0xbadf00d's user avatar
  • 141
1 vote
1 answer
85 views

Why can't I detect concept drift with linear regression?

When I was first trying to detect concept drift, it seemed naively to me to be a problem of detecting whether noisy data was veering off horizontal trajectory (non-trivial slope above some arbitrary ...
Sanger Steel's user avatar
0 votes
0 answers
28 views

Prooving the convergence rate of noisy estimators (machine Learning)

I want to estimate a quantity and have two choices for estimators (they both sample from the same distribution). I suspect one of them is nosier and has a slower convergence rate. I want to ...
postnubilaphoebus's user avatar
3 votes
0 answers
52 views

Remove Additive Zero Mean Gaussian Noise

I have six values, and each value is corrupted by Additive-Zero-Mean-Gaussian Noise with var = 0.05). Each value is range from 0 to 1. Is there anyway for me to remove these additive noise?
wrek's user avatar
  • 315
4 votes
1 answer
507 views

Detecting peaks in noisy quasi-periodic time series

I have the following acceleration data taken from my phone when rowing The negative peak is the point at which the oars go in the water. I would like to be able to identify this peak in real time ...
Will P's user avatar
  • 43
5 votes
1 answer
515 views

XGBoost when P>>N

Someone built an XGBoost classification model using each pixel in an image (256*256) as a separate feature, plus a few other features. However they only have 500 data points. The target classes were ...
Alex's user avatar
  • 165
1 vote
1 answer
315 views

GAN artifacts on borders

not quite a math-question, but I have a doubt. I'm trying to build from scratch the Pix2pix network, on the facades dataset, and I think I finally got a good model (from the paper I borrowed just the ...
Alberto's user avatar
  • 859
3 votes
2 answers
206 views

What is the relationship between noise reduction and dimension reduction?

My understanding is that unsupervised methods like PCA, autoencoders and K-means shape a data space such that the modified representation of the data either nicely separates different families of data ...
Douw Marx's user avatar
0 votes
1 answer
61 views

Ways to detect noise in multi-class classification training data using text embeddings (BERT)

So I have a dataset with a column of text and and labels (5 different labels) associated with it. The labels describe the potential answer to the type of question being asked in the text column. For ...
user11715878's user avatar
1 vote
1 answer
100 views

Means to estimate the decay rate of a noisy decay process?

I have a decay process which appears essentially like $$ f(t) = \xi(t)\exp[-t/\tau],$$ where $\xi(t)$ is a stationary Gaussian noise with some mean, variance, and correlation function. Given a ...
kevinkayaks's user avatar
3 votes
1 answer
286 views

Adding noise to non-negative imputed data

I have an hourly time series of wind speed data that spans 8 years. Wind speed must be non-negative but there are only ~80 out of ~70000 values that are exactly 0; the remainder are either positive or ...
qdread's user avatar
  • 215
0 votes
1 answer
154 views

What is Hamming Window in Audio Analysis?

I am reading this paper. The paper writes ... an input waveform of t seconds is converted into a sequence of 128-D log mel filter bank features computed with 25 ms Hamming window every 10 s. The ...
onexpeters's user avatar
1 vote
0 answers
30 views

How to infer noise map from noisy data?

I am given noisy measurements of the solution of a PDE at different times, each measurement is drawn from a Gaussian distribution having as mean the solution of the PDE at that time and a standard ...
user360500's user avatar
0 votes
0 answers
80 views

Adding noise to a dataset to "hide" its distribution

Given a dataset that follows "some"[1] distribution, how can we add noise to the dataset, such that when the original dataset plus the noise is shown to someone, it looks uniformly random[2]?...
pranavJ's user avatar
1 vote
1 answer
64 views

Estimating the standard deviation

I have data of a function $f(t)$, where I always have just a single data point per $t$. It is known that the data is affected by noise, where the noise itself is also a function of $t$. Now I want to ...
Johny Dow's user avatar
2 votes
1 answer
291 views

What jitter value should I use for this scatter plot?

I plotted a scatterplot between humidity and temperature (air) in centigrade. I got the following graph; It is evident that the points fall in discrete columns. This might be because of the rounding ...
Ritik P. Nayak's user avatar
1 vote
1 answer
23 views

Training twice on non-injective data

I have a large dataset of 30000 points, but most my Ys are the same while all Xs are different. Ys are from different samples, so I had means of Y for each sample and I used means alongside Xs to ...
Shayan Kabiri's user avatar
0 votes
0 answers
286 views

What is the combined standard deviation of Poisson photon shot noise and Gaussian read noise?

On various image sensor web sites (e.g., https://www.thorlabs.com/newgrouppage9.cfm?objectgroup_id=10773, https://www.microscopyu.com/tutorials/ccd-signal-to-noise-ratio you'll see something to the ...
planetfolly's user avatar
1 vote
0 answers
54 views

Additive error model for non-linear case

I have checked other questions regarding additive noise model, but I could not convince myself for non-linear case. Assume the data vector $\mathbf{d}$ is described by a possibly non-linear ...
T. B.'s user avatar
  • 9
4 votes
2 answers
245 views

Does a targets-permutation test prove that regression find a real pattern?

I need to solve a standard ("vanilla") regression problem meaning that I have a 2D array of real-valued features (X) and 1D array of real-valued targets (<...
Roman's user avatar
  • 554

1
2 3 4 5 6