Skip to main content

Questions tagged [filter]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
0 votes
0 answers
17 views

remove "non stable" occurences in a time series

I have a signal, corresponding to a time series, in which I want to identify "outliers", actually corresponding to samples for which the signal is not "sufficiently stable" (see ...
MysteryGuy's user avatar
0 votes
0 answers
17 views

Remove non repeatable stochastic noise from an image

Assume that you have these coordinates inside an image. The algorithm for creating these crosses comes from FAST-algorithm for corner detection. But the problem with FAST-algorithm is that some of ...
euraad's user avatar
  • 425
0 votes
0 answers
68 views

Using Bayesian statistics in time series forecasting

I would like to forecast demand count time series of taxi fleets at different locations on the map at different points in time. I.e. multivariate demand Time series forecasting. Given hierarchinal ...
Jose_Peeterson's user avatar
0 votes
0 answers
104 views

Hodrick and Prescott (HP) filter in R

I'm applying the HP filter to time series data to remove the cyclical component of a time series (TS) from raw data. Data do have a yearly frequency. I write in R: ...
Maximilian's user avatar
0 votes
0 answers
24 views

Search strategy clarification for a systematic scoping review on existing statistical gene-based prediction methods

In a systematic scoping review, I use the following databases: MEDLINE, Embase, Scopus and Web of Science. In MEDLINE and Embase, I use MeSH terms only to exclude animal and plant studies, but not ...
Dovini Jayasinghe's user avatar
0 votes
0 answers
31 views

Wiener filter coefficients for time series

I am tasked to find the wiener coefficients for a time series forecasting dataset with daily seasonality. I split the dataset into 24hour segments an find the wiener coefficients between day N and day ...
Creative T's user avatar
2 votes
0 answers
20 views

Is it possible to define a specific criterion prior to imputing missing values? [closed]

it's hard to summarise the problem in a question so I'll explain it here. I have a variable called "Plant Species" that is missing a lot of values because of the sampling method used. For ...
DeeDee's user avatar
  • 21
1 vote
0 answers
121 views

Inferring a random walk from noisy "images"

I'm interested in the following inference / filtering problem in a hidden Markov model setting. Suppose we have a simple random walk $x_t\in\mathbb{Z}$ and observations are "images" ...
Austen's user avatar
  • 111
0 votes
1 answer
30 views

What is the point of the re-sampling step in a particle filter?

My general understanding of particle filters is that you represent your state as a collection of discrete particles which you then transform using your state propagation equation. What I don't ...
FourierFlux's user avatar
0 votes
1 answer
52 views

Do we need to propagate state covariance matrix 'P' during missing observations in the Extended Kalman Filter?

Basically as the title says: in a scenario, we have missing observations where the entire state vector is unknown for consecutive time steps. Do we just run through the prediction section of the ...
user383687's user avatar
6 votes
1 answer
34 views

Would fitting / filtering result in a loss of information?

I have a very general (and probably also quite naive) question: Let's say I have arbitrary data points in {x, y], when I plot all these and fit and/or filter them (...
Ben's user avatar
  • 3,523
0 votes
1 answer
31 views

I can't understand how this function works (linear filter with a filter kernel)

on page 13 of the Dayan and Abbott book onTheoretical Neuroscience, there is this formula $$r_{approx}(t)= \int\limits_{-\infty}^\infty{d\tau w(\tau) \rho(t-\tau)}$$ Let's assume that $w(t)$ is a ...
Vaaal's user avatar
  • 587
0 votes
1 answer
116 views

Approximating a 1-d Kalman Filter with non-Gaussian Observation Noise

I'm looking for a Bayesian filter where observations are generated according to $s_t = \gamma s_{t-1} + w_p$ and $w_p \sim Normal(0, \sigma_p^2)$. Both $\gamma$ and the variance of the process noise $\...
ratatosk's user avatar
-5 votes
1 answer
112 views

Wait so probabilities of 0 or 1 CAN change? $P(A|B)=1$ does not imply $P(A)=1$ because $0 < P(A=B) < 1$?

Nassim Nicholas Taleb says here no probability that is 0 or 1 should ever change. Despite these 6 questions Does an unconditional probability of 1 or 0 imply a conditional probability of 1 or 0 if ...
BCLC's user avatar
  • 2,444
1 vote
0 answers
58 views

Guessing filters from responses to step signals

Consider a signal $X$ filtered by a kernel $p$ with finite support $[t_0,t_1]$ and $\int_{t_0}^{t_1}p(t)\,\text{d}t = 1$, yielding the response function $$\overline{X}(T) = \int_{t_0}^{t_1} X(T + t)\ ...
Hans-Peter Stricker's user avatar
0 votes
1 answer
143 views

Setting the observation likelihood threshold for outlier detection if you know know the percentage of outliers

Let's assume I have a sensor that gives me measurements $z$ and I know that $50\%$ of the measurements I read are outliers (more than 3 standard deviations away from the real measurement distribution)....
MattSt's user avatar
  • 350
0 votes
1 answer
266 views

Smoothing sensor signals with response times - how to

I have a Sensor (e. g. for temperature) that has a response time ($t_{99}$ -> this is the time that the sensor needs to give an output of 99 % of the actual value) of let's say 10 seconds. This ...
stats_ts's user avatar
0 votes
0 answers
316 views

When is it acceptable to exclude data from analysis?

I have Alcohol, Tobacco, and Firearms (ATF) trace data that lists categories/offenses related to when a firearm was recovered by law enforcement. However, in many cases there are offenses that are ...
James R.'s user avatar
1 vote
1 answer
37 views

Estimate the Image Using Multi Many Realizations of Its Convolution with a Known Filters Using Wiener Filter

Suppose we have a corrupted image $Y = H*X + \epsilon$ that is formed by taking an image $X$, convolving it with a point-spread function $H$, and adding gaussian noise $\epsilon$. Then we know that ...
Sunay Joshi's user avatar
1 vote
0 answers
30 views

What is the significance behind having small kernel sizes over having one large kernel size that covers the entire input in a CNN?

I have hardly ever seen anyone cover the entire input image with a filter of the same dimensions. I was wondering why that is the case, and if the performance in say, an image detection application ...
hridayns's user avatar
  • 113
0 votes
0 answers
34 views

In stochastic filtering, can observations depend on lagged observation values?

Say I have a latent state like $$ dX_t = dW_t $$ and observations like $$ dY_t = f(X_t - Y_t)dt + dZ_t $$ Can I get filtering estimates of $X_t$ using a standard Kalman filter framework despite the ...
robsmith11's user avatar
0 votes
0 answers
46 views

How can a stochastic filter be forecasted?

After the covid-19 lockdown, employment (and other economic indicators) were shocked. I want to model a recovery in employment levels. I used an AR(2) model of quarter-on-quarter (q/q) employment ...
ahorn's user avatar
  • 306
2 votes
2 answers
919 views

What is a "filter" and what does "filtering" mean in statistics/engineering/computer science?

I see the term "filter" in many neuroscience papers including those with heavy statistical content ("spatial filter", "temporal filter", etc.), as well as those with ...
Sia's user avatar
  • 139
0 votes
0 answers
251 views

Help with time series comparisons using periodograms

I have a dataset consisting of time series signals of different lengths obtained from different groups of patients. I am trying to understand the commonalities of the time series of each group. ...
Merry's user avatar
  • 255
1 vote
1 answer
257 views

Is there a way to filter a heatmap based on relevance?

I'm currently working with ComplexHeatmap and a very large dataset from RNAseq (~13,000 genes/columns). The heatmap output (based off of clustering) at the moment contains so many columns it's ...
user avatar
5 votes
2 answers
115 views

Seeking recommended literature search terms for a solution to a specific kind of data structure?

Hopefully this isn't considered too off-topic. I'm working in industry these days and came up with a solution to an analysis problem we'd been facing. I'd like to get a sense as to whether said ...
Mike Lawrence's user avatar
1 vote
0 answers
135 views

What is wrong with my approach on a custom way of creating Gabor-filter convolution kernels?

Disclosure: I am not a prominent mathematician (current bachelor student) like others on this website and my approach has been mostly pragmatic. Please do tell me if I can improve the formulation of ...
G.S. Luimstra's user avatar
2 votes
0 answers
81 views

How does choosing an even window size actually add a cyclical component to the model?

I am new to Time Series Analysis. Say, we have a time series $(y_{t})_{t}$ that we want to filter with a moving average filter. I have been told that we should choose the window size $L$ of the filter ...
MinaThuma's user avatar
  • 139
1 vote
1 answer
139 views

Why "run the filter longer than needed and remove the initial values" will solve the issue of recursive solving equations?

Consider sequence of random variables $w_i$ iid normal(0,1). Given the equation, $x_t=x_{t-1}-0.9x_{t-2}+w_t$ with $t$ discrete, I want to solve for $x_t$ recursively by prescribing $x_1,x_2$. The ...
user45765's user avatar
  • 1,445
1 vote
0 answers
46 views

Does feature detector(filter) has to be a sqaure matrix?

I am going through a course on Convolutional neural networks, where in the convolution step, the feature detector matrix was square shaped. Is there any mathematical significance that Feature ...
Nancy's user avatar
  • 33
1 vote
1 answer
182 views

How are the values of the Grubbs table calculated for use in the Grubbs filter?

I am looking at Grubbs test for outliers. The approach seems simple enough but for completeness I have two questions. The Grubbs test is defined as $$G_{\rm{test}} = \frac{\left| x_{i} - \bar{x} \...
user27119's user avatar
  • 328
1 vote
2 answers
246 views

How can a given conv neural net layer handle filters of different size?

traditional method is to use multiple filters of same dimensions but with different weights and stack the output (basically concatenate them) that is then to be fed into the next conv layer. If I ...
Kevin Kim's user avatar
0 votes
1 answer
400 views

Time series with long "idle" periods - is it safe to eliminate those periods?

Suppose that I have as input a time series $T = \{ t_1, t_2, ..., t_M \}$ where each point is sampled at a fixed time interval (e.g. every 10 ms). The problem is that $T$ contains a lot of periods ...
Elise Le's user avatar
1 vote
0 answers
79 views

recommendation based on multiple user

I am learning about the recommendation system. How can I make a system where it takes multiple users as input and based on the rating and another attribute it gives recommendation? I have data sets ...
TheTechGuy's user avatar
0 votes
1 answer
28 views

Need precision about an example in a book about bayesian filters

My question is about the example here : https://github.com/w407022008/Kalman-and-Bayesian-Filters-in-Python/blob/master/02-Discrete-Bayes.ipynb#Adding-Uncertainty-to-the-Prediction paragraph : ...
hl037_'s user avatar
  • 101
2 votes
2 answers
326 views

Do earlier hidden layers learn more concepts/features than later ones, in neural networks?

I am wondering whether there is a general statement of the sort "earlier layers in neural networks learn more concepts/features than later layers" or the other way around. The output layer not being ...
Tom's user avatar
  • 1,373
1 vote
1 answer
112 views

Small noise of state process and filtering

Assume we have a linear state-space model: $$ z_{k} = Hx_{k} + v_{k}\\ x_{k} = F x_{k-1}+ w_{k}. $$ We are interested in filtering, i.e. we aim to estimate $E[x_{n}|z_{0}, \dots, z_{n}]$. If the ...
ABK's user avatar
  • 676
1 vote
0 answers
120 views

Effect of log transformation or standardization of a regressor in the filtering step

We are working with a dataset that has hundreds of biomarkers (many of which are correlated) and often they have many missing values. Our initial goal was to use an elastic net but that would require ...
Blain Waan's user avatar
  • 3,605
1 vote
1 answer
402 views

Optimality of Bayesian filtering

In Kalman filter, we can show it's a minimum variance filter, which I believe is due to the linearity of system and the Gaussianity of noise. It comes to me that what is the optimality criterion used ...
shionlau's user avatar
1 vote
0 answers
2k views

Kalman filter for AR(1) plus noise

I am working the following AR(1) plus noise state-space model $$ z_{t} = x_{t} + v_{t}\\ x_{t} = \phi x_{t-1} + c + w_{t} $$ Therefore, the transition matrix is $[\phi]$, the observation matrix is $[1]...
ABK's user avatar
  • 676
1 vote
0 answers
626 views

Exponential moving average before computing std

In which cases it would make sense to use exponentially weighted moving average (EWMA) before, for example, computing sample variance or other statistical analysis? Could you give an example when one ...
ABK's user avatar
  • 676
2 votes
0 answers
45 views

Signal Decomposition

I have two time dependent signal sources X & Y. Both can be modeled as having a linear combination of time dependent individual components and common components; so X(t)=a(t)+C(t)+noise, Y(t)=b(t)+...
Y. Reznichenko's user avatar
1 vote
1 answer
28 views

Filter out linearly interpolated historical data points

I am reading in historical sensor data from a plant. I found out that there are intermittent periods where between time t1 and time t2, the data points are linearly interpolated. I came to know, that ...
chupa_kabra's user avatar
0 votes
1 answer
2k views

What is the difference between 1x1 convolutions and convolutions with "SAME" padding?

In general, 1x1 convolutions are used to reduce the dimensionality of filter space. I referred this answer. But we can also reduce the dimensionality of filter ...
Kaushal28's user avatar
  • 191
0 votes
1 answer
2k views

How can we determine the appropriate number of hidden layers, kernels in convolutional neural network (CNN)?

I have checked a lot of questions here and in other websites. What I concluded is that there is no rules for choosing the right number of hyper-parameters in CNN, all what can we do is just trying ...
Khalil Meg's user avatar
0 votes
0 answers
202 views

Noise reduction with known noise distribution

I have a time signal with a known noise distribution parameters (gaussian, sd is known). I would like to estimate the true value statistically and in the best case obtain a confidence interval. As I ...
MikeHuber's user avatar
  • 1,239
1 vote
0 answers
47 views

Time series filtering notation

I am looking at some suggested filters for tidal data but am having trouble understanding the notation. For example, Godin (1972) suggests a low-pass filter for tidal data that is a combination of 24-...
mikeck's user avatar
  • 205
1 vote
0 answers
279 views

Is there a filter function that performs similar to moving average but does not loose data?

I need to smooth a time series using a low pass filter. A simple moving average is working fine for me, however, using a moving average causes an inevitable loss of data a the beginning of the ...
mickkk's user avatar
  • 949
1 vote
1 answer
97 views

Partial spline. Reference [closed]

I have a well done and perfectly working protocol to smooth my experimental data. I do the following: I have a variable of size 1000. Iteratively I choose random 100 points and spline them using the ...
zlon's user avatar
  • 718
2 votes
0 answers
233 views

Faster RCNN - Pyramid of Filters vs Pyramid of Anchors (Reference Boxes)

I'm reading faster RCNN paper now and trying to understand what is the difference between Pyramid of Filters and Pyramid of Anchors methods from the scale point of view. I mean if I use only one ...
ChaosPredictor's user avatar