Questions tagged [filter]
The filter tag has no usage guidance.
140
questions
0
votes
0
answers
28
views
Using Bayesian statistics in time series forecasting
I would like to forecast demand count time series of taxi fleets at different locations on the map at different points in time. I.e. multivariate demand Time series forecasting.
Given hierarchinal ...
- 244
0
votes
0
answers
31
views
Hodrick and Prescott (HP) filter in R
I'm applying the HP filter to time series data to remove the cyclical component of a time series (TS) from raw data. Data do have a yearly frequency.
I write in R:
...
- 333
0
votes
0
answers
15
views
Search strategy clarification for a systematic scoping review on existing statistical gene-based prediction methods
In a systematic scoping review, I use the following databases: MEDLINE, Embase, Scopus and Web of Science. In MEDLINE and Embase, I use MeSH terms only to exclude animal and plant studies, but not ...
0
votes
0
answers
12
views
Wiener filter coefficients for time series
I am tasked to find the wiener coefficients for a time series forecasting dataset with daily seasonality. I split the dataset into 24hour segments an find the wiener coefficients between day N and day ...
2
votes
0
answers
20
views
Is it possible to define a specific criterion prior to imputing missing values? [closed]
it's hard to summarise the problem in a question so I'll explain it here.
I have a variable called "Plant Species" that is missing a lot of values because of the sampling method used. For ...
- 21
1
vote
0
answers
107
views
Inferring a random walk from noisy "images"
I'm interested in the following inference / filtering problem in a hidden Markov model setting. Suppose we have a simple random walk $x_t\in\mathbb{Z}$ and observations are "images" ...
- 111
0
votes
1
answer
25
views
What is the point of the re-sampling step in a particle filter?
My general understanding of particle filters is that you represent your state as a collection of discrete particles which you then transform using your state propagation equation.
What I don't ...
- 151
0
votes
0
answers
12
views
How do you automatically check for feature columns like ID?
I'm working on a fraud dataset that comes with a column for ID. I can remove this column manually, but plotting its distribution made me wonder if there's an automatic way to remove a column like this....
- 369
0
votes
1
answer
35
views
Do we need to propagate state covariance matrix 'P' during missing observations in the Extended Kalman Filter?
Basically as the title says: in a scenario, we have missing observations where the entire state vector is unknown for consecutive time steps. Do we just run through the prediction section of the ...
2
votes
1
answer
26
views
Would fitting / filtering result in a loss of information?
I have a very general (and probably also quite naive) question: Let's say I have arbitrary data points in {x, y], when I plot all these and fit and/or filter them (...
- 3,123
0
votes
1
answer
27
views
I can't understand how this function works (linear filter with a filter kernel)
on page 13 of the Dayan and Abbott book onTheoretical Neuroscience, there is this formula
$$r_{approx}(t)= \int\limits_{-\infty}^\infty{d\tau w(\tau) \rho(t-\tau)}$$
Let's assume that $w(t)$ is a ...
- 547
0
votes
0
answers
25
views
How to determine periodicity (autocorrelation) in the 2d projection of a 3D helical trajectory
Some microscopic objects are helically moving in 3d and we acquisite their moving trajectories as a set of 2d points. The obtained trajectories are shown on the image.
For each of these trajectories ...
- 101
0
votes
0
answers
52
views
How to mitigate spatially correlated noise in image
I was wondering how can I mitigate spatially correlated noise following power law with filtering or other techniques?
For instance for astrophsics you can simulate it
...
0
votes
0
answers
18
views
Probability that a linear filtration of a stochastic process exceeds a value for a given number of consecutive samples
I am trying to work out how to calculate the probability that a weighted sum of IID random variables will remain above a given threshold for a particular amount of time ie;
Given a collection of IID ...
- 1
0
votes
0
answers
58
views
How to filter or otherwise remove periodic noise from an image?
We are taking mostly-empty images of a system with a slow, low-noise CCD. Many of our images look like the following, where the object of interest is the bright streak, while a clear repeating pattern ...
- 81
0
votes
1
answer
80
views
Approximating a 1-d Kalman Filter with non-Gaussian Observation Noise
I'm looking for a Bayesian filter where observations are generated according to $s_t = \gamma s_{t-1} + w_p$ and $w_p \sim Normal(0, \sigma_p^2)$. Both $\gamma$ and the variance of the process noise $\...
- 13
-5
votes
1
answer
100
views
Wait so probabilities of 0 or 1 CAN change? $P(A|B)=1$ does not imply $P(A)=1$ because $0 < P(A=B) < 1$?
Nassim Nicholas Taleb says here
no probability that is 0 or 1 should ever change.
Despite these 6 questions
Does an unconditional probability of 1 or 0 imply a conditional probability of 1 or 0 if ...
- 2,394
1
vote
0
answers
57
views
Guessing filters from responses to step signals
Consider a signal $X$ filtered by a kernel $p$ with finite support $[t_0,t_1]$ and $\int_{t_0}^{t_1}p(t)\,\text{d}t = 1$, yielding the response function
$$\overline{X}(T) = \int_{t_0}^{t_1} X(T + t)\ ...
0
votes
1
answer
133
views
Setting the observation likelihood threshold for outlier detection if you know know the percentage of outliers
Let's assume I have a sensor that gives me measurements $z$ and I know that $50\%$ of the measurements I read are outliers (more than 3 standard deviations away from the real measurement distribution)....
- 350
0
votes
1
answer
223
views
Smoothing sensor signals with response times - how to
I have a Sensor (e. g. for temperature) that has a response time ($t_{99}$ -> this is the time that the sensor needs to give an output of 99 % of the actual value) of let's say 10 seconds. This ...
- 3
0
votes
0
answers
166
views
When is it acceptable to exclude data from analysis?
I have Alcohol, Tobacco, and Firearms (ATF) trace data that lists categories/offenses related to when a firearm was recovered by law enforcement. However, in many cases there are offenses that are ...
- 1
1
vote
1
answer
33
views
Estimate the Image Using Multi Many Realizations of Its Convolution with a Known Filters Using Wiener Filter
Suppose we have a corrupted image $Y = H*X + \epsilon$ that is formed by taking an image $X$, convolving it with a point-spread function $H$, and adding gaussian noise $\epsilon$. Then we know that ...
- 13
1
vote
0
answers
28
views
What is the significance behind having small kernel sizes over having one large kernel size that covers the entire input in a CNN?
I have hardly ever seen anyone cover the entire input image with a filter of the same dimensions. I was wondering why that is the case, and if the performance in say, an image detection application ...
- 113
0
votes
0
answers
33
views
In stochastic filtering, can observations depend on lagged observation values?
Say I have a latent state like
$$ dX_t = dW_t $$
and observations like
$$ dY_t = f(X_t - Y_t)dt + dZ_t $$
Can I get filtering estimates of $X_t$ using a standard Kalman filter framework despite the ...
- 172
0
votes
0
answers
43
views
How can a stochastic filter be forecasted?
After the covid-19 lockdown, employment (and other economic indicators) were shocked. I want to model a recovery in employment levels. I used an AR(2) model of quarter-on-quarter (q/q) employment ...
- 226
2
votes
2
answers
618
views
What is a "filter" and what does "filtering" mean in statistics/engineering/computer science?
I see the term "filter" in many neuroscience papers including those with heavy statistical content ("spatial filter", "temporal filter", etc.), as well as those with ...
- 139
0
votes
0
answers
215
views
Help with time series comparisons using periodograms
I have a dataset consisting of time series signals of different lengths obtained from different groups of patients. I am trying to understand the commonalities of the time series of each group. ...
- 163
1
vote
1
answer
218
views
Is there a way to filter a heatmap based on relevance?
I'm currently working with ComplexHeatmap and a very large dataset from RNAseq (~13,000 genes/columns). The heatmap output (based off of clustering) at the moment contains so many columns it's ...
Flora M
5
votes
2
answers
112
views
Seeking recommended literature search terms for a solution to a specific kind of data structure?
Hopefully this isn't considered too off-topic. I'm working in industry these days and came up with a solution to an analysis problem we'd been facing. I'd like to get a sense as to whether said ...
- 13.2k
1
vote
0
answers
122
views
What is wrong with my approach on a custom way of creating Gabor-filter convolution kernels?
Disclosure: I am not a prominent mathematician (current bachelor student) like others on this website and my approach has been mostly pragmatic. Please do tell me if I can improve the formulation of ...
2
votes
0
answers
64
views
How does choosing an even window size actually add a cyclical component to the model?
I am new to Time Series Analysis. Say, we have a time series $(y_{t})_{t}$ that we want to filter with a moving average filter. I have been told that we should choose the window size $L$ of the filter ...
- 139
1
vote
1
answer
114
views
Why "run the filter longer than needed and remove the initial values" will solve the issue of recursive solving equations?
Consider sequence of random variables $w_i$ iid normal(0,1). Given the equation, $x_t=x_{t-1}-0.9x_{t-2}+w_t$ with $t$ discrete, I want to solve for $x_t$ recursively by prescribing $x_1,x_2$. The ...
- 1,365
1
vote
0
answers
38
views
Does feature detector(filter) has to be a sqaure matrix?
I am going through a course on Convolutional neural networks, where in the convolution step, the feature detector matrix was square shaped.
Is there any mathematical significance that Feature ...
- 33
1
vote
1
answer
148
views
How are the values of the Grubbs table calculated for use in the Grubbs filter?
I am looking at Grubbs test for outliers. The approach seems simple enough but for completeness I have two questions.
The Grubbs test is defined as
$$G_{\rm{test}} = \frac{\left| x_{i} - \bar{x} \...
- 308
1
vote
2
answers
127
views
How can a given conv neural net layer handle filters of different size?
traditional method is to use multiple filters of same dimensions but with different weights and stack the output (basically concatenate them) that is then to be fed into the next conv layer.
If I ...
- 21
0
votes
1
answer
230
views
Time series with long "idle" periods - is it safe to eliminate those periods?
Suppose that I have as input a time series $T = \{ t_1, t_2, ..., t_M \}$ where each point is sampled at a fixed time interval (e.g. every 10 ms).
The problem is that $T$ contains a lot of periods ...
- 87
1
vote
0
answers
67
views
recommendation based on multiple user
I am learning about the recommendation system. How can I make a system where it takes multiple users as input and based on the rating and another attribute it gives recommendation? I have data sets ...
- 111
0
votes
1
answer
28
views
Need precision about an example in a book about bayesian filters
My question is about the example here : https://github.com/w407022008/Kalman-and-Bayesian-Filters-in-Python/blob/master/02-Discrete-Bayes.ipynb#Adding-Uncertainty-to-the-Prediction paragraph : ...
- 101
2
votes
2
answers
198
views
Do earlier hidden layers learn more concepts/features than later ones, in neural networks?
I am wondering whether there is a general statement of the sort "earlier layers in neural networks learn more concepts/features than later layers" or the other way around.
The output layer not being ...
- 1,304
1
vote
1
answer
103
views
Small noise of state process and filtering
Assume we have a linear state-space model:
$$
z_{k} = Hx_{k} + v_{k}\\
x_{k} = F x_{k-1}+ w_{k}.
$$
We are interested in filtering, i.e. we aim to estimate $E[x_{n}|z_{0}, \dots, z_{n}]$. If the ...
- 426
1
vote
0
answers
109
views
Effect of log transformation or standardization of a regressor in the filtering step
We are working with a dataset that has hundreds of biomarkers (many of which are correlated) and often they have many missing values. Our initial goal was to use an elastic net but that would require ...
- 3,515
1
vote
1
answer
356
views
Optimality of Bayesian filtering
In Kalman filter, we can show it's a minimum variance filter, which I believe is due to the linearity of system and the Gaussianity of noise. It comes to me that what is the optimality criterion used ...
- 53
1
vote
0
answers
2k
views
Kalman filter for AR(1) plus noise
I am working the following AR(1) plus noise state-space model
$$
z_{t} = x_{t} + v_{t}\\
x_{t} = \phi x_{t-1} + c + w_{t}
$$
Therefore, the transition matrix is $[\phi]$, the observation matrix is $[1]...
- 426
1
vote
0
answers
530
views
Exponential moving average before computing std
In which cases it would make sense to use exponentially weighted moving average (EWMA) before, for example, computing sample variance or other statistical analysis? Could you give an example when one ...
- 426
2
votes
0
answers
43
views
Signal Decomposition
I have two time dependent signal sources X & Y. Both can be modeled as having a linear combination of time dependent individual components and common components; so X(t)=a(t)+C(t)+noise, Y(t)=b(t)+...
1
vote
1
answer
26
views
Filter out linearly interpolated historical data points
I am reading in historical sensor data from a plant. I found out that there are intermittent periods where between time t1 and time t2, the data points are linearly interpolated. I came to know, that ...
- 165
0
votes
1
answer
2k
views
What is the difference between 1x1 convolutions and convolutions with "SAME" padding?
In general, 1x1 convolutions are used to reduce the dimensionality of filter space. I referred this answer.
But we can also reduce the dimensionality of filter ...
- 191
0
votes
1
answer
1k
views
How can we determine the appropriate number of hidden layers, kernels in convolutional neural network (CNN)?
I have checked a lot of questions here and in other websites. What I concluded is that there is no rules for choosing the right number of hyper-parameters in CNN, all what can we do is just trying ...
- 121
0
votes
0
answers
181
views
Noise reduction with known noise distribution
I have a time signal with a known noise distribution parameters (gaussian, sd is known). I would like to estimate the true value statistically and in the best case obtain a confidence interval.
As I ...
- 1,229
1
vote
0
answers
41
views
Time series filtering notation
I am looking at some suggested filters for tidal data but am having trouble understanding the notation. For example, Godin (1972) suggests a low-pass filter for tidal data that is a combination of 24-...
- 205