Questions tagged [filter]

The tag has no usage guidance.

Filter by
Sorted by
Tagged with
1
vote
0answers
51 views

Introduction to filtering - Kalman filtering

I would like to make a preamble and then introducing the Kalman filter in a work I am writing. In your opinion which are the topics I have to cite? Do you think I should cite Kolmogorov and Wiener ...
1
vote
0answers
37 views

Savitzky-Golay with switching between scales based on slope

I am having fun with Savitzky-Golay (aka H-P, W-H) and I am bumping into limits of the filter. It makes me want a better filter. Is there a filter that adjusts filter order based on "slope" but ...
1
vote
1answer
163 views

Why is it harder to obtain diagnostic knowledge $p(h\mid z)$ than causal knowledge $p(z\mid h)$? [closed]

Causal knowledge is $p(z\mid h)$, i.e. the probability that a certain state $h$ causes a certain state of the observable variable $z$. Diagnostic knowledge on the other hand refers to the knowledge we ...
2
votes
0answers
155 views

Deseasonalizing a time series using a Wiener-Kolmogorov filter

I am trying to eliminate seasonality from a time series using Wiener-Kolmogorov filter, I am following the methodology explained in here this paper about signal extraction which is the same followed ...
5
votes
1answer
2k views

Why taking the first difference is the same as an AR(1) filter?

Today, my professor said that for highly correlated time series, taking the first differentiation is like applying an AR(1) filter.. Unfortunately I a was not able to ask him after the lecture. I am ...
3
votes
2answers
247 views

How can I detect when a key was pressed with accelerometer or gyroscope data?

I have a dataset (~20k samples) of sensor data gathered from a smartphone. What I want to do with it is to detect those spikes you can see in the graphs below. They occur when the user presses a ...
2
votes
0answers
69 views

Noise estimation in LTE using bandpass filter

Can noise estimation in LTE be done using bandpass filter? As per my study in wireless systems to estimate noise power, if pilot sequence is known is done as |y(k)-p(k)h(k)|^2, where p(k) is pilot ...
2
votes
1answer
770 views

How can I filter out GPS “spidering” from tracks created when user is stationary?

We collect vehicle mileage data from GPS loggers (based on a 5 second sample of location) and need an accurate measure of the mileage travelled. However what's caught us out is that some vehicles ...
8
votes
2answers
3k views

Filtering vs Smoothing in Bayesian Estimation

I am facing a posterior distribution in a MCMC application that aims to sample an unobservable variable $x=\{x_t\}_{t=0}^{T}$ given an observed series $y=\{y_t\}^T_{t=0}$. However, the conditional ...
2
votes
1answer
2k views

Using Hodrick-Prescott filters for analyzing and forecasting business time series

I have many time series data at work. Some are annual, some are quarterly, and some are monthly. I am exploring Digital Signal Processing (DSP) as a complementary approach to ARIMA and other methods. ...
0
votes
1answer
435 views

Inverse of exponential smoothing

Suppose that a time series $s_t$ it is known to be obtained via exponential smoothing of an underlying signal $x_t$, that is $$ s_{0}= x_0 $$ and $$ s_{t} = (1-\alpha)\,x_t+\alpha\,s_{t-1}. $$ ...
1
vote
1answer
220 views

Fast Gaussian Filtering Using a Permutohedral Lattice

I'm trying to implement http://www.dabi.temple.edu/~zoran/papers/KostaAAAI13.pdf but am stack at understanding equations 10) and 11). They claim that the sum of the Gauss kernel multiplied with the ...
1
vote
0answers
815 views

Signal Processing Steps for Raw Accelerometer Data

A project I am engaged with involves taking raw accelerometer data (in g's ) and analyzing for the existence of tremors (the accelerometer is attached to an individuals hand). I am relatively new to ...
1
vote
0answers
121 views

Recursive Bayesian estimation for the coefficients of a convex combination

I'm given a sequential measurements of vectors $\vec{v}_t\in R^{K+1}$ such that $v_{t,0}$ is a convex combination of $\{v_{t,k}\}_{k\ge 1}$ (i.e. $v_{t.0}=\sum_{k\ge 1}{w_{t,k}v_{t,k}}$ for some ...
2
votes
1answer
47 views

5 low signal-to-noise measurements of same signal, know noise distribution and signal distribution. Recover?

I have 5 measurements of $x(t)$ at each instance in time, all contaminated with Gaussian noise. My signal-to-noise ratio is bad. \begin{align} y_1(t)=x(t)+n_1(t)\\ y_2(t)=x(t)+n_2(t)\\ y_3(t)=x(t)+...
2
votes
1answer
610 views

Savitzky-Golay (aka Hodrick–Prescot or Whittaker-Henderson) vs. Kernel

Is there a clear analytic link from Kernel smoothing, particularly the Nadaraya–Watson estimator, (S-G, H-P, or W-H) smoothing filter? The "filter" is called by different names in different fields ...
1
vote
0answers
64 views

How could I use a Bayes classifier to categorise emails?

I've been looking into using Bayes to classify incoming emails to one of several distinct "owners" (so more complex than a spam filter that only has two outcomes). I don't have a stats background, so ...
1
vote
2answers
2k views

Lag-free filter methods for time series

I'm currently working with accelerometer based raw data (100 hz). Now I want to low pass filter this timeseries of accelerations for further analyses. I tried different filters like the simple moving ...
0
votes
1answer
1k views

Applying exactly same WEKA filter on train and test data (What to use in setinputFormat traindata or test data) )

I am using WEKA for classification. I need to perform pre-processing before it. I want to do three thing , tf-idf conversion, normalization and discretization. But I want exactly same pre-processing ...
3
votes
2answers
257 views

AIC on Savitzky-Golay width

I want to use a Savitzky-Golay filter to smooth some data. There is a right width to use based on the data that it is smoothing. A number of papers basically use "eyeball norm" on the parameters but ...
1
vote
0answers
522 views

Low-pass filter on subset of data

I have a time series with 15-minute sampling frequency. When the data is greater than 0, there is a lot of noise, but when the data is less than 0, there is very little noise. I am trying to find a ...
1
vote
1answer
76 views

Is $E[1_A | \mathscr{F_t}] = 0 ~\text{or} ~ 1 \ \Rightarrow E[1_A | \mathscr{F_{s}}] = E[1_A | \mathscr{F_t}]$ is only almost surely?

Spin-off from my previous question: Prove/Disprove $E[1_A | \mathscr{F_t}] = 0 ~\text{or} ~ 1 \ \Rightarrow E[1_A | \mathscr{F_{s}}] = E[1_A | \mathscr{F_t}]$ Apparently the conclusion holds true ...
2
votes
2answers
175 views

Why does a probability of 0 or 1 remain unchanged with new information, intuitively?

Related to these questions: Prove/Disprove $E[1_A | \mathscr{F_t}] = 0 ~\text{or} ~ 1 \ \Rightarrow E[1_A | \mathscr{F_{s}}] = E[1_A | \mathscr{F_t}]$ Does an unconditional probability of 1 or 0 ...
1
vote
1answer
3k views

Outlier filtering in 2D data in python

I have following data given: My curve fits it acceptable for my needs. I use here 4th degree polynomial. (data is limited to 0-100 percent range for both axis!) What I want to try now is to filter ...
2
votes
0answers
780 views

Cross correlation between 2 filtered time series

I have 2 band pass filtered time series for 30-90 day band I would like to understand the lagged correlation between these 2 series in this band. The issue is that autocorrelations exist in both ...
0
votes
0answers
55 views

Residual plot looks like inverted change plot for prediction model

I have a very big problem with my predictive model. What i essentially do is that I predict the volume in a tank by studying the flow into and out of the tank. I use the two flows to construct a ...
10
votes
1answer
364 views

Prove/Disprove $E[1_A | \mathscr{F_t}] = 0 ~\text{or} ~ 1 \ \Rightarrow E[1_A | \mathscr{F_{s}}] = E[1_A | \mathscr{F_t}]$

Prove/Disprove $E[1_A | \mathscr{F_t}] = 0 \ \text{or} \ 1 \ \text{a.s.} \ \Rightarrow E[1_A | \mathscr{F_{s}}] = E[1_A | \mathscr{F_t}] \ \text{a.s.}$ Given a filtered probability space $(\Omega, \...
5
votes
0answers
689 views

creating random variable with certain auto-correlation in R

I want to create a random variable with a given autocorrelation in R. The target autocorrelation is defined by: $$acf_{target}=(lag+1)^{(-b)}$$ with $b=1.41519$ which I derived from a natural ...
2
votes
1answer
3k views

Use tm_filter to search for multiple words

I´m new to R, so please bear with me. So, I know I can use the following to search for a word in several documents. ...
3
votes
1answer
109 views

What is the limiting distribution of the Bayesian Filtering

I've got a question about the iterative Bayesian filtering, the general form of which is shown as follows: $P(x|z_0,...z_{k+1})\propto P(z_{n+1}|x)P(x|z_0,...,z_k),\,k=0,1,\dots$. $P(x|z_0)=P_0(x)$ ...
1
vote
0answers
25 views

Where can I find good references regarding to noise filtering and prediction in time series?

I want to model the error structure of every certain time period obtained from the past errors produced by the predictions of nonlinear time series. I would like to know if someone knows specialized ...
1
vote
2answers
71 views

Online learning that “forgets” older aspects learned? (short-term memory)

I am looking for an online learning classifier that is highly adaptable and has only short-term memory. I need such a think in a object tracking system with high-dimensional feature vectors. Maybe a ...
1
vote
0answers
470 views

First order condition of HP Filter

The HP Filters for growth and cyclical components is written as: $$\min_{g_t}\sum_t \left[(y_t-g_t)^2+\lambda\left[(g_{t+1}-g_t)-(g_t-g_{t-1})\right]^2\right].$$ Hodrick and Prescott, on their paper ...
2
votes
0answers
82 views

Hodrick-Prescott Filter, Time Series, SDE, and Ito Isometry

The background of this question is a paper written by Morten O.Ravn and Harald Uhlig, titled "On Adjusting The Hodrick-Prescott Filter For The Frequency of Observations" Consider the decomposition of ...
0
votes
1answer
694 views

Smoothing a particle filtered 2D trajectory

I am currently developing a very basic particle filter for a 2D robot localization task. My process is defined by a really simple velocity / steering angle based motion model. I am re-weighting the ...
2
votes
1answer
1k views

Linear discriminant analysis (Fisher) = Bayes?

I'd like to ask a question, I am reading book right now about mail filtering, both methods: naïve Bayes and Fisher are there very similar in implementation. I am also writing a paper on Bayesian spam ...
0
votes
0answers
358 views

Find repetitive patterns in matrices below

How can I identify the repetitive patterns from the matrices below? My problem is that the patterns in the matrix are different from matrix to matrix (dependent on the input data). I need some machine ...
5
votes
1answer
6k views

Beginner level: Help in learning Kalman Smoother (Part 1) [closed]

Parameter estimation of Linear Dynamical system is a tutorial which explains Kalman Filter, Smoothing, and Expectation Maximization. I have followed the derivation for Kalman Filter. But cannot ...
1
vote
0answers
222 views

Filtering noise in a data set

I'm trying to filter out noise from a Sonar. The idea is that the sonar in aimed upwards, after an object has come in it's range, I want to be able to tell if the object is moving away or coming ...
4
votes
1answer
373 views

Deriving the Bayes Filter Correction Equation

The correction rule for Bayes filters is: $$p\left(x_{k}|D_{k}\right)=\dfrac{p\left(y_{k}|x_{k}\right)\cdot p\left(x_{k}|D_{k-1}\right)}{p\left(y_{k}|D_{k-1}\right)} $$ For: State at time $k$ is $...
6
votes
1answer
687 views

How does backpropagation learn convolution filters?

I've understood how the backpropagation algorithm uses the partial derivatives of the weights to train a normal neural network. However, I cannot quite understand how the algorithm changes the filters....
1
vote
1answer
85k views

Filtering a dataframe in R based on multiple Conditions [closed]

I am new to using R. I am trying to figure out how to create a df from an existing df that excludes specific participants. For example I am looking to exclude Women over 40 with high bp. I have ...
1
vote
0answers
401 views

How can I smooth a set of discrete data points for the purpose of schedule planning?

Disclaimer: I do not have a background in statistics or the math behind filtering, save one long-time-ago college course. I have a well defined problem space. I am calculating hourly staffing ...
0
votes
1answer
371 views

What statistical method to correct systematic error in the output of a economic optimization model?

I am working with an economic optimization model which attempts to model the dynamics of a certain commodity market (prices, quantities, production etc.) for different frequencies (monthly, quarterly, ...
1
vote
1answer
1k views

Filtering using a SARIMA model in R

I am not an expert in statistics, but I would like to work on a SARIMAX model representing power consumption. The exogeneous variable would be the temperature, but for now I found here I might need to ...
0
votes
1answer
625 views

Heteroscedasticity filter for time series

I am looking for a method or package in R that can remove heteroscedasticity from time series. Specifically, I have a number of time series to which I want to fit a VAR model. Each time series may or ...
2
votes
1answer
417 views

What does “AR(p) filtered series” mean?

I guess this means that omitting some variables in a certain interval, say, $(x_1, x_2, x_3, x_4, x_5) \to (x_1, x_5)$ in AR(4) model. Is it right? Or does this means eliminating autocorrelations ...
1
vote
2answers
218 views

Does filtering of data by effect size violate some assumption of P value adjustment methods?

I have pre- and post-treatment continuous data for a large number of variables that I am analyzing for treatment effect. Normally I would obtain the P values and then adjust them for multiple testing ...
0
votes
1answer
156 views

Content-control (web filtering) using machine learning

I'm trying to build a content-control (web filtering) application using machine learning (just for training purposes). For example define gaming sites. I'm somewhat familiar with machine learning ...
3
votes
1answer
2k views

What is a density function?

I know about histograms and also know that if we connect the mid-points on the top of bars in a histogram we will get a frequency polygon. This polygon could then be 'smoothed' in a way that it ...