The tag has no usage guidance.

learn more… | top users | synonyms

0
votes
1answer
18 views

Filtering data depending upon conditions in 8-9 columns and then applying regression on the filtered data [on hold]

I am new to R programming and need to analyse a very large set of data, I have around 660 rows to be analysed and 9-10 columns.For each row I have values in the columns like (0,1,2,3,4,5) I need to ...
0
votes
0answers
11 views

Filtering an ARMA-GARCH on residuals

I had previously fitted my time series with different ARMA-GARCH models using the "rugarch" package"in R. I found no correlation and no heteroscedasticity in the residuals. I obtained (through Pair ...
0
votes
0answers
5 views

How is likelihood determined in this Discrete Bayes Filter?

I am following along in this book - Kalman-and-Bayesian-Filters-in-Python And in chapter 2 I'm trying to fully understand what is happening and I seem to be stuck on on this "likelihood". Lets ...
0
votes
0answers
15 views

What does mean by the number of pixel positions in CNN

I am doing project in machine learning using deep CNN. I need to understand how to choose hyperparameters (number of filter, shape of filter, max pooling shaep,..). I am using image of 42*42 (...
1
vote
2answers
71 views

Lag-free filter methods for time series

I'm currently working with accelerometer based raw data (100 hz). Now I want to low pass filter this timeseries of accelerations for further analyses. I tried different filters like the simple moving ...
0
votes
1answer
34 views

Applying exactly same WEKA filter on train and test data (What to use in setinputFormat traindata or test data) )

I am using WEKA for classification. I need to perform pre-processing before it. I want to do three thing , tf-idf conversion, normalization and discretization. But I want exactly same pre-processing ...
0
votes
0answers
13 views

Filter Feature Selection approaches for continuous variables?

I've noticed that correlation-based filtering for selecting features in high dimensional data require discretization of continuous variables, like e.g. Fast Correlation-based Filtering or regular CFS. ...
1
vote
1answer
33 views

AIC on Savitzky-Golay width

I want to use a Savitzky-Golay filter to smooth some data. There is a right width to use based on the data that it is smoothing. A number of papers basically use "eyeball norm" on the parameters but ...
0
votes
0answers
29 views

Help with PCA filter in Weka

I have questions about the PCA filter . Question 1: How can I improve the speed of filtering PCA? My base has 7200 attributes, but takes many hours to complete the process. 2nd Question: After I do ...
1
vote
0answers
49 views

Low-pass filter on subset of data

I have a time series with 15-minute sampling frequency. When the data is greater than 0, there is a lot of noise, but when the data is less than 0, there is very little noise. I am trying to find a ...
0
votes
0answers
8 views

Meaningful filtration of small values in approx. power law distribution

This is most likely elementary question. I observe distribution of values: Are there any general methods that could take only meaningful observations from this (i.e. only large enough)? So far I ...
0
votes
0answers
11 views

Find high occurring low values

In numerical data, where outlier is defined not by infrequency but value (this is a real-world data-set where low values would be indicative of noise). Although this is real world data, there is no ...
0
votes
1answer
48 views

Is $E[1_A | \mathscr{F_t}] = 0 ~\text{or} ~ 1 \ \Rightarrow E[1_A | \mathscr{F_{s}}] = E[1_A | \mathscr{F_t}]$ is only almost surely?

Spin-off from my previous question: Prove/Disprove $E[1_A | \mathscr{F_t}] = 0 ~\text{or} ~ 1 \ \Rightarrow E[1_A | \mathscr{F_{s}}] = E[1_A | \mathscr{F_t}]$ Apparently the conclusion holds true ...
2
votes
2answers
79 views

Why does a probability of 0 or 1 remain unchanged with new information, intuitively?

Related to these questions: Prove/Disprove $E[1_A | \mathscr{F_t}] = 0 ~\text{or} ~ 1 \ \Rightarrow E[1_A | \mathscr{F_{s}}] = E[1_A | \mathscr{F_t}]$ Does an unconditional probability of 1 or 0 ...
0
votes
1answer
220 views

Outlier filtering in 2D data in python

I have following data given: My curve fits it acceptable for my needs. I use here 4th degree polynomial. (data is limited to 0-100 percent range for both axis!) What I want to try now is to filter ...
3
votes
0answers
78 views

Cross correlation between 2 filtered time series

I have 2 band pass filtered time series for 30-90 day band I would like to understand the lagged correlation between these 2 series in this band. The issue is that autocorrelations exist in both ...
0
votes
0answers
30 views

Residual plot looks like inverted change plot for prediction model

I have a very big problem with my predictive model. What i essentially do is that I predict the volume in a tank by studying the flow into and out of the tank. I use the two flows to construct a ...
10
votes
1answer
295 views

Prove/Disprove $E[1_A | \mathscr{F_t}] = 0 ~\text{or} ~ 1 \ \Rightarrow E[1_A | \mathscr{F_{s}}] = E[1_A | \mathscr{F_t}]$

Prove/Disprove $E[1_A | \mathscr{F_t}] = 0 \ \text{or} \ 1 \ \text{a.s.} \ \Rightarrow E[1_A | \mathscr{F_{s}}] = E[1_A | \mathscr{F_t}] \ \text{a.s.}$ Given a filtered probability space $(\Omega, \...
0
votes
0answers
17 views

Determining “n” in moving averages

I'm monitoring seven variables (flow into tank, flow out of tank, volume in tank, concentration of the flow in, concentration of the flow out, concentration in tank and weight inside tank). The weight ...
3
votes
0answers
132 views

creating random variable with certain auto-correlation in R

I want to create a random variable with a given autocorrelation in R. The target autocorrelation is defined by: $$acf_{target}=(lag+1)^{(-b)}$$ with $b=1.41519$ which I derived from a natural ...
1
vote
1answer
288 views

Use tm_filter to search for multiple words

I´m new to R, so please bear with me. So, I know I can use the following to search for a word in several documents. ...
3
votes
1answer
59 views

What is the limiting distribution of the Bayesian Filtering

I've got a question about the iterative Bayesian filtering, the general form of which is shown as follows: $P(x|z_0,...z_{k+1})\propto P(z_{n+1}|x)P(x|z_0,...,z_k),\,k=0,1,\dots$. $P(x|z_0)=P_0(x)$ ...
1
vote
0answers
17 views

Where can I find good references regarding to noise filtering and prediction in time series?

I want to model the error structure of every certain time period obtained from the past errors produced by the predictions of nonlinear time series. I would like to know if someone knows specialized ...
1
vote
2answers
50 views

Online learning that “forgets” older aspects learned? (short-term memory)

I am looking for an online learning classifier that is highly adaptable and has only short-term memory. I need such a think in a object tracking system with high-dimensional feature vectors. Maybe a ...
0
votes
0answers
10 views

what is the difference between the ewma filter and mean filter

i have a sample and short question in which cases it is advantageous to use current and past informations such as ewma filter z=lamda*xi +(1-lamda)*xi-1 or to ...
1
vote
0answers
111 views

First order condition of HP Filter

The HP Filters for growth and cyclical components is written as: $$\min_{g_t}\sum_t \left[(y_t-g_t)^2+\lambda\left[(g_{t+1}-g_t)-(g_t-g_{t-1})\right]^2\right].$$ Hodrick and Prescott, on their paper ...
2
votes
0answers
39 views

Hodrick-Prescott Filter, Time Series, SDE, and Ito Isometry

The background of this question is a paper written by Morten O.Ravn and Harald Uhlig, titled "On Adjusting The Hodrick-Prescott Filter For The Frequency of Observations" Consider the decomposition of ...
0
votes
1answer
53 views

How to perform a filtering to get the low, middle and high frequency components of an image?

I am currently looking into some image processing project and just wondering how to obtain the low, middle and high frequency components of an image? For example, as this picture showed (I got it from ...
0
votes
1answer
145 views

Smoothing a particle filtered 2D trajectory

I am currently developing a very basic particle filter for a 2D robot localization task. My process is defined by a really simple velocity / steering angle based motion model. I am re-weighting the ...
1
vote
1answer
278 views

Linear discriminant analysis (Fisher) = Bayes?

I'd like to ask a question, I am reading book right now about mail filtering, both methods: naïve Bayes and Fisher are there very similar in implementation. I am also writing a paper on Bayesian spam ...
0
votes
0answers
88 views

Find repetitive patterns in matrices below

How can I identify the repetitive patterns from the matrices below? My problem is that the patterns in the matrix are different from matrix to matrix (dependent on the input data). I need some machine ...
5
votes
1answer
1k views

Beginner level: Help in learning Kalman Smoother (Part 1)

Parameter estimation of Linear Dynamical system is a tutorial which explains Kalman Filter, Smoothing, and Expectation Maximization. I have followed the derivation for Kalman Filter. But cannot ...
1
vote
0answers
56 views

Filtering noise in a data set

I'm trying to filter out noise from a Sonar. The idea is that the sonar in aimed upwards, after an object has come in it's range, I want to be able to tell if the object is moving away or coming ...
0
votes
0answers
14 views

Largest value in a set with a nearly equal distribution between value +/- 20%?

I am working through a data analysis task in a contract, and trying to build a generalized spreadsheet that can be used for this and similar analysis. I'm tripped up by a requested procedure, and ...
3
votes
1answer
61 views

Deriving the Bayes Filter Correction Equation

The correction rule for Bayes filters is: $$p\left(x_{k}|D_{k}\right)=\dfrac{p\left(y_{k}|x_{k}\right)\cdot p\left(x_{k}|D_{k-1}\right)}{p\left(y_{k}|D_{k-1}\right)} $$ For: State at time $k$ is $...
3
votes
1answer
186 views

How does backpropagation learn convolution filters?

I've understood how the backpropagation algorithm uses the partial derivatives of the weights to train a normal neural network. However, I cannot quite understand how the algorithm changes the filters....
1
vote
0answers
85 views

Filtering of time series

I would like information (references) about the reason why time series should be filtered before being used in a VAR model. Thank you in advance, Nikos.
0
votes
0answers
18 views

How to fit raw signal

I think this is simple, but somehow it's not working. I'm trying to fit my raw data so that basically I end up with beautiful 1 peak. I tried butterworth filtering, Gaussian fitting, a combination of ...
0
votes
1answer
11k views

Filtering a dataframe in R based on multiple Conditions [closed]

I am new to using R. I am trying to figure out how to create a df from an existing df that excludes specific participants. For example I am looking to exclude Women over 40 with high bp. I have ...
0
votes
0answers
20 views

Forecasting the business cycle?

I am wondering what is the best way to forecast the business cycle based on the past. Currently I feed the seasonally-adjusted GDP index data to a Hodrick–Prescott filter, extract the cyclical ...
1
vote
0answers
101 views

How can I smooth a set of discrete data points for the purpose of schedule planning?

Disclaimer: I do not have a background in statistics or the math behind filtering, save one long-time-ago college course. I have a well defined problem space. I am calculating hourly staffing ...
0
votes
1answer
197 views

What statistical method to correct systematic error in the output of a economic optimization model?

I am working with an economic optimization model which attempts to model the dynamics of a certain commodity market (prices, quantities, production etc.) for different frequencies (monthly, quarterly, ...
1
vote
1answer
433 views

Filtering using a SARIMA model in R

I am not an expert in statistics, but I would like to work on a SARIMAX model representing power consumption. The exogeneous variable would be the temperature, but for now I found here I might need to ...
2
votes
1answer
207 views

Matrix Factorization Recommendation Systems with Only “Like” Ratings

I'm trying to build a recommendation system, but I only have data on what my users have "liked", i.e. all non-missing data has the same numeric value. Is it possible for me to use matrix ...
0
votes
1answer
168 views

Heteroscedasticity filter for time series

I am looking for a method or package in R that can remove heteroscedasticity from time series. Specifically, I have a number of time series to which I want to fit a VAR model. Each time series may or ...
2
votes
1answer
130 views

What does “AR(p) filtered series” mean?

I guess this means that omitting some variables in a certain interval, say, $(x_1, x_2, x_3, x_4, x_5) \to (x_1, x_5)$ in AR(4) model. Is it right? Or does this means eliminating autocorrelations ...
1
vote
2answers
136 views

Does filtering of data by effect size violate some assumption of P value adjustment methods?

I have pre- and post-treatment continuous data for a large number of variables that I am analyzing for treatment effect. Normally I would obtain the P values and then adjust them for multiple testing ...
0
votes
1answer
97 views

Content-control (web filtering) using machine learning

I'm trying to build a content-control (web filtering) application using machine learning (just for training purposes). For example define gaming sites. I'm somewhat familiar with machine learning ...
1
vote
1answer
188 views

What is a density function?

I know about histograms and also know that if we connect the mid-points on the top of bars in a histogram we will get a frequency polygon. This polygon could then be 'smoothed' in a way that it ...
1
vote
0answers
125 views

Filtering with HMM

I want to use HMM for filtering, i.e. to find $p(x_t|y_{1:t})$. I see that the forward algorithm calculates the forward variable as a joint probability; $\alpha_t(i) = p(y_{1:t},x_t=S_i|\lambda)$, ...