The tag has no wiki summary.

learn more… | top users | synonyms

0
votes
0answers
7 views

Aggregating Equity Intraday Ticks

I am currently working with intraday equity data. The ticks are sourced from Bloomberg API. Bloomberg only timestamps down to the second (not millisecond) and data is not in order. In many instances ...
0
votes
0answers
19 views

Find repetitive patterns in matrices below

How can I identify the repetitive patterns from the matrices below? My problem is that the patterns in the matrix are different from matrix to matrix (dependent on the input data). I need some machine ...
3
votes
1answer
159 views

Beginner level: Help in learning Kalman Smoother (Part 1)

Parameter estimation of Linear Dynamical system is a tutorial which explains Kalman Filter, Smoothing, and Expectation Maximization. I have followed the derivation for Kalman Filter. But cannot ...
0
votes
0answers
24 views

filtering out information

I would like to filter out the effect of temperature among many variables that explain the electricity demand and produce temperature adjusted electricity demand. Can you suggest any way of doing ...
1
vote
0answers
14 views

Filtering noise in a data set

I'm trying to filter out noise from a Sonar. The idea is that the sonar in aimed upwards, after an object has come in it's range, I want to be able to tell if the object is moving away or coming ...
0
votes
0answers
63 views

How to apply a Gaussian filter to co-ordinate data

I'm working on a project to investigate the correlation of surface finish and face sealing effectiveness. I have a trace of the surface of my seal and the next step is to apply a Gaussian filter to ...
0
votes
0answers
11 views

Largest value in a set with a nearly equal distribution between value +/- 20%?

I am working through a data analysis task in a contract, and trying to build a generalized spreadsheet that can be used for this and similar analysis. I'm tripped up by a requested procedure, and ...
3
votes
1answer
48 views

Deriving the Bayes Filter Correction Equation

The correction rule for Bayes filters is: $$p\left(x_{k}|D_{k}\right)=\dfrac{p\left(y_{k}|x_{k}\right)\cdot p\left(x_{k}|D_{k-1}\right)}{p\left(y_{k}|D_{k-1}\right)} $$ For: State at time $k$ is ...
2
votes
1answer
42 views

How does backpropagation learn convolution filters?

I've understood how the backpropagation algorithm uses the partial derivatives of the weights to train a normal neural network. However, I cannot quite understand how the algorithm changes the ...
0
votes
0answers
50 views

WEKA filter for thresholding a numeric attribute to make it binary

I have a numeric attribute with values in the range of [0.0,1.0]. I want to apply a filter in WEKA that will convert the attribute to be binary where false <= 0.3 < true. However, I can't figure ...
0
votes
0answers
10 views

a joint probability distribution that passes through all markov network graph without being filtered

from filter view of the Markov Network where only those distributions can pass that satisfy all conditional independence statements given by the graph. • Can we think of distribution that can pass ...
0
votes
0answers
29 views

Gabor filters: Large variance compared to the mean

I am trying to extract Gabor features from an input image. So, I have setup a series of Gabor filters with different parameters (frequency, angle and standard deviation) and I am convolving each of ...
0
votes
0answers
17 views

filter feature selection output and cross validation

If I use a filter method for ranking the features like Relief. suppose I have 100 features with 1000 sample and I used cross validation 3-fold . therefore I have 3 ranks for may features . at the end ...
1
vote
0answers
56 views

Filtering of time series

I would like information (references) about the reason why time series should be filtered before being used in a VAR model. Thank you in advance, Nikos.
0
votes
0answers
14 views

How to fit raw signal

I think this is simple, but somehow it's not working. I'm trying to fit my raw data so that basically I end up with beautiful 1 peak. I tried butterworth filtering, Gaussian fitting, a combination of ...
0
votes
1answer
537 views

Filtering a dataframe in R based on multiple Conditions [closed]

I am new to using R. I am trying to figure out how to create a df from an existing df that excludes specific participants. For example I am looking to exclude Women over 40 with high bp. I have ...
0
votes
0answers
18 views

Forecasting the business cycle?

I am wondering what is the best way to forecast the business cycle based on the past. Currently I feed the seasonally-adjusted GDP index data to a Hodrick–Prescott filter, extract the cyclical ...
1
vote
0answers
30 views

How can I smooth a set of discrete data points for the purpose of schedule planning?

Disclaimer: I do not have a background in statistics or the math behind filtering, save one long-time-ago college course. I have a well defined problem space. I am calculating hourly staffing ...
0
votes
1answer
86 views

What statistical method to correct systematic error in the output of a economic optimization model?

I am working with an economic optimization model which attempts to model the dynamics of a certain commodity market (prices, quantities, production etc.) for different frequencies (monthly, quarterly, ...
1
vote
1answer
228 views

Filtering using a SARIMA model in R

I am not an expert in statistics, but I would like to work on a SARIMAX model representing power consumption. The exogeneous variable would be the temperature, but for now I found here I might need to ...
2
votes
1answer
75 views

Matrix Factorization Recommendation Systems with Only “Like” Ratings

I'm trying to build a recommendation system, but I only have data on what my users have "liked", i.e. all non-missing data has the same numeric value. Is it possible for me to use matrix ...
0
votes
1answer
55 views

Heteroscedasticity filter for time series

I am looking for a method or package in R that can remove heteroscedasticity from time series. Specifically, I have a number of time series to which I want to fit a VAR model. Each time series may or ...
0
votes
0answers
23 views

Filter algorithm for a system

I have a system with the following structure: $X_{t+1} = X_t + E_{t+1}$ $E_{t+1} \sim N(0, \Sigma)$ $Y_{t+1} = f(X_{t+1})$ $Y_{t+1} \sim {\rm Uniform}(a_{t+1}, b_{t+1})$ So $X$ is a vector of ...
0
votes
0answers
34 views

Are trend/cycle filters intended to be used in predictive models, or just analysis?

I am relatively new to time series modelling and for a task I have I've had good success (in terms of forecast error) by first splitting the data into a trend and cycle components using a ...
2
votes
1answer
78 views

What does “AR(p) filtered series” mean?

I guess this means that omitting some variables in a certain interval, say, $(x_1, x_2, x_3, x_4, x_5) \to (x_1, x_5)$ in AR(4) model. Is it right? Or does this means eliminating autocorrelations ...
1
vote
2answers
74 views

Does filtering of data by effect size violate some assumption of P value adjustment methods?

I have pre- and post-treatment continuous data for a large number of variables that I am analyzing for treatment effect. Normally I would obtain the P values and then adjust them for multiple testing ...
0
votes
1answer
79 views

Content-control (web filtering) using machine learning

I'm trying to build a content-control (web filtering) application using machine learning (just for training purposes). For example define gaming sites. I'm somewhat familiar with machine learning ...
1
vote
1answer
107 views

What is a density function?

I know about histograms and also know that if we connect the mid-points on the top of bars in a histogram we will get a frequency polygon. This polygon could then be 'smoothed' in a way that it ...
1
vote
0answers
66 views

Filtering with HMM

I want to use HMM for filtering, i.e. to find $p(x_t|y_{1:t})$. I see that the forward algorithm calculates the forward variable as a joint probability; $\alpha_t(i) = p(y_{1:t},x_t=S_i|\lambda)$, ...
0
votes
0answers
28 views

filtering discrete data

I looking for good ways to estimate a true state of the system from a set of observations, that take several integer values. E.g. I have an sequence of observations X_i of the unknown variable Y. Both ...
1
vote
1answer
264 views

Moving average filter for outlier removal

I am using a moving average filter to smooth data for outlier removal. By changing the number of average points, I am getting different result. My data are multi-dimensional feature vectors. I ...
1
vote
0answers
190 views

Band pass filter giving ‘wrong’ turning point

I am trying to run a Christiano FitzGerald band pass filter to estimate a long-run trend (with period in excess of 70 years). My data are the demeaned natural log of a commodity price index. My ...
3
votes
0answers
73 views

How to extract uniform noise from the mixture distribution

Let $X$ be a sample from the distribution which is the mixture of the useful signal with the distribution $\xi(\theta)$ and a uniform noise $U[a, b]$. The probability of observing $U[a, b]$, the ...
0
votes
1answer
351 views

Weka batch filtering on random projection

I am new to Weka. I have used the following statement in Weka CLI: ...
1
vote
0answers
39 views

What is the difference between a one-sided filter and a two-sided filter when looking at time series analysis?

I'm looking to understand the difference between the two and grasp in which situations each might be preferred over the other.
0
votes
2answers
169 views

Smoothing algorithm for saturating function

I have a noisy readout of a curve that is monotonically increasing or decreasing for a narrow range of points and then quickly saturates. I don't know exactly where the saturation point is, but from ...
2
votes
2answers
71 views

Method to remove bad values in time series (bad values known to take on a particular value)

This sounds easy, but I don't know of a good statistical method for it. I have a time series that has (good) data points that range from ~3.5 to 30. The data are collected by an automated sensor. ...
1
vote
1answer
202 views

Non-causal of variable threshold algorithm

I'm not entirely sure this question belongs here. (Maybe better suited on stackoverflow or theoretical computer science). But here it goes. I'm reading a paper called. "Time-Frequency Analysis of ...
0
votes
0answers
52 views

Removing data above the mean

I am working with a data set, for which to remove noisy data I take the average of the sample itself and cut anything above that average. What I'm trying to understand here is does this have a name ...
1
vote
1answer
481 views

Multivariate exponential smoothing and Kalman filter equivalence

Suppose the time-series $X$ is hidden state Gaussian random walk and we observe $Y = X + e$, where $e$ is gaussian white noise independent of $X$. The Kalman estimator of $X$ in this case has a ...
1
vote
0answers
49 views

Question about asymptotics of steepest descent method in the context of adaptive filtering

The model which will be used is defined as $e(n) = d(n) - y(n)$ with $y(n) = x(n)^Tw(n)$. where $e(n)$ is the error term of the n-th observation, $x(n)$ the input vector of the n-th ...
2
votes
1answer
823 views

State Space formulation of Hodrick-Prescott filter

I would like to apply the Kalman filter in order to get a causal Hodrick-Prescott filter. The Hodrick-Prescott filter models a time series $(y_t)_{t=0}^T$ as $$ y_t = \tau_t + c_t $$ where $\tau_t$ is ...
1
vote
1answer
1k views

Formula for one-sided Hodrick-Prescott filter

I am not very familiar with filters. The Hodrick-Prescott filter as one can find it e.g. in wikipedia is two-sided. I also found an R implementation for this in the R package mFilter. There the filter ...
1
vote
0answers
148 views

MSE of filtered noisy signal - Derivation

I'm working on understanding the derivation of the optimal time constant for filters based on minimizing mean squared error. Unfortunately the text made a big jump between steps and lost me. Here's ...
3
votes
2answers
1k views

Filtering using ARMA model in R

I have two time-series, x and y. I would like to prewhiten x by fitting an ARMA(p,q) (or in ...
1
vote
0answers
33 views

Filtering virtual economy transactions

I am currently writing a pet-project for a browser based game which consists of two parts, first part is a data miner which observes the transactions in the market. And second part is as you might ...
5
votes
1answer
274 views

Determining smoothing parameter in HP filter for hourly data

I'm trying to determine a smoothing parameter for the Hodrick-Prescott filter. I've seen that there are papers on the topic but they are far too advanced for my comprehension. If I have a data set, ...
3
votes
1answer
85 views

Distinguishing statistical (global) and network (local) effects

I am analyzing how users of specific service affect each other by observing communication between them and changes in membership plans. The social network consists of 3M users and 40M connections ...