The tag has no wiki summary.

learn more… | top users | synonyms

0
votes
0answers
8 views

Spike detection and removal in position data

Is there any good filter to remove big spikes in position data? I think lowpass filter should be good but is it possible to filter 2D position data with assumption its joint distributed? I mean, not ...
11
votes
1answer
456 views

Smoothing - when to use it and when not to?

There is quite an old post on William Briggs' blog which looks at the pitfalls of smoothing data and carrying that smoothed data through to analysis. The key argument is namely: If, in a moment of ...
0
votes
0answers
12 views

What is the effective kernel for smoothing methods?

I'm learning different smoothing methods and the term "effective kernel" came up and I don't really understand it. By definition, for a smoothing method, the vector of estimates ...
2
votes
0answers
29 views

Beginner level: How to plug in the smoothing equations into E step (Part 2)

Considering Gaussian Linear Dynamical system, $x_{t+1} = Ax_t + w_t$ $y_t = Cx_t + v_t$ $w_t = N(0,Q)$, $v_t = N(0,R)$ By Kalman Filter we are estimating the state variables and the state estimate ...
3
votes
1answer
159 views

Beginner level: Help in learning Kalman Smoother (Part 1)

Parameter estimation of Linear Dynamical system is a tutorial which explains Kalman Filter, Smoothing, and Expectation Maximization. I have followed the derivation for Kalman Filter. But cannot ...
3
votes
2answers
46 views

Symmetry in moving average smoothing in “Forecasting: principles and practice”

In the textbook Forecasting: principles and practice by Hyndman and Athana­sopou­los, in the moving average smoothing section (Sec 6.2), the authors speak of even order moving average smoothing not ...
0
votes
0answers
39 views

Why is locally weighted regression is a nonparametric method? How is it implemented in R?

I'm wondering where does the "nonparametric" label of locally weighted regression like LOESS or LOWESS comes from, i.e. why they are nonparametric methods? Also, I would like to know in general how ...
0
votes
0answers
24 views

Base sales in multivariate time series | MCMC model

I have been looking around online for good resources that explain how one would go about calculating base sales when preforming marketing mix modeling. I was told by a colleague that essentially they ...
0
votes
0answers
13 views

Smoothing of a 2D Empirical Distribution

I have a number of data points $\theta \in \mathbb{R}^2$ with corresponding values $x \in \mathbb{N}$. I am assuming the $x$ are realisations from a distribution $f(X | \theta)$. Given I have a lot ...
5
votes
2answers
172 views

How to use Pearson correlation correctly with time series

I have 2 time-series (both smooth) that I would like to cross-correlate to see how correlated they are. I intend to use the Pearson correlation coefficient. Is this appropriate? My second question ...
0
votes
0answers
93 views

“Future-independent” smoothing methods (as exponential smoothing)

I'm searching for time series smoothing algorithms, which give "future-independent" results - each next smoothed value depends only on previous data (smoothed or not smoothed), but not on any future ...
2
votes
1answer
40 views

Residuals from lowess curve

I am trying to obtain the residuals from a lowess fit. I’m using the lowess( ) function. Is there a way to do this?
2
votes
1answer
62 views

Is time-series an appropriate method to model data sampled at widely irregular time intervals ?

I am relatively inexperienced with data analysis. My question: Is time-series an appropriate method to fit trends to data sampled at widely-irregular time intervals such as forests of different ages ...
0
votes
0answers
105 views

lowess: R (& python-statsmodels) vs MATLAB (& biopython)

There seems to be two different interpretations of what LOWESS really means: one from R (also used by python-statsmodels), and one from MATLAB (also used by biopython), see comparison below. Could ...
0
votes
0answers
26 views

what is the best method of smoothing time series for product share data?

I am having a share data for products presribed over period of time.The share is calculated like for 3 products each one 1/3 share and like that,where products may vary and hence their share. What is ...
0
votes
0answers
58 views

What is the difference between Bézier splines and Loess curves?

I'm a bit naive on this topic, and wanted to understand the difference in the mechanics of Bézier splines and Loess curves as curve-fitting methods.
1
vote
1answer
45 views

Convert categorical percentage data into an overall mean

I have survey data in which the answer choices were "categorical" (0, <15%, 15-30%, 30-45%, 45-60%, 60-75%, 75-90%, >90%). In retrospect, this should have been a free response question, but I'm ...
2
votes
2answers
47 views

How to do smoothing of data with low counts?

We are a facing a problem of fluctuating shares for each week for the products we are studying due to low counts. e.g. If we have 4 products and for each one we have say like 5,8,3,6 details and we ...
0
votes
1answer
55 views

Smoothing parameter for spline curve with duplicate points

I have body mass and age data for a population of individuals. I want to fit a cubic smoothing spline curve to the data. I'm using smooth.spline in R, which warns against using cross-validation to ...
1
vote
0answers
97 views

In Kneser-Ney smoothing, how are unseen words handled?

From what I have seen, the (second-order) Kneser-Ney smoothing formula is in some way or another given as $ \begin{align} P^2_{KN}(w_n|w_{n-1}) &= \frac{\max \left\{ C\left(w_{n-1}, w_n\right) - ...
1
vote
2answers
76 views

What is the effect of having skewed dependent variable on scatterplot result

The histogram of my dependent variable is as following: I draw the scatter plot of my dependent variable and independent variable, and the result is as following picture? I am wondering if the ...
1
vote
0answers
10 views

Smoothing strategies for features assuming values from countably infinite domains

I am in the midst of programming a simple Naive Bayes classifier as an exercise. It is supposed to perform word-sense disambiguation on natural language phrases, e.g. predicting the correct meaning of ...
1
vote
0answers
14 views

How to compare fit of discrete process with discrete underlying process?

I am basically looking for an equivalent to something like an $\mathbb{R}^2$ for a model on a dataset that is itself simply a collection of points. That is, if my data set is (trivial example): ...
0
votes
0answers
73 views

How to apply Gaussian kernel to smooth density of points on 2D (theoretically)

I have a set of discrete points on a 2D surface and need to build a heat map or a distribution of the density of the points. However, I also need to smooth out the density/distribution by applying ...
0
votes
0answers
24 views

In practice, how is the penalisation matrix for splines, created by smoothCon (package mgcv), specified?

Evaluating a smooth object with smoothCon provides, besides several other things, the ordinary "untouched" spline bases ...
2
votes
2answers
118 views

Which forecasting method for load profiles

I'm new to this forum and I'm quite new to forecasting. Currently I'm trying to learn the basics about exponential smoothing, ARIMA etc. Now I want to forecast the total energy consumption of a rather ...
0
votes
1answer
50 views

What R packages are available for nonparametric regression of two predictors?

I would like to fit a nonparametric regression model with two predictors. What R packages are available? I found 'loess' function in 'stats' package and 'gam' function in 'gam' package. Is there ...
0
votes
0answers
29 views

How are local and global plug-in bandwidths different in kernel smoothing regression?

I'm looking into an R package 'lokern.' It provides two bandwidths selectors, global and local plug-in bandwidth. I would like to understand the difference between two methods. My understanding is ...
1
vote
0answers
30 views

How can I smooth a set of discrete data points for the purpose of schedule planning?

Disclaimer: I do not have a background in statistics or the math behind filtering, save one long-time-ago college course. I have a well defined problem space. I am calculating hourly staffing ...
2
votes
1answer
1k views

Smoother lines for ggplot2

This question probably has a simple solution, still the thing is I've written a code to plot mortality in 2 different groups and that is, death in obese patients vs not obese. Now their are 2 groups ...
3
votes
3answers
728 views

In Naive Bayes, why bother with Laplacian smoothing when we have unknown words in the test set?

I was reading over Naive Bayes Classification today. I read, under the heading of Parameter Estimation with add 1 smoothing: "Let $c$ refer to a class (such as Positive or Negative), and let $w$ ...
0
votes
0answers
12 views

Invalid measurement data removing

I have a lot of data (temperatures and similar things) collected using automatic devices. Lots of these data is let's call it smooth: even when the temperature increases/decreases fast there are few ...
3
votes
1answer
256 views

scatterplot smoothing in r with big dataset: different methods

I have a large dataset (>300,000 rows) with two variables. y is binary and x is continuous & numeric. I'd like to plot y and add smooth curve against x. I understand that loess(y~x) is a solution, ...
5
votes
2answers
111 views

Smoothing dirty data?

I'm trying to do analysis on revenue data and due to circumstances beyond my pay grade, there are large spikes in the data around the time financial reports are due, followed by corresponding low ...
0
votes
0answers
73 views

How to compare GCV diagnostic plots of a spline object

I am trying to analyse diagnostic plots of a spline object, I am using the package Fields from R. I am lost with the GCV function plot, since I can't find any guidelines of what to look for when ...
3
votes
1answer
52 views

How can I estimate the tail of a distribution with a truncated distribution?

The broadband speed data I'm working with have all values over 30Mbps placed into a >30 category. The distribution is thus truncated. This leads to the final column in the histogram below being a ...
2
votes
2answers
918 views

Kalman Filtering, Smoothing and Parameter Estimation for State Space Models in R and C#

I am in the process of writing an open source State Space Analysis suite in C# (for fun). I have implemented a number of different Kalman-Based Filters (Kalman Filter, Information Filter and the ...
2
votes
0answers
83 views

Supersmoother algorithm - tutorial?

I came across a reference to a "Supersmoother" algorithm that was applied to prices in a timeseries. Does anyone know of a tutorial in Python or whether Pandas has such a capability?
1
vote
1answer
113 views

Kernel density estimator function explanation needed

I'm studying about kernel density estimation and from wikipedia I get this formula: $$\hat{f}_h(x, h) = \frac{1}{n}\sum^{n}_{i=1}K_h(x-x_i) = \frac{1}{nh}\sum_{i=1}^nK(\frac{x-x_i}{h}).$$ I think ...
0
votes
0answers
75 views

How to smooth time-series NDVI data using polynomial regression

I have a time-series NDVI image. The image has 26 bands. 26 bands mean that images taken in 8-day time interval and counted in Julian days (97 to 297). For example; first band of the image is NDVI ...
7
votes
2answers
511 views

Selection of k knots in regression smoothing spline equivalent to k categorical variables?

I'm working on a predictive cost model where the patient's age (an integer quantity measured in years) is one of the predictor variables. A strong nonlinear relationship between age and risk of a ...
1
vote
1answer
86 views

R - How to smooth a conversion of weekly sales into daily sales?

I have weekly sales figures, and would like to convert them into daily sales figures, making a simple hypothesis that there are 7 days with equal sales "power". Let's imagine that I have: ...
1
vote
0answers
31 views

Line smoothness/waviness

I am working on a project which requires me to watch video of athletes and measure the straightness/smoothness/waviness, whatever term is acceptable, of their spine. Dividing the spine into segments ...
2
votes
0answers
87 views

Modeling binary outcomes - inaccurate model when using logistic regression?

I am trying to model the probability of a binary outcome with the independent variable being an hour variable. I understand that linear regression is not the correct method for this type of task (I ...
6
votes
1answer
85 views

Kernel smoothing for Edgeworth expansion

Suppose I have an estimator which includes an indicator function in the objective function, then the objective function is not smooth. But if I want to approximate the behavior of this estimator in ...
1
vote
0answers
60 views

How to set initial values for Holt-Winters backcasting?

I'm trying to figure out how to set initial seasonal values for backcasting of the multiplicative Holt Winters. As pointed out in this thread: Holt Winters initialization using backcasting like SPSS, ...
1
vote
0answers
72 views

smoothing nodes values on a graph given adjacency matrix

I am currently looking for a method to smooth values on a graph (composed of vertices and edges). For example I have a graph with a set of nodes V and I want to be able to smooth it. I could have ...
1
vote
1answer
107 views

What is a density function?

I know about histograms and also know that if we connect the mid-points on the top of bars in a histogram we will get a frequency polygon. This polygon could then be 'smoothed' in a way that it ...
6
votes
2answers
950 views

Laplace smoothing and Dirichlet prior

On the wikipedia article of Laplace smoothing (or additive smoothing), it is said that from a Bayesian point of view, this corresponds to the expected value of the posterior distribution, using a ...
0
votes
1answer
498 views

Using Moving-average smoothing in forecast package [closed]

I tried to use the non-centred moving average, that means just using past values by setting the option centre = FALSE, but unfortunately you get the centred results. Can anyone detect the failure ...