Smoothing methods in data analysis, like splines or kernel smoothers. See https://en.wikipedia.org/wiki/Smoothing, which has a long list of various smoothing methods.

learn more… | top users | synonyms

0
votes
0answers
22 views

What does monotone polynomial plot represent?

I am trying to understand monotone and isotonic regression. I believe they will produce curve which are monotonely increasing or decreasing. In most of what I read on the net, the change is shown in a ...
0
votes
0answers
9 views

How to determine the optimal time averaging window

I have two large time series datasets with some background noises. I would assume the two datasets are either lag correlated or lead correlated. I tried to use time averaging to smooth out the dataset ...
0
votes
0answers
8 views

Good Turing smoothing

which smoothing algorithm is easy and effective in case of implementation point of view... My training corpus is a hex dump looks like, 64 FA EB 63 31 D2 62 22 19 BD 64 B5 63 17 4F 48 62 A8 64 11 0F ...
0
votes
1answer
26 views

Smoothing a particle filtered 2D trajectory

I am currently developing a very basic particle filter for a 2D robot localization task. My process is defined by a really simple velocity / steering angle based motion model. I am re-weighting the ...
0
votes
0answers
15 views

How can I smooth a set of distributions?

I have a set of results from MCMC modelling of a variable at discrete time points and I would like to know what kind of approaches I could take to smooth the results, given I would expect some kind of ...
2
votes
0answers
23 views

Tensor product smooth and Isotropic smooth

In 'mgcv', it is possible to fit two or more dimensional thin plate regression splines but not basis like cubic and P-splines. However, with tensor product smooth (te), we can use all the basis. My ...
0
votes
1answer
23 views

Choosing smoothing parameters across multiple Naïve Bayes classifiers with different number of categories

I would like to train multiple Naïve Bayes classifiers with different number of categories, and also have a global threshold for how certain one classifier must be in order for the classification to ...
2
votes
1answer
37 views

strange predicted values from LOESS model

I am trying to predict density function using LOESS in R. However, the predicted values I got are not in the estimated LOESS line. ...
2
votes
0answers
29 views

The correct use of tensor product in gam (mgcv) function

I want to resurrect a question that I asked two months ago (Comparing gam models using ti( )), but adding more explanations. The aim of my analyses was to compare several gam models with different ...
1
vote
1answer
44 views

can I apply loess or spline regression in mixed model?

My situation right now is that I have the mixed model with quadratic term but it doesn't perform very well. So I am wondering if I can apply loess or spline regression to the mixed model instead of ...
0
votes
1answer
47 views

Spike detection and removal in position data

Is there any good filter to remove big spikes in position data? I think lowpass filter should be good but is it possible to filter 2D position data with assumption its joint distributed? I mean, not ...
11
votes
1answer
571 views

Smoothing - when to use it and when not to?

There is quite an old post on William Briggs' blog which looks at the pitfalls of smoothing data and carrying that smoothed data through to analysis. The key argument is namely: If, in a moment of ...
1
vote
1answer
23 views

What is the effective kernel for smoothing methods?

I'm learning different smoothing methods and the term "effective kernel" came up and I don't really understand it. By definition, for a smoothing method, the vector of estimates ...
2
votes
0answers
39 views

Beginner level: How to plug in the smoothing equations into E step (Part 2)

Considering Gaussian Linear Dynamical system, $x_{t+1} = Ax_t + w_t$ $y_t = Cx_t + v_t$ $w_t = N(0,Q)$, $v_t = N(0,R)$ By Kalman Filter we are estimating the state variables and the state estimate ...
3
votes
1answer
222 views

Beginner level: Help in learning Kalman Smoother (Part 1)

Parameter estimation of Linear Dynamical system is a tutorial which explains Kalman Filter, Smoothing, and Expectation Maximization. I have followed the derivation for Kalman Filter. But cannot ...
3
votes
2answers
61 views

Symmetry in moving average smoothing in “Forecasting: principles and practice”

In the textbook Forecasting: principles and practice by Hyndman and Athana­sopou­los, in the moving average smoothing section (Sec 6.2), the authors speak of even order moving average smoothing not ...
0
votes
0answers
55 views

Why is locally weighted regression is a nonparametric method? How is it implemented in R?

I'm wondering where does the "nonparametric" label of locally weighted regression like LOESS or LOWESS comes from, i.e. why they are nonparametric methods? Also, I would like to know in general how ...
0
votes
0answers
31 views

Base sales in multivariate time series | MCMC model

I have been looking around online for good resources that explain how one would go about calculating base sales when preforming marketing mix modeling. I was told by a colleague that essentially they ...
0
votes
0answers
15 views

Smoothing of a 2D Empirical Distribution

I have a number of data points $\theta \in \mathbb{R}^2$ with corresponding values $x \in \mathbb{N}$. I am assuming the $x$ are realisations from a distribution $f(X | \theta)$. Given I have a lot ...
5
votes
3answers
550 views

How to use Pearson correlation correctly with time series

I have 2 time-series (both smooth) that I would like to cross-correlate to see how correlated they are. I intend to use the Pearson correlation coefficient. Is this appropriate? My second question ...
0
votes
0answers
99 views

“Future-independent” smoothing methods (as exponential smoothing)

I'm searching for time series smoothing algorithms, which give "future-independent" results - each next smoothed value depends only on previous data (smoothed or not smoothed), but not on any future ...
2
votes
1answer
63 views

Residuals from lowess curve

I am trying to obtain the residuals from a lowess fit. I’m using the lowess( ) function. Is there a way to do this?
2
votes
1answer
82 views

Is time-series an appropriate method to model data sampled at widely irregular time intervals ?

I am relatively inexperienced with data analysis. My question: Is time-series an appropriate method to fit trends to data sampled at widely-irregular time intervals such as forests of different ages ...
0
votes
0answers
171 views

lowess: R (& python-statsmodels) vs MATLAB (& biopython)

There seems to be two different interpretations of what LOWESS really means: one from R (also used by python-statsmodels), and one from MATLAB (also used by biopython), see comparison below. Could ...
0
votes
0answers
29 views

what is the best method of smoothing time series for product share data?

I am having a share data for products presribed over period of time.The share is calculated like for 3 products each one 1/3 share and like that,where products may vary and hence their share. What is ...
0
votes
0answers
74 views

What is the difference between Bézier splines and Loess curves?

I'm a bit naive on this topic, and wanted to understand the difference in the mechanics of Bézier splines and Loess curves as curve-fitting methods.
1
vote
1answer
52 views

Convert categorical percentage data into an overall mean

I have survey data in which the answer choices were "categorical" (0, <15%, 15-30%, 30-45%, 45-60%, 60-75%, 75-90%, >90%). In retrospect, this should have been a free response question, but I'm ...
2
votes
2answers
53 views

How to do smoothing of data with low counts?

We are a facing a problem of fluctuating shares for each week for the products we are studying due to low counts. e.g. If we have 4 products and for each one we have say like 5,8,3,6 details and we ...
3
votes
1answer
92 views

Smoothing parameter for spline curve with duplicate points

I have body mass and age data for a population of individuals. I want to fit a cubic smoothing spline curve to the data. I'm using smooth.spline in R, which warns against using cross-validation to ...
2
votes
0answers
174 views

In Kneser-Ney smoothing, how are unseen words handled?

From what I have seen, the (second-order) Kneser-Ney smoothing formula is in some way or another given as $ \begin{align} P^2_{KN}(w_n|w_{n-1}) &= \frac{\max \left\{ C\left(w_{n-1}, w_n\right) - ...
1
vote
2answers
89 views

What is the effect of having skewed dependent variable on scatterplot result

The histogram of my dependent variable is as following: I draw the scatter plot of my dependent variable and independent variable, and the result is as following picture? I am wondering if the ...
1
vote
0answers
14 views

Smoothing strategies for features assuming values from countably infinite domains

I am in the midst of programming a simple Naive Bayes classifier as an exercise. It is supposed to perform word-sense disambiguation on natural language phrases, e.g. predicting the correct meaning of ...
1
vote
0answers
14 views

How to compare fit of discrete process with discrete underlying process?

I am basically looking for an equivalent to something like an $\mathbb{R}^2$ for a model on a dataset that is itself simply a collection of points. That is, if my data set is (trivial example): ...
0
votes
0answers
107 views

How to apply Gaussian kernel to smooth density of points on 2D (theoretically)

I have a set of discrete points on a 2D surface and need to build a heat map or a distribution of the density of the points. However, I also need to smooth out the density/distribution by applying ...
0
votes
0answers
40 views

In practice, how is the penalisation matrix for splines, created by smoothCon (package mgcv), specified?

Evaluating a smooth object with smoothCon provides, besides several other things, the ordinary "untouched" spline bases ...
2
votes
2answers
142 views

Which forecasting method for load profiles

I'm new to this forum and I'm quite new to forecasting. Currently I'm trying to learn the basics about exponential smoothing, ARIMA etc. Now I want to forecast the total energy consumption of a rather ...
0
votes
1answer
68 views

What R packages are available for nonparametric regression of two predictors?

I would like to fit a nonparametric regression model with two predictors. What R packages are available? I found 'loess' function in 'stats' package and 'gam' function in 'gam' package. Is there ...
0
votes
0answers
35 views

How are local and global plug-in bandwidths different in kernel smoothing regression?

I'm looking into an R package 'lokern.' It provides two bandwidths selectors, global and local plug-in bandwidth. I would like to understand the difference between two methods. My understanding is ...
1
vote
0answers
37 views

How can I smooth a set of discrete data points for the purpose of schedule planning?

Disclaimer: I do not have a background in statistics or the math behind filtering, save one long-time-ago college course. I have a well defined problem space. I am calculating hourly staffing ...
2
votes
1answer
3k views

Smoother lines for ggplot2

This question probably has a simple solution, still the thing is I've written a code to plot mortality in 2 different groups and that is, death in obese patients vs not obese. Now their are 2 groups ...
4
votes
3answers
2k views

In Naive Bayes, why bother with Laplacian smoothing when we have unknown words in the test set?

I was reading over Naive Bayes Classification today. I read, under the heading of Parameter Estimation with add 1 smoothing: "Let $c$ refer to a class (such as Positive or Negative), and let $w$ ...
0
votes
0answers
14 views

Invalid measurement data removing

I have a lot of data (temperatures and similar things) collected using automatic devices. Lots of these data is let's call it smooth: even when the temperature increases/decreases fast there are few ...
3
votes
1answer
347 views

scatterplot smoothing in r with big dataset: different methods

I have a large dataset (>300,000 rows) with two variables. y is binary and x is continuous & numeric. I'd like to plot y and add smooth curve against x. I understand that loess(y~x) is a solution, ...
5
votes
2answers
117 views

Smoothing dirty data?

I'm trying to do analysis on revenue data and due to circumstances beyond my pay grade, there are large spikes in the data around the time financial reports are due, followed by corresponding low ...
3
votes
1answer
59 views

How can I estimate the tail of a distribution with a truncated distribution?

The broadband speed data I'm working with have all values over 30Mbps placed into a >30 category. The distribution is thus truncated. This leads to the final column in the histogram below being a ...
2
votes
2answers
1k views

Kalman Filtering, Smoothing and Parameter Estimation for State Space Models in R and C#

I am in the process of writing an open source State Space Analysis suite in C# (for fun). I have implemented a number of different Kalman-Based Filters (Kalman Filter, Information Filter and the ...
2
votes
0answers
107 views

Supersmoother algorithm - tutorial?

I came across a reference to a "Supersmoother" algorithm that was applied to prices in a timeseries. Does anyone know of a tutorial in Python or whether Pandas has such a capability?
1
vote
1answer
127 views

Kernel density estimator function explanation needed

I'm studying about kernel density estimation and from wikipedia I get this formula: $$\hat{f}_h(x, h) = \frac{1}{n}\sum^{n}_{i=1}K_h(x-x_i) = \frac{1}{nh}\sum_{i=1}^nK(\frac{x-x_i}{h}).$$ I think ...
7
votes
2answers
594 views

Selection of k knots in regression smoothing spline equivalent to k categorical variables?

I'm working on a predictive cost model where the patient's age (an integer quantity measured in years) is one of the predictor variables. A strong nonlinear relationship between age and risk of a ...
1
vote
1answer
117 views

R - How to smooth a conversion of weekly sales into daily sales?

I have weekly sales figures, and would like to convert them into daily sales figures, making a simple hypothesis that there are 7 days with equal sales "power". Let's imagine that I have: ...