Smoothing methods in data analysis, like splines or kernel smoothers. See https://en.wikipedia.org/wiki/Smoothing, which has a long list of various smoothing methods.

learn more… | top users | synonyms

0
votes
0answers
7 views

Evaluate various censored estimators goodness-of-fit with and without ground truth

I would like to evaluate the goodness-of-fit for non-parametric estimators (Kaplan-Meier, Nelson-Aalen and their smoothed variations) of the survival function on a number of datasets. For some of them,...
0
votes
0answers
6 views

Validating how to treat unseen NGrams for Kneser Ney smoothing

I'm implementing the Kneser Ney smoothing algorithm and I would like to validate my understanding on how to treat unseen ngrams in the formula. The formula for Kneser Ney is as follows: $P_{(KN)}(...
0
votes
0answers
13 views

Can we set smoothing parameter to zero in mgcv package? [on hold]

Is there anyway to set mgcv package in R to apply no smoothing parameter in penalized regression, so that it turns to an ordinary spline regression?
0
votes
0answers
32 views

Demand Forecasting Models

I want to forecast demand of various products using time series data of 2 years (using loops on products in R), frequency is daily and demand is to be forecasted for next 90 days I have used the ...
0
votes
0answers
9 views

Simple Good-Turing smoothing - probability of unseen if no frequency of 1

I am using the Simple Good-Turing estimation procedure to estimate probabilities for events, some of which have not been seen in the sample. In the procedure, the probability of unseen events is ...
0
votes
0answers
16 views

Raster time series smoothing

I'm searching for an R package to raster time series smoothing. Currently, I'm using an approach like this one (using the equation suggested by Hamunyella et al., 2013) ...
2
votes
1answer
29 views

In Kneser Ney smoothing, how to implement the recursion in the formula?

I'm working in a project trying to implement the Kneser-Key algorithm. I think I got up to the step of implementing this formula for bigrams: $P_{(KN)}(w_i|w_{i-1}) = \frac{max(c(w_{-1}, w_{1}) - \...
0
votes
0answers
14 views

Non-parametric smoothing a small sample time series with fixed/known t(0) and t(n)

I would be grateful for any suggestions on how to smooth a time series with the following properties: We observe $t(i)$ for every integer $i=0...T$. $0 < t(i) < \infty$ $T$ is typically small (...
3
votes
2answers
222 views

Fitting a smoothed curve to a noisy data

I have a variable with sales data over time. It is very noisy at a disaggregate level but if you look at it as a whole, you can see a smoothing curve that follows a polynomial pattern. Is there a way ...
0
votes
2answers
69 views

Kneser-Ney for unigrams?

I was wondering if it is at all possible to use Kneser-Ney to smooth word unigram probabilites? The basic idea behind back-off is to use (n-1)-gram frequencies when an n-gram has 0 count. This is ...
0
votes
0answers
9 views

Naive Bayes and smoothing

For simplicity, let's say that we want to perform binary classification using Naive Bayes on a Boolean function. That is, the target function is $c: \{0, 1\}^n \rightarrow \{0, 1\}$. Hence, the two ...
3
votes
1answer
24 views

Behaviour of neighbouring points in LOESS smoothing when data is not uniformly distributed

The purpose of loess is to create an 'average' value of the response for any $x$, by using points in the region of $x$ to create a local regression line. From what I understand, picking the ...
0
votes
1answer
25 views

Why do gam predictions not match gam smooth?

I am studying the effect of organic farming on honey reserves in honeybee colonies. I am trying to see how an increase in the percentage of organically farmed land (at various buffers around bee hives)...
0
votes
1answer
36 views

Smoothing time series

I have very basic experience with time series analysis and I am struggling to figure out the following. I have this graph which shows the counts of people in and out of a store for a day, the counts ...
2
votes
0answers
30 views

The Pros and Cons of Smoothing spline

I have a general question. Recently I just learnt Basis Expansion and Regularization. There are several interesting techniques including: cubic spline, natural spline, b-spline and smoothing spline. ...
0
votes
0answers
57 views

Kneser Ney smoothing, why the maths allows division by 0?

I'm studying Natural Language Processing and the various smoothing approaches. I'm finding a little hard to understand how to handle unknown words with the Kneser-Ney smoothing. In particular I'm ...
1
vote
1answer
31 views

AIC on Savitzky-Golay width

I want to use a Savitzky-Golay filter to smooth some data. There is a right width to use based on the data that it is smoothing. A number of papers basically use "eyeball norm" on the parameters but ...
0
votes
0answers
29 views

Locally weighted regression vs. splines

What's the pros/cons of splines approaches compared to locally weighted regression approaches for the purposes of (a) scatter plot smoothing and (b) prediction? Obviously, in the case of prediction I ...
1
vote
0answers
35 views

Simple Good-Turing Probabilities higher than old probabilities

I've implemented Simple Good-Turing to get new probabilities of unigrams on my Corpus. Everything works fine. I'm just confused on how come the probability of a word after the Good-Turing discount be ...
0
votes
0answers
27 views

Comparison of different forecasting models

I have a time series data of 1000 points (in %) for each of the different machines. I tried different forecasting techniques to make a one step prediction. The goal is to find out one common ...
0
votes
0answers
25 views

Detecting Significant Change in Time Series With Small Sample Size

I am trying to determine when exactly "sexual revolutions" where occurring in the US from 1964 to 1985. I have a yearly time series (22 data points) representing the number of publications containing ...
0
votes
0answers
67 views

Outlier removal in GLM

I was trying to solve the question below. I tried fitting a GLM (Gamma, with location and diseases as covariates) to fit the data, but the deviance was too large, possibly because of outliers. I Just ...
0
votes
0answers
23 views

R LOESS losing precision along the way

I am trying to draw a smoothing line on a frequency graph with ggplot2. My data are represented like this: ...
1
vote
0answers
33 views

Linear Smoothers and cross validation. That is easy, but is it?

Consider the nonparametric regression problem $Y_i=m(x_i)+\epsilon_i$. Let $m_h$ be the estimator based on smoothing parameter $h$ $$\widehat{\mbox{MSE}}(h) = n^{-1} \sum_{i=1}^n \Big( \frac{Y_i-\...
1
vote
1answer
71 views

How to smooth data and force monotonicity

I have some data which I would like to smooth so that the smoothed points are monotonically decreasing. My data sharply decreases and then begins to plateau. Here's an example using R ...
1
vote
0answers
24 views

Learn a random walk process with RTS smoother

I'm trying to learn a random walk process as described at section 7 of http://www.cs.cmu.edu/~epxing/papers/SDM08_Ahmed.pdf I have a set of points over N epochs. Given a set of clusters $K$, every ...
1
vote
0answers
32 views

Trouble deriving nonparametric regression result

I'm working through Computational Statistics, Second Edition by Givens and Hoeting and I cannot reproduce a result stated in the book. We're considering a nonparametric regression problem where the ...
1
vote
0answers
100 views

Is the Laplace/Lidstone smoothing parameter (talking about Multinomial/Bernoulli Naive Bayes) related to the particular structure of the dataset?

I'm working with Multinomial and Bernoulli Naive Bayes implementation of scikit-learn (python) for text classification. I'm using the 20_newsgroups dataset. From the scikit documentation we have: <...
1
vote
0answers
71 views

What are the published methods to smooth a time series of weights?

I am looking for methods to smooth a time series of human weights, in particular I would like to know methods that are quoted or described in articles published in peer-reviewed journals. I am also ...
0
votes
1answer
77 views

Difference between smoothing spline and penalised spline

I have read many documents, and I am confused about the difference between smoothing splines and penalised splines. Are those two the same? Can someone please suggest any good document which can ...
1
vote
1answer
41 views

Smooth Non Normally Distributed Data

I have ~16,000 probability sets of goals scored [maximum of 12] like below [some % are rounded hence do not add up to 100%]: ...
1
vote
0answers
50 views

Calculating bias and variance in a LOESS fit

I have datasets that can form several different curvy patterns between the dependent and independent variables. The 'true' relationship likely depends on a large number of factors that aren't easily ...
1
vote
0answers
36 views

Time Series Shocks with Exponential Decay

Imagine a piano key played in an auditorium: The amplitude of the sound wave is perhaps highest in the first milliseconds, then slowly decays to zero if no other notes are played. If other notes are ...
0
votes
1answer
100 views

Alternate distance metrics for two time series

I have time-series data of different houses. Assume it is power consumption data. Now, I want to cluster the houses following similar power consumption pattern utmost. So, the various distance metrics ...
0
votes
1answer
105 views

R forecast ets initial level and trend

I am trying to understand how the ets function of R in the forecast package computes initial level $l_0$, and initial trend $b_0$. I was under the impression that they are set to the intercept and ...
1
vote
0answers
17 views

How to deal with frequencies that don't appear in the held-out set?

The held out probability is defined as $$P_{HO}\left(x\right)=\frac{t_{r}}{N_{r}\cdot\left|S^{H}\right|}$$ where: $t_{r}$ is the total number of times events that appeared $r$ times in the training ...
1
vote
0answers
7 views

Fast bivariate smoother?

I need a nonparametric regression for two explanatory variables that is faster than GAM. lm = gam(X3 ~ s(X1,X2,k=8),data=dataset) If the sample size is about 125,...
0
votes
0answers
23 views

Fit mixture of distributions vs. smoothing to time series data in R

I have a time series data and I want to represent this as in the right side of below plot: Authors of the paper claim that they fit a mixture of models to raw power data using mclust function. I ...
2
votes
1answer
65 views

What methods can be used to pick the optimal kernel smoothing width parameter

Kernel smoothers for a function f(x) usually have a parameter which control the width of the region which is used to smooth the value of the function, say at a ...
6
votes
0answers
99 views

When to choose GAM over LOESS? [duplicate]

I am smoothing the relationship between a binary variable y and a continuous variable x. For this I've looked into both GAM and ...
1
vote
0answers
50 views

Influence matrix in Cross validation

I am trying to implement cross validation to find the apropriate smoothing parameter value for smoothing spline I am able to implement the leave out one , in which one data point is left out , then ...
0
votes
0answers
74 views

Use Kalman Filter to post-process an elevation profile?

I'm so sorry that I haven't studied mathematics or physics. Now I have to smooth an elevation profile (the altitudes of a GPX track), because there are really nasty outliers. I've implemented a ...
0
votes
0answers
111 views

Robustness Weights in LOESS behaving strangely

I've been playing around with writing my own LOESS module in Python (2 reasons: first, I wanted the practice, and second, the implementation in statsmodels doesn't ...
1
vote
1answer
153 views

LOESS for subset of data

I want to analyze the performance of two classifiers. For that I have a dataset with 30000 observations that each have an independent variable interactions and a ...
2
votes
0answers
49 views

How to smooth standard deviations?

So, I have some idea how to smooth a set of points to produce a smooth estimate of the conditional mean. Suppose that I want to estimate the standard deviation locally. That is, I have a set of ...
2
votes
0answers
78 views

centering constraints on ti() terms in MGCV

I have some raw data on which I compute percent changes, and I calculate rolling averages of two different lengths on the percent changes. I want to use a tensor interaction of the two rolling ...
2
votes
0answers
72 views

Truncated power basis function and continuity in b-splines

I do not understand how adding a truncated power basis function leads to continuity in B-Splines. Could someone please provide a low level explanation?
0
votes
0answers
24 views

how forecast with penalized b-spline

I have inflasi data, and i will forecast future with penalized b-spline, but I have problem after having found lambda. Do you know syntax to uose in SAS for forecasting values from a penalized b-...
3
votes
1answer
109 views

why add one in inverse document frequency

My textbook lists the idf as $log(1+\frac{N}{n_t})$ where $N$: Number of Documents $n_t$: Number of Documents containing term $t$ Wikipedia lists this formula as a smoothed version of the actual $...
2
votes
1answer
86 views

Does a radial basis function network work in high dimensions?

It seems that a single-layer radial basis function network with normalized weights is the same thing as kernel smoothing (see e.g. Haykin Neural Networks: a Comprehensive Foundation, Section 5.12). ...