Questions tagged [smoothing]

Smoothing methods in data analysis, like splines or kernel smoothers, also regression smoothers like lowess.

Filter by
Sorted by
Tagged with
0 votes
0 answers
8 views

How does Kneser Ney estimate ngrams with BOS without dividing by zero?

The recursive formula for (unmodified) Kneser Ney smoothing is (per Jurafsky08 3.40) $P_{KN}(w_i | h) = \frac{\text{max}(c_{KN}(h\ w_i) - d, 0)}{\sum_v c_{KN}(h \ v )} + \lambda(h) P_{KN}(w_i | h)$ ...
user2740's user avatar
  • 1,366
0 votes
0 answers
13 views

Assessing probability that one set of measurements extends the other with Kalman smoother

I have two sets of N-dimensional measurements following each other with a certain time gap in between. Let's name those sets $A$ and $B$, respectively. All observations have constant Gaussian white ...
emerte's user avatar
  • 1
0 votes
0 answers
12 views

Why would you smooth a logical variable in a GAM?

I just used the gam.scope function in the R-package gam to create possible scopes of each explanatory variable of my model to ...
Lena Ortega's user avatar
4 votes
1 answer
104 views

Is it possible to use smooth functions as part of a nonlinear regression?

Background I am fitting nonlinear regressions with a single response and two predictors. I know the relationship between the response and each of the predictors, but I do not know how they interact. I ...
mkt's user avatar
  • 17.2k
0 votes
1 answer
41 views

Inferences with Filtered and Smoothed state estimates from a tracking problem

I am working on a synthetic tracking problem in a two-dimensional space, where the start and end positions are known and noisy measurements of the state variables are given at discrete time steps. As ...
Carlo Berger's user avatar
3 votes
1 answer
171 views

How does lowess handle gaps in time series?

I have been using the lowess smoother to calculate trends for time series data for a while now but until now my data was always without gaps. I now have to work with data where there are quite large ...
Krautsultan's user avatar
1 vote
1 answer
50 views

Understanding Identifiability problem of multiple smooths in GAMs/Additive models

I am a beginner into additive models and GAMs, but have good enough knowledge of linear regression. I was going through this wikipedia article to understand more, but can't seem to understand the ...
user101874's user avatar
1 vote
0 answers
52 views

Smoothing of GPS tracks - remove noise and stop-go clusters

I know there are several posts about this, but I could not find exactly what I need. I have GPS track data (from an underwater vehicle) for short intervals of 1 second (time-stamps on data). The data ...
Mona's user avatar
  • 11
1 vote
0 answers
48 views

Functional Data Analysis with Missing Data

I am trying to analyze a dataset containing several observations for n = 27. Unfortunately, not every participant answered the question every time, i.e., when plotting the graph it has holes in it. I ...
Soph's user avatar
  • 11
2 votes
0 answers
39 views

GAM partial response plot interpretation

I've made the gam in the code below (in R), but I'm struggling to interpret the results. Specifically, the partial response plots for all but one of the variables is linear, and the CI lines cross in ...
blitz1259's user avatar
0 votes
0 answers
24 views

Smoothing without losing data points

How would I smooth a series of data without losing the ends of a series? For example, if I have $i_1, i_2, \dots, i_n$, and I apply a moving average, I lose the start and end of the series ($i_1$ and $...
user389220's user avatar
1 vote
1 answer
92 views

Kernel Smoothing for Time Series data

I have generated a time series data set of measurements that are a bit noisy and I want to apply kernel smoothing to the data. My time series data is not regular however, meaning that the time ...
Jade131621's user avatar
4 votes
1 answer
268 views

How 'by' factor works with 'fs' random smooth in gam?

I've a large dataset including a response bmk, a continuous predictor delay, a group factor (...
denis's user avatar
  • 85
1 vote
0 answers
25 views

Is there a statistically sound method of smoothing a data series without removing the edges?

Recently I've been plotting a lot of data, and often I find myself using a moving average to smooth out values that oscillate or otherwise fluctuate a lot. However, the problem with this is that it ...
Outis Nemo's user avatar
0 votes
1 answer
62 views

Assessing differences in slope of a loess (or any smoother) fit

I have continuous data for x (for example "time") and y with several groups. I would like to show/test that the slope of groupC changes over time (is positive and rises) while groups A and B ...
ATpoint's user avatar
  • 29
3 votes
1 answer
126 views

Gam plots y-axis range is different for model that that from predict function

I got some idea about gam (generalized additive model) plot from a course of Noam Ross. However, I have several confusion and got stucked. I fit a logistic gam and can interpret by using ...
Ly T's user avatar
  • 41
3 votes
1 answer
39 views

The difference between smoothing problems in stochastic processes and statistics

I am taking a class and we are about to talk about Kalman filters and smoothing and I am trying to do some reading ahead. On the Wikipedia page for 'Smoothing (stochastic processes)' it says 'the ...
yungtoad's user avatar
  • 131
5 votes
1 answer
178 views

Spline basis explicitly including a linear term; basis functions generated by the default call to "s()" function of mgcv package

I'm curious about the basis functions generated by the call to "s()" function with default parameter values, but even more specifically I'm curious about a smoother for a single variable ...
UsDAnDreS's user avatar
  • 139
2 votes
1 answer
66 views

Why smoothCon gives different estimates of smooths than mgcv?

I am learning gam by myself. I tried to find posts that can help me to finding and understanding smooth functions i.e., basis functions from gam using mgcv. From a ...
user avatar
1 vote
1 answer
78 views

GAM: covariate specific fits do not appear to match approximate significance of smooth terms

I have fit a GAM with two continuous independent variables and one discrete covariate with two levels ("gray" and "brown"). All variables are scaled and the model runs fine: ...
mluerig's user avatar
  • 681
2 votes
1 answer
42 views

How is bandwidth and roughness penalty in nonparametric regression connected?

Nonparametric regression, or smoothing, concerns finding a function that smooths the data, and appears in additive models or generalized additive models. You could say we are trying to fit the model $$...
cgmil's user avatar
  • 1,313
3 votes
0 answers
103 views

When fitting a generalised additive model, how to choose how much to smooth?

When fitting a GAM, is there a rule (of thumb) for deciding if $k$ (max number of degrees of freedom for a spline) is large enough or not? How much should edf be below $k'$? And is that an absolute ...
user34927's user avatar
  • 229
2 votes
0 answers
32 views

How do professional statisticians evaluate or mathematically justify choice of one smoothing method from another? [closed]

I would like to fit some economic data and evaluate it. Usually I used only regression techniques, but recently I found out about smoothing techniques which could be used to diminish noise in dataset ...
SCP_org's user avatar
  • 21
2 votes
1 answer
30 views

Curve quantification

I have some longitudinal measurement data of 15,000. I smoothed that data with B-spline smoothing and got the following curve. I then want to quantify this curve and extract features for clustering ...
NakataKoo's user avatar
0 votes
0 answers
25 views

How to determine periodicity (autocorrelation) in the 2d projection of a 3D helical trajectory

Some microscopic objects are helically moving in 3d and we acquisite their moving trajectories as a set of 2d points. The obtained trajectories are shown on the image. For each of these trajectories ...
Ivan Z's user avatar
  • 101
4 votes
2 answers
283 views

Rank of smoothing matrix

The solution to the minimizing problem: $$RSS(f, \lambda) = \sum_{i=1}^n (y_i - f(x_i))^2 + \lambda \int (f''(t))^2 dt$$ is written in matrix form as $f = N\theta = \sum_{j = 1}^n N_j(x_i) \theta_j$ ...
Lejoon's user avatar
  • 175
0 votes
0 answers
23 views

What are other algos like Friedman's non parametric super smoother?

Friedman's super smoother is a non parametric smoother. http://fmwww.bc.edu/RePEc/bocode/s/supsmooth_doc.pdf What are other algo's that are non parametric smoothers as well?
SatoshiReport's user avatar
0 votes
0 answers
261 views

Timeseries - How are the window/span size and 'frac' in LOWESS related?

Many software implementations of locally weighted scatterplot smoothing (LOWESS), such as those in Python and R, require a frac parameter as an input. I am confused as to how frac is mathematically ...
Kartik's user avatar
  • 3
1 vote
1 answer
619 views

Is it possible to regularize a covariance matrix?

I have many "parallel" time-series (about 100). They have relatively short history. I calculate a covariance matrix between these time-series. Now, I believe that the observed covariance ...
Roman's user avatar
  • 544
0 votes
0 answers
32 views

Estimating factor smoother interactions using GAM but factors do not have equal range

I would like to estimate a depth effect on species abundance for each site, thus a factor-smooth interaction term in the model. Using R mgcv package, the model is: ...
tiantianchen's user avatar
  • 2,041
0 votes
0 answers
45 views

Hat matrix for penalized spline

Suppose the penalized spline given by equation $$ \hat{x}=(W+\lambda_{1}D^{T}D+2\lambda_{2})^{-1}\left[Wy+\lambda_{2}(A+B)\right], $$ where $\lambda_{1},\lambda_{2}$ are positive constants and $A,B$ ...
justik's user avatar
  • 101
6 votes
2 answers
968 views

Smoothing spline seems to fit too precisely?

I am trying to fit some time series data to a smoothing spline in R. However, it seems like the spline is fitting the data too perfectly, meaning overfitting. I was trying to figure out what settings ...
krishnab's user avatar
  • 1,502
3 votes
1 answer
910 views

How to decide which variables to smooth in GAM

When specifying it's formula GAM has s function for smoothing. Let's say I want to fit mtcars. I can do the following: Approach 1: ...
littleworth's user avatar
2 votes
0 answers
158 views

How to choose the smoothing method and factors that best capture the trend in data?

I have a time-series data that I will be using to forecast the future using machine learning algorithms such as LSTM. Prior to the forecast, I need to smooth the data to remove the noise and capture ...
Traveling Salesman's user avatar
2 votes
1 answer
78 views

What reasons beyond interpretability are there to use additive models over a complex, multivariate smoother?

Let's adopt for the second the notation from the R package mgcv. ...
cgmil's user avatar
  • 1,313
1 vote
0 answers
36 views

Can do smoothing after making time-series data stationary

Can do smoothing after making time-series data stationary? And is it useful to do smoothing before making data stationary?
Ruslan Pylypyuk's user avatar
4 votes
2 answers
101 views

Determining spline basis dimension using Wood's statistical test

In Simon Wood's book Generalized Additive Models (2nd ed.) on page 243, he describes the following procedure for checking that the basis dimension is too small: Fortunately informal checking that the ...
cgmil's user avatar
  • 1,313
1 vote
1 answer
642 views

Change y-axis value in R GAM plot from smoothed to actual response variable

I know similar questions have been asked before, but I've looked through them all and I'm still confused, in part because the other examples seem to be more complicated, so thought I'd throw mine up. ...
GRSB.1's user avatar
  • 25
1 vote
1 answer
125 views

GAM - setting K-values before or after testing the different models

I'm working with GAM and I'm testing different models with and without certain variables, and I need to set k-values for the different smoothers. Do I need to use the exact same k-values prior to ...
Lotte's user avatar
  • 11
0 votes
1 answer
130 views

Why does Friedman's Supersmoother predict using $X$ instead of $y$?

I am working on friedman's supersmoother with python notebook: https://notebook.community/moreati/supersmoother/examples/Supersmoother. In this example, data is generated randomly: ...
user3104352's user avatar
1 vote
0 answers
148 views

Intuition for bandwidth and degrees of freedom in kernel smoothers

For Kernel smoothers such as local polynomial regression smoothers (the Nadaraya-Watson smoother), we consider $y = m(x) + \epsilon$, for $m(x)$ some smooth function we are trying to estimate and $\...
Dan Phillips's user avatar
0 votes
0 answers
867 views

plot.gam partial effects plot: is there another function to plot more than one variable?

The package description of plot.gam describes under Warning: "The function can not deal with smooths of more than 2 variables!" My gam model has a base model that includes habitat variables, ...
frog_dan's user avatar
1 vote
1 answer
99 views

Splines free of Gibbs phenomenon for smoothing

I'm trying to do 3-dimensional smoothing of some observations we made. To simplify things, here's a visualisation of one slice of the data we have: Note that the observations are gridded and contain ...
Raphael's user avatar
  • 314
1 vote
0 answers
38 views

Confidence Interval for Centered Moving Average of Autocorrelated Timeseries Data (Smoothing)

I want to compute the confidence intervals of moving average (MA) which I apply to a time series with both high-frequency and low-frequency terms. I'm interested in the slowly changing part only (...
nicrie's user avatar
  • 141
3 votes
1 answer
391 views

Is the modeling strategy of GAM in MGCV equivalent to ridge regression when there are no smoothing terms?

According to GAM, it utilizes a penalized likelihood, which is maximized by penalized iteratively re-weighted least squares (P-IRLS), to obtain parameter estimations. The likelihood is defined as: ...
Chinyako's user avatar
0 votes
1 answer
74 views

How are the penalized splines defined here?

Based on 'Semiparametric Regression with R' (https://link.springer.com/chapter/10.1007%2F978-1-4939-8853-2_1), a penalized spline $$ f(x)=\beta_{0}+\beta_{1} x+\sum_{k=1}^{K} u_{k}\left(x-\kappa_{k}\...
buenavista62's user avatar
0 votes
1 answer
161 views

How to show that the gradient of the smoothed surrogate loss function leads to perceptron update?

This is about the contents of section 1.2.1 and 1.2.1.1 of the book "Neural Networks and Deep Learning: A Textbook". The link to the sections is here. The question arises from the following ...
zzzhhh's user avatar
  • 303
5 votes
1 answer
67 views

What is the name of this kind of smoothing?

Essentially, the code below produces something that is similar to a running mean, but instead of 0/1 weights on a window, it has gaussian weights centered at the x-position in consideration. I'm ...
teunbrand's user avatar
  • 151
1 vote
1 answer
366 views

Online time-series smoothing algorithm for sparse data

I am working on building a real-time system for processing and aggregating somewhat sparse and irregular survey measurements (ranges from 0-100, usually on the order of 20-100 measurements). I am ...
Amrit Baveja's user avatar
0 votes
1 answer
219 views

Number of knots in smoothing splines, residual plots and sample size?

I'm working with a dataset of 56 samples, so I am trying to keep the complexity of the regression model down. However, I have rather complex non linear relationships between some of the predictors and ...
hanne's user avatar
  • 3

1
2 3 4 5
8