Questions tagged [smoothing]
Smoothing methods in data analysis, like splines or kernel smoothers, also regression smoothers like lowess.
389
questions
0
votes
0
answers
8
views
How does Kneser Ney estimate ngrams with BOS without dividing by zero?
The recursive formula for (unmodified) Kneser Ney smoothing is (per Jurafsky08 3.40)
$P_{KN}(w_i | h) = \frac{\text{max}(c_{KN}(h\ w_i) - d, 0)}{\sum_v c_{KN}(h \ v )} + \lambda(h) P_{KN}(w_i | h)$
...
- 1,366
0
votes
0
answers
13
views
Assessing probability that one set of measurements extends the other with Kalman smoother
I have two sets of N-dimensional measurements following each other with a certain time gap in between. Let's name those sets $A$ and $B$, respectively. All observations have constant Gaussian white ...
- 1
0
votes
0
answers
12
views
Why would you smooth a logical variable in a GAM?
I just used the gam.scope function in the R-package gam to create possible scopes of each explanatory variable of my model to ...
4
votes
1
answer
104
views
Is it possible to use smooth functions as part of a nonlinear regression?
Background
I am fitting nonlinear regressions with a single response and two predictors. I know the relationship between the response and each of the predictors, but I do not know how they interact. I ...
- 17.2k
0
votes
1
answer
41
views
Inferences with Filtered and Smoothed state estimates from a tracking problem
I am working on a synthetic tracking problem in a two-dimensional space, where the start and end positions are known and noisy measurements of the state variables are given at discrete time steps. As ...
3
votes
1
answer
171
views
How does lowess handle gaps in time series?
I have been using the lowess smoother to calculate trends for time series data for a while now but until now my data was always without gaps.
I now have to work with data where there are quite large ...
- 137
1
vote
1
answer
50
views
Understanding Identifiability problem of multiple smooths in GAMs/Additive models
I am a beginner into additive models and GAMs, but have good enough knowledge of linear regression. I was going through this wikipedia article to understand more, but can't seem to understand the ...
- 25
1
vote
0
answers
52
views
Smoothing of GPS tracks - remove noise and stop-go clusters
I know there are several posts about this, but I could not find exactly what I need. I have GPS track data (from an underwater vehicle) for short intervals of 1 second (time-stamps on data). The data ...
- 11
1
vote
0
answers
48
views
Functional Data Analysis with Missing Data
I am trying to analyze a dataset containing several observations for n = 27. Unfortunately, not every participant answered the question every time, i.e., when plotting the graph it has holes in it. I ...
- 11
2
votes
0
answers
39
views
GAM partial response plot interpretation
I've made the gam in the code below (in R), but I'm struggling to interpret the results. Specifically, the partial response plots for all but one of the variables is linear, and the CI lines cross in ...
- 21
0
votes
0
answers
24
views
Smoothing without losing data points
How would I smooth a series of data without losing the ends of a series? For example, if I have $i_1, i_2, \dots, i_n$, and I apply a moving average, I lose the start and end of the series ($i_1$ and $...
1
vote
1
answer
92
views
Kernel Smoothing for Time Series data
I have generated a time series data set of measurements that are a bit noisy and I want to apply kernel smoothing to the data. My time series data is not regular however, meaning that the time ...
- 111
4
votes
1
answer
268
views
How 'by' factor works with 'fs' random smooth in gam?
I've a large dataset including a response bmk, a continuous predictor delay, a group factor (...
- 85
1
vote
0
answers
25
views
Is there a statistically sound method of smoothing a data series without removing the edges?
Recently I've been plotting a lot of data, and often I find myself using a moving average to smooth out values that oscillate or otherwise fluctuate a lot. However, the problem with this is that it ...
- 111
0
votes
1
answer
62
views
Assessing differences in slope of a loess (or any smoother) fit
I have continuous data for x (for example "time") and y with several groups. I would like to show/test that the slope of groupC changes over time (is positive and rises) while groups A and B ...
- 29
3
votes
1
answer
126
views
Gam plots y-axis range is different for model that that from predict function
I got some idea about gam (generalized additive model) plot from a course of Noam Ross. However, I have several confusion and got stucked.
I fit a logistic gam and can interpret by using ...
- 41
3
votes
1
answer
39
views
The difference between smoothing problems in stochastic processes and statistics
I am taking a class and we are about to talk about Kalman filters and smoothing and I am trying to do some reading ahead.
On the Wikipedia page for 'Smoothing (stochastic processes)' it says 'the ...
- 131
5
votes
1
answer
178
views
Spline basis explicitly including a linear term; basis functions generated by the default call to "s()" function of mgcv package
I'm curious about the basis functions generated by the call to "s()" function with default parameter values, but even more specifically I'm curious about a smoother for a single variable ...
- 139
2
votes
1
answer
66
views
Why smoothCon gives different estimates of smooths than mgcv?
I am learning gam by myself. I tried to find posts that can help me to finding and understanding smooth functions i.e., basis functions from gam using mgcv. From a ...
user369390
1
vote
1
answer
78
views
GAM: covariate specific fits do not appear to match approximate significance of smooth terms
I have fit a GAM with two continuous independent variables and one discrete covariate with two levels ("gray" and "brown"). All variables are scaled and the model runs fine:
...
- 681
2
votes
1
answer
42
views
How is bandwidth and roughness penalty in nonparametric regression connected?
Nonparametric regression, or smoothing, concerns finding a function that smooths the data, and appears in additive models or generalized additive models. You could say we are trying to fit the model
$$...
- 1,313
3
votes
0
answers
103
views
When fitting a generalised additive model, how to choose how much to smooth?
When fitting a GAM, is there a rule (of thumb) for deciding if $k$ (max number of degrees of freedom for a spline) is large enough or not? How much should edf be below $k'$? And is that an absolute ...
- 229
2
votes
0
answers
32
views
How do professional statisticians evaluate or mathematically justify choice of one smoothing method from another? [closed]
I would like to fit some economic data and evaluate it. Usually I used only regression techniques, but recently I found out about smoothing techniques which could be used to diminish noise in dataset ...
- 21
2
votes
1
answer
30
views
Curve quantification
I have some longitudinal measurement data of 15,000. I smoothed that data with B-spline smoothing and got the following curve.
I then want to quantify this curve and extract features for clustering ...
- 33
0
votes
0
answers
25
views
How to determine periodicity (autocorrelation) in the 2d projection of a 3D helical trajectory
Some microscopic objects are helically moving in 3d and we acquisite their moving trajectories as a set of 2d points. The obtained trajectories are shown on the image.
For each of these trajectories ...
- 101
4
votes
2
answers
283
views
Rank of smoothing matrix
The solution to the minimizing problem: $$RSS(f, \lambda) = \sum_{i=1}^n (y_i - f(x_i))^2 + \lambda \int (f''(t))^2 dt$$ is written in matrix form as $f = N\theta = \sum_{j = 1}^n N_j(x_i) \theta_j$ ...
- 175
0
votes
0
answers
23
views
What are other algos like Friedman's non parametric super smoother?
Friedman's super smoother is a non parametric smoother. http://fmwww.bc.edu/RePEc/bocode/s/supsmooth_doc.pdf
What are other algo's that are non parametric smoothers as well?
0
votes
0
answers
261
views
Timeseries - How are the window/span size and 'frac' in LOWESS related?
Many software implementations of locally weighted scatterplot smoothing (LOWESS), such as those in Python and R, require a frac parameter as an input. I am confused as to how frac is mathematically ...
- 3
1
vote
1
answer
619
views
Is it possible to regularize a covariance matrix?
I have many "parallel" time-series (about 100). They have relatively short history. I calculate a covariance matrix between these time-series.
Now, I believe that the observed covariance ...
- 544
0
votes
0
answers
32
views
Estimating factor smoother interactions using GAM but factors do not have equal range
I would like to estimate a depth effect on species abundance for each site, thus a factor-smooth interaction term in the model. Using R mgcv package, the model is:
...
- 2,041
0
votes
0
answers
45
views
Hat matrix for penalized spline
Suppose the penalized spline given by equation
$$
\hat{x}=(W+\lambda_{1}D^{T}D+2\lambda_{2})^{-1}\left[Wy+\lambda_{2}(A+B)\right],
$$
where $\lambda_{1},\lambda_{2}$ are positive constants and $A,B$
...
- 101
6
votes
2
answers
968
views
Smoothing spline seems to fit too precisely?
I am trying to fit some time series data to a smoothing spline in R. However, it seems like the spline is fitting the data too perfectly, meaning overfitting. I was trying to figure out what settings ...
- 1,502
3
votes
1
answer
910
views
How to decide which variables to smooth in GAM
When specifying it's formula GAM has s function for smoothing.
Let's say I want to fit mtcars.
I can do the following:
Approach 1:
...
- 141
2
votes
0
answers
158
views
How to choose the smoothing method and factors that best capture the trend in data?
I have a time-series data that I will be using to forecast the future using machine learning algorithms such as LSTM. Prior to the forecast, I need to smooth the data to remove the noise and capture ...
2
votes
1
answer
78
views
What reasons beyond interpretability are there to use additive models over a complex, multivariate smoother?
Let's adopt for the second the notation from the R package mgcv.
...
- 1,313
1
vote
0
answers
36
views
Can do smoothing after making time-series data stationary
Can do smoothing after making time-series data stationary? And is it useful to do smoothing before making data stationary?
4
votes
2
answers
101
views
Determining spline basis dimension using Wood's statistical test
In Simon Wood's book Generalized Additive Models (2nd ed.) on page 243, he describes the following procedure for checking that the basis dimension is too small:
Fortunately informal checking that the ...
- 1,313
1
vote
1
answer
642
views
Change y-axis value in R GAM plot from smoothed to actual response variable
I know similar questions have been asked before, but I've looked through them all and I'm still confused, in part because the other examples seem to be more complicated, so thought I'd throw mine up. ...
- 25
1
vote
1
answer
125
views
GAM - setting K-values before or after testing the different models
I'm working with GAM and I'm testing different models with and without certain variables, and I need to set k-values for the different smoothers. Do I need to use the exact same k-values prior to ...
- 11
0
votes
1
answer
130
views
Why does Friedman's Supersmoother predict using $X$ instead of $y$?
I am working on friedman's supersmoother with python notebook: https://notebook.community/moreati/supersmoother/examples/Supersmoother.
In this example, data is generated randomly:
...
- 101
1
vote
0
answers
148
views
Intuition for bandwidth and degrees of freedom in kernel smoothers
For Kernel smoothers such as local polynomial regression smoothers (the Nadaraya-Watson smoother), we consider $y = m(x) + \epsilon$, for $m(x)$ some smooth function we are trying to estimate and $\...
- 421
0
votes
0
answers
867
views
plot.gam partial effects plot: is there another function to plot more than one variable?
The package description of plot.gam describes under Warning: "The function can not deal with smooths of more than 2 variables!" My gam model has a base model that includes habitat variables, ...
- 59
1
vote
1
answer
99
views
Splines free of Gibbs phenomenon for smoothing
I'm trying to do 3-dimensional smoothing of some observations we made. To simplify things, here's a visualisation of one slice of the data we have:
Note that the observations are gridded and contain ...
- 314
1
vote
0
answers
38
views
Confidence Interval for Centered Moving Average of Autocorrelated Timeseries Data (Smoothing)
I want to compute the confidence intervals of moving average (MA) which I apply to a time series with both high-frequency and low-frequency terms.
I'm interested in the slowly changing part only (...
- 141
3
votes
1
answer
391
views
Is the modeling strategy of GAM in MGCV equivalent to ridge regression when there are no smoothing terms?
According to GAM, it utilizes a penalized likelihood, which is maximized by penalized iteratively re-weighted least squares (P-IRLS), to obtain parameter estimations. The likelihood is defined as:
...
- 41
0
votes
1
answer
74
views
How are the penalized splines defined here?
Based on 'Semiparametric Regression with R' (https://link.springer.com/chapter/10.1007%2F978-1-4939-8853-2_1), a penalized spline
$$
f(x)=\beta_{0}+\beta_{1} x+\sum_{k=1}^{K} u_{k}\left(x-\kappa_{k}\...
0
votes
1
answer
161
views
How to show that the gradient of the smoothed surrogate loss function leads to perceptron update?
This is about the contents of section 1.2.1 and 1.2.1.1 of the book "Neural Networks and Deep Learning: A Textbook". The link to the sections is here. The question arises from the following ...
- 303
5
votes
1
answer
67
views
What is the name of this kind of smoothing?
Essentially, the code below produces something that is similar to a running mean, but instead of 0/1 weights on a window, it has gaussian weights centered at the x-position in consideration.
I'm ...
- 151
1
vote
1
answer
366
views
Online time-series smoothing algorithm for sparse data
I am working on building a real-time system for processing and aggregating somewhat sparse and irregular survey measurements (ranges from 0-100, usually on the order of 20-100 measurements). I am ...
- 113
0
votes
1
answer
219
views
Number of knots in smoothing splines, residual plots and sample size?
I'm working with a dataset of 56 samples, so I am trying to keep the complexity of the regression model down. However, I have rather complex non linear relationships between some of the predictors and ...
- 3