Questions tagged [smoothing]
Smoothing methods in data analysis, like splines or kernel smoothers, also regression smoothers like lowess.
338
questions
0
votes
0answers
11 views
Is it better to smooth non-positive definite matrices or drop items that are problematic?
We are conducting a factor analysis in R, beginning by estimating mixed correlation matrices using the psych package. When doing so, we get the following warning: "Matrix was not positive ...
0
votes
0answers
8 views
Choose the right Sigma for Gaussian low-pass filter
I have the following problem:
I have a time series with counted data. I now want to smooth it using a Gaussian low-pass filter.
Is there a method to determine the sigma value? The window should have a ...
0
votes
0answers
12 views
Is alpha of 1 (smoothing constant) appropriate?
I have 252 observations starting from 2020-01-02 to 2020-12-31 and have plotted the time series using
...
0
votes
0answers
37 views
Determine family in GAMs of negative values
I have these values (negative and positive) and I want to determine the nonlinear relationship between variable and predictor using generalized additive models (GAMs).
...
1
vote
0answers
41 views
Estimating difference smooth effects for each level of a factor - WHY?
I am wondering why you would want to use a by variable smooth s(time, by= x)? (time is non-linear hence why I am using GAMs)
I am using GAMs to explore if my ...
1
vote
2answers
106 views
How to show that smoothing spline fit preserves the local regression part of the fit
We need to show that a smoothing spline of $y_i$ to $x_i$ retains the local regression part of the fit.
For linear regression, this problem seems trivial because it is relatively easy to move from $...
0
votes
1answer
82 views
What do the values for initialization method mean in statsmodels simple exponential smoothing?
I'm trying to use Statsmodels' simple exponential smoothing for time series analysis.
There are various methods available for initializing the recursions (estimated, heuristic, known).
Can someone ...
0
votes
0answers
14 views
Terminology for smoothing methods: kernel density estimation, kernel smoothing, etc
I'm using what I have been referring to as 'kernel density estimation' to estimate the rates of a series of variables a, b, c from noisy observations distributed in ...
0
votes
0answers
34 views
Fitting a GAM to a small dataset, most suitable k, smoother type?
I am a part-time M.Sc candidate that has been advised to apply a GAM to my temperature reconstructions instead of a LOESS smoother. I have read a lot of papers on GAM, have watched some webinars on it ...
0
votes
0answers
17 views
Is there a way to do such kind of smoothing for log likelihood?
Let's say we have a sequential decision making problem. At each step, we need to make a decision, and the decision made in this step will determine the possible actions for the next step.
Now I have a ...
0
votes
1answer
25 views
Moving average (running mean) - how to keep all observations in smoothed time-series?
Let's suppose I have a time-series of 100 daily values and I want to compute a 5-day moving average of this time series.
I would do as follows:
...
2
votes
0answers
35 views
Roughness penalty matrix for the Fourier basis functions
I am trying to understand how the roughness penalty matrix for the Fourier basis functions is calculated in R using the fda ...
0
votes
0answers
13 views
How can I smooth the predictions of a supervised model by learning its trend?
This is the predictions of a binary classification model. The model is doing predicitons continuously, and these values are the sum of positive labels during a 10 hours period. As you can see, some of ...
1
vote
0answers
33 views
Reference for spar parameter in R's implementation of the smoothing spline
In the R's implementation of the smoothing spline which is smooth.spline function, there is a parameter, called spar that ...
1
vote
0answers
10 views
Smoothing of a surface [closed]
A land surveyor made a topographic map of a property in the hills by measuring the elevation at different points. The x-y coordinates are more or less "random", meaning no regular grid:
...
0
votes
0answers
23 views
Trend estimation: strange behaviour of loess
I am trying to estimate the trend of a time series. I have used a non-robust loess smoothing with a window size of 30 points. Looking at the raw data, it is clear that there is no increase at the end ...
1
vote
2answers
78 views
What is the lag associated with Moving Average smoothing?
In a tutorial I came across this:
"Recall that the forecast value is: $\hat{y}_{t+1} = \frac{y_t + y_{t-1} + ... + y_{t-m+1}}{m}$
It's worth pondering that formula for a minute. While easy to ...
0
votes
0answers
21 views
Automatic selection of smoothing parameter with time series
Cross validation allows to automatically select a reasonable smoothing parameter for a given data set, avoiding underfitting and overfitting. Cross validation requires independent data, an assumption ...
2
votes
2answers
58 views
How is this seven day rolling average calculated?
Eric Topol posted the below on Twitter claiming that Europe is "turning COVID around" based on the trend in the past <7 days. However, while the case trend hitherto appears smooth, I was ...
1
vote
1answer
166 views
How to prove the LOOCV formula for smoothing matrix?
When I read The Elements of statistical learning, in the section of "5.5 Automatic Selection of the Smoothing Parameters", equation (5.26) gives a LOOCV formula for corresbonding $\lambda$
...
1
vote
1answer
34 views
How to make predictions using smoothing splines
In ordinary least squares regression, for outcome vector $y$ and design matrix $X$ (full rank), the estimated coefficient values are $\hat{\beta} = (X^TX)^{-1}X^TY$. Given a new set of covariates $X_{...
1
vote
0answers
61 views
STL decomposition of a daily time series (only business days)
I'm currently working with a daily (business days) time series which has a monthly seasonality and an overall positive trend over the last two years. I want to estimate the error component of the time ...
5
votes
2answers
82 views
Is smoothing an appropriate solution to deal with model diagnostics in a GAMLSS?
I have just recently started using GAMLSS models (after being pointed in that direction in this question), and I'm wondering wether it's 'legit' to use smoothing (i.e. cubic splines in my case) to ...
2
votes
1answer
40 views
Smoothing technique that takes into account subtotals
I have a simple task of fitting a smooth line with monthly points across six years. The line doesn't need to go through any specific point, but the yearly totals need to be specific values.
Is there a ...
0
votes
0answers
24 views
Biase of ASE estimation Kernel Regression
I'm trying to calculate the bias of the estimator $p(h)=n^{-1}\displaystyle\sum_{i=1}^{n}(Y_{j}-\hat{m}_{h}(X_{j})^{2}w(X_{j})$ of the averaged squared error. The result I find in the literature is ...
0
votes
0answers
9 views
What's the good way to find outliers in the continuous physical process measurements time-series? (picture attached) [duplicate]
What's the best way to find outliers in the time-series like the one attached?
Just using the knowledge that it is a measurement of a continuous physical process. The data is quite short (200-500 ...
3
votes
1answer
64 views
How to fit a smooth polynomial?
We have $N$ samples of the unknown function $f(x)$ on the finite interval $[a, b]$. The samples are subject to white noise of known variance. We want to approximate the function $f(x)$ by a polynomial ...
0
votes
1answer
38 views
Quantify smoothness of vector field
I have different vectors field and I want to characterize their "smoothness" or tendency to align, or like their heterogeneity. Here are two examples, a relatively smooth vector field on the ...
0
votes
0answers
26 views
How is the smoothing spline penalty computed in practice?
I'm digging into smoothing splines and finding good resources, but no one talks about how to actually calculate the penalty $\int \hat{f}^{"}(x)^2 dx$ in the standard smoothing spline loss:
Since <...
1
vote
1answer
65 views
What's the best way to find outliers in the time-series, encountering that it is a real-world mechanical process (process continuity)?
What's the best way to find outliers in the time-series, encountering continuity?
I attached two time-series that I'm interested to filter. One is less noisy, and one is a bit noisier. I'm mostly ...
2
votes
1answer
77 views
Smoothing algorithm for anomalies
To construct a plot, I'm looking for an algorithm which can handle inf and (very) negative values.
If I have infinity values everything is a line but not infinities.
Example
...
1
vote
0answers
36 views
Hyper-parameter optimization for regression to avoid overfiting/underfiting
I am using Penalized spline to smooth noisy data. Those splines are non parametric regression models which only rely on a smoothing parameter $\lambda \geq 0$ (which has to be chosen).
I would like to ...
0
votes
0answers
19 views
How choose smoothing function for Generative Additive Models? (GAMs)
I would like to know which strategies are used in practice to choose the correct smoothing function for each features in the GAMs, both for classification and regression tasks. I thought of plotting ...
1
vote
3answers
149 views
How to avoid the ending drop of the moving average curve?
I'm playing around with the built-in matlab movmean function which, using default options, creates the simple and centered moving average of given data (I'm pretty ...
1
vote
0answers
41 views
Smooth Curve Fit
I uses supsmu to fit a smooth curve to a scatter plot of data .
I am quite new to curve fitting, but my graph looks somewhat sinusoidal. What would you recommend in terms of next steps to determine a ...
0
votes
1answer
48 views
How to smooth an existing PDF?
I've generated a PDF of binned data using the python package binsmooth. The PDF is plotted in the following image:
I am trying to smooth the PDF so as to provide a more intuitive interpretation of ...
0
votes
0answers
34 views
Smoothing methods for unevenly sampled data
I have categorical, time-series data distributed in space. It is very noisy, but over the whole series there are big shifts in distribution - my goal is to see how these shifts progressed in space. So,...
1
vote
0answers
54 views
Multidimensional Smoothing spline in R
How create multidimensional smoothing spline for regression in R? Is there a function to implement thin-plate spline or tensor product?
I see the function ssanova() based on tensor product but I don't ...
2
votes
1answer
84 views
What does low rank smoother mean?
I read a paper on thin plate splines that stated several advantages. Thin plate splines:
Do not rely on ’knots’ as many other spline methods do
Are of isotropic character (i.e. unaffected by the ...
1
vote
0answers
24 views
Prediction with smooth.spline [closed]
I am a bit unsure how the newdata= argument is set in the prediction function. Here I have found the optimal degree of freedom to be 4 via 10-fold CV. I have fitted ...
0
votes
0answers
125 views
Smoothing spline with df chosen by 10-fold CV
So instead of using the (LOOCV) cv argument in smooth.spline(), I would like to use a block of 10-fold CV code to find the optimal degree of freedom.
I find that the smooth.spline model is somewhat ...
1
vote
0answers
53 views
What smoothing parameter makes sense for a LOESS calibration curve?
I am creating a calibration curve to asses the fit of a logistic regression.
Does it make more sense to use the local or global optimum smoothing parameter for the LOESS line?
The orange line uses ...
0
votes
0answers
104 views
Choosing between additive and multiplicative Holt-Winters Model
While using the Holt-Winters model for seasonality, I am unable to choose a better fit between additive and multiplicative models. I used to look at RMSE value and choose the one with the lower RMSE. ...
1
vote
1answer
54 views
Confused about the cubic regression basis in the R package mgcv
To understand the cubic regression basis constructed in the R package mgcv, I plotted out the 5 bases generated within [0,1]:
...
1
vote
0answers
29 views
Using prior information for a GAM smooth function to reduce standard errors
I have some data that I want to model in a GAM. However, there are few observations, generally leading to high standard errors.
...
0
votes
1answer
110 views
What is the correct mgcv syntax for interacting a smooth with an interaction of linear predictors?
I have a question about mgcv's formula syntax for interactions with smooths s() (I am actually using this syntax within ...
2
votes
3answers
238 views
How to avoid smoothing to 0 at edges of R density plot
When using the density function in R, it includes smooth transitions down to 0 at both ends of the data. Is there a way to prevent this? As a trivial example, ...
0
votes
1answer
118 views
Smoothing a binned averages
I am trying to smooth some binned data. I have a discrete variable X which might best be thought of as time and a continuous variable Y. I want to know the average Y value for each value of X and this ...
1
vote
1answer
33 views
How to measure smoothness of inputs over outputs?
I know similar questions have been asked for time series data. But my question is a little bit different.
Consider that we have input dataset $X \in R^{N \times M}$, where $M$ is the dimension of ...
1
vote
0answers
42 views
Smoothing 3D motion tracking data [closed]
I'm working on an application that tracks an object's position in 3D space through time. The hardware generates (x,y,z,t) data points, with a non-uniform sampling ...
