Questions tagged [smoothing]
Smoothing methods in data analysis, like splines or kernel smoothers, also regression smoothers like lowess.
367
questions
0
votes
0
answers
9
views
Smoothing temperature data using python
Hey there I have a datalogger which logs my temperature (internally and externally) as well as the humidity using a DHT22 sensor on a pizero wh.
My current temperature graph (no smoothing) looks like ...
- 101
0
votes
0
answers
20
views
The correct pre-processing of time-series data when using LSTM
I have a time-series data that I would like to use for forecasting the data trend using LSTM. I followed
https://towardsdatascience.com/the-complete-guide-to-time-series-analysis-and-forecasting-...
2
votes
1
answer
33
views
What reasons beyond interpretability are there to use additive models over a complex, multivariate smoother?
Let's adopt for the second the notation from the R package mgcv.
...
- 1,063
1
vote
0
answers
8
views
Can do smoothing after making time-series data stationary
Can do smoothing after making time-series data stationary? And is it useful to do smoothing before making data stationary?
0
votes
0
answers
13
views
Making a distribution rougher or smoother [duplicate]
When I have a given distribution, how can I transform it to make it appear rougher or smoother? I need to give the user the option of magnifying or reducing differences in the density plot, and I don'...
4
votes
2
answers
67
views
Determining spline basis dimension using Wood's statistical test
In Simon Wood's book Generalized Additive Models (2nd ed.) on page 243, he describes the following procedure for checking that the basis dimension is too small:
Fortunately informal checking that the ...
- 1,063
1
vote
1
answer
75
views
Change y-axis value in R GAM plot from smoothed to actual response variable
I know similar questions have been asked before, but I've looked through them all and I'm still confused, in part because the other examples seem to be more complicated, so thought I'd throw mine up. ...
- 25
0
votes
1
answer
30
views
GAM - setting K-values before or after testing the different models
I'm working with GAM and I'm testing different models with and without certain variables, and I need to set k-values for the different smoothers. Do I need to use the exact same k-values prior to ...
- 1
0
votes
1
answer
45
views
Why does Friedman's Supersmoother predict using $X$ instead of $y$?
I am working on friedman's supersmoother with python notebook: https://notebook.community/moreati/supersmoother/examples/Supersmoother.
In this example, data is generated randomly:
...
- 101
1
vote
0
answers
18
views
Intuition for bandwidth and degrees of freedom in kernel smoothers
For Kernel smoothers such as local polynomial regression smoothers (the Nadaraya-Watson smoother), we consider $y = m(x) + \epsilon$, for $m(x)$ some smooth function we are trying to estimate and $\...
- 387
0
votes
0
answers
171
views
plot.gam partial effects plot: is there another function to plot more than one variable?
The package description of plot.gam describes under Warning: "The function can not deal with smooths of more than 2 variables!" My gam model has a base model that includes habitat variables, ...
- 49
0
votes
1
answer
49
views
Splines free of Gibbs phenomenon for smoothing
I'm trying to do 3-dimensional smoothing of some observations we made. To simplify things, here's a visualisation of one slice of the data we have:
Note that the observations are gridded and contain ...
- 122
1
vote
0
answers
22
views
Confidence Interval for Centered Moving Average of Autocorrelated Timeseries Data (Smoothing)
I want to compute the confidence intervals of moving average (MA) which I apply to a time series with both high-frequency and low-frequency terms.
I'm interested in the slowly changing part only (...
- 21
2
votes
1
answer
103
views
Is the modeling strategy of GAM in MGCV equivalent to ridge regression when there are no smoothing terms?
According to GAM, it utilizes a penalized likelihood, which is maximized by penalized iteratively re-weighted least squares (P-IRLS), to obtain parameter estimations. The likelihood is defined as:
...
- 31
0
votes
0
answers
17
views
Is there a smoothing technique taking account of covariates?
Hypothetical scenario:
(Just to explain my needs. Maybe not meaningful in reality)
Suppose we have recorded $\{y_{ij}\}_{i=1}^{n_j}$, the annual consumption expenditure per capita for $n_j$ households ...
- 15
0
votes
1
answer
32
views
How are the penalized splines defined here?
Based on 'Semiparametric Regression with R' (https://link.springer.com/chapter/10.1007%2F978-1-4939-8853-2_1), a penalized spline
$$
f(x)=\beta_{0}+\beta_{1} x+\sum_{k=1}^{K} u_{k}\left(x-\kappa_{k}\...
0
votes
0
answers
22
views
Rolling average plus exponential weighted average - Crazy?
Hi I am trying to predict covid data with neural network models for a Uni project. The covid death data is not reported in Scotland so a 7 day rolling average is definitely needed.
No smoothing:
7 ...
0
votes
1
answer
69
views
How to show that the gradient of the smoothed surrogate loss function leads to perceptron update?
This is about the contents of section 1.2.1 and 1.2.1.1 of the book "Neural Networks and Deep Learning: A Textbook". The link to the sections is here. The question arises from the following ...
- 1
0
votes
0
answers
23
views
Kneser Ney Smoothing P continuation denominator clarification
When I first read the P continuation formula from Wikipedia
$$
p_{KN}(w_{i})={\frac {|\{w':0<c(w',w_{i})\}|}{|\{(w',w''):0<c(w',w'')\}|}}
$$
I got the impression on the denominator I only count ...
- 101
5
votes
1
answer
58
views
What is the name of this kind of smoothing?
Essentially, the code below produces something that is similar to a running mean, but instead of 0/1 weights on a window, it has gaussian weights centered at the x-position in consideration.
I'm ...
teunbrand
1
vote
1
answer
91
views
Online time-series smoothing algorithm for sparse data
I am working on building a real-time system for processing and aggregating somewhat sparse and irregular survey measurements (ranges from 0-100, usually on the order of 20-100 measurements). I am ...
- 113
0
votes
0
answers
20
views
How can I use GAM for factor variables?
I have factor variables like season (Summer and Winter), Time of Day (Morning, Afternoon, Late morning and Evening)and Landscape feature (Agriculture, Grassland etc._. I want to see the effect of ...
0
votes
1
answer
96
views
Number of knots in smoothing splines, residual plots and sample size?
I'm working with a dataset of 56 samples, so I am trying to keep the complexity of the regression model down. However, I have rather complex non linear relationships between some of the predictors and ...
- 3
0
votes
0
answers
85
views
Using Autoencoders to denoise timeseries
I've been trying to use Autoencoders to denoise time series using R. In my script, I create a function that receives the series to be denoised (as a vector) and the number of lagged observations ...
6
votes
2
answers
163
views
Estimating growth rate from noisy data?
Let's say we want to estimate growth rate from noisy data. Due to noise, simple calculation will result in very poor estimates (calculating growth rates just exacerbates noise), so smoothing is needed....
- 3,173
0
votes
1
answer
53
views
Smoothing sensor signals with response times - how to
I have a Sensor (e. g. for temperature) that has a response time ($t_{99}$ -> this is the time that the sensor needs to give an output of 99 % of the actual value) of let's say 10 seconds. This ...
- 3
2
votes
0
answers
45
views
(How) Can you combine moving average and exponential smoothing filters to get smoother trends?
Goal: Have a machine perform smoothing on time series data set to have a smooth looking trend. The correctness of the trend is balanced by being optimal i.e., minimize the MSE (Mean Standard Error).
...
- 13.6k
0
votes
0
answers
131
views
How to choose smoothing parameter in RBF interpolation
I'm using Scipy RBF interpolator to create a function that moves in between points of my multi-dimensional dataset (4 dimensions).
The interpolator has a smoothing parameter that I'd like to use to ...
0
votes
0
answers
19
views
Constrained interpolation/smoothing of multi-dimensional time series
Consider an $N$ dimensional time series $x_i(t),~i\in\{0,1,\cdots, N-1\}$ where $x_i(t)$ is smooth. It turns out that for all $t$: $x_i(t)>x_{i-1}(t)$.
The multi-dimensional series is sampled at ...
- 205
2
votes
0
answers
165
views
Creating a temporal soap film GAM with mgcv
I've been experimenting with using soap film smoothers in mgcv to estimate the densities of a species over space given aerial imagery of the species' extent and in ...
3
votes
1
answer
159
views
When to use smoothing of the predictor in a time dependent incident dynamic ROC analysis, and is there any arguments not to use smoothing?
In the article “Estimation of incident dynamic AUC in practice”, Geloven et al. 2020 (https://www.sciencedirect.com/science/article/pii/S0167947320301869#b10) they describe:
“In recent years, several ...
- 89
1
vote
0
answers
29
views
How to use a temperature raster (e.g., PRISM) to constrain temperature values in a thin plate spline regression and interpolation
I have point data with temperature, latitude, longitude, and elevation. I am interpolating across space to the extent of those points, and have been using elevation as a covariate in the ...
- 113
4
votes
1
answer
247
views
xgboost demand model with a smooth effect for the price variable
Question
The question is: how to smooth out kinks in individual demand curves in a GBDT model without underfitting on the price variable?
Background
We have some GBDTs demand models already in place (...
- 153
0
votes
2
answers
43
views
How can I fit a smoother to a 2-dimensional parametric curve (with R)? [closed]
I have a dataset of GPS traces of lat/lon and time for some routes (ex: NYC-Boston). Since I have multiple traces for each route, I would like to find the "average" trace, or some kind of ...
- 141
0
votes
0
answers
42
views
GAM: Difference between first and last observation
I'm using GAMs (bam() in mgcv, R 4.0.5) to investigate how two individuals talking together vary in their realizations of specific sounds during their conversation. I find GAMs useful because they ...
0
votes
1
answer
566
views
Confidence Interval for Centered Moving Average of Timeseries Data (Smoothing)
I'm having trouble finding a good resource on this.
I'm plotting some timeseries data over the last 200 years that has a clear trend, although there is also a lot of noise. I have smoothed the data ...
- 135
0
votes
1
answer
67
views
How to clean data to produce a smooth histogram
Is there any way to 'clean' a set of data to produce a smooth histogram, ie. without overrepresentative bins? Looking for solution in python3.
At the moment I have a histogram with overcounted bins:
...
- 1
0
votes
0
answers
243
views
Determine family in GAMs of negative values
I have these values (negative and positive) and I want to determine the nonlinear relationship between variable and predictor using generalized additive models (GAMs).
...
- 3
1
vote
0
answers
53
views
Estimating difference smooth effects for each level of a factor - WHY?
I am wondering why you would want to use a by variable smooth s(time, by= x)? (time is non-linear hence why I am using GAMs)
I am using GAMs to explore if my ...
- 11
1
vote
2
answers
153
views
How to show that smoothing spline fit preserves the local regression part of the fit
We need to show that a smoothing spline of $y_i$ to $x_i$ retains the local regression part of the fit.
For linear regression, this problem seems trivial because it is relatively easy to move from $...
- 27
0
votes
1
answer
667
views
What do the values for initialization method mean in statsmodels simple exponential smoothing?
I'm trying to use Statsmodels' simple exponential smoothing for time series analysis.
There are various methods available for initializing the recursions (estimated, heuristic, known).
Can someone ...
0
votes
0
answers
30
views
Terminology for smoothing methods: kernel density estimation, kernel smoothing, etc
I'm using what I have been referring to as 'kernel density estimation' to estimate the rates of a series of variables a, b, c from noisy observations distributed in ...
- 1
0
votes
0
answers
19
views
Is there a way to do such kind of smoothing for log likelihood?
Let's say we have a sequential decision making problem. At each step, we need to make a decision, and the decision made in this step will determine the possible actions for the next step.
Now I have a ...
- 153
0
votes
1
answer
31
views
Moving average (running mean) - how to keep all observations in smoothed time-series?
Let's suppose I have a time-series of 100 daily values and I want to compute a 5-day moving average of this time series.
I would do as follows:
...
- 284
2
votes
0
answers
78
views
Roughness penalty matrix for the Fourier basis functions
I am trying to understand how the roughness penalty matrix for the Fourier basis functions is calculated in R using the fda ...
- 257
0
votes
0
answers
65
views
How can I smooth the predictions of a supervised model by learning its trend?
This is the predictions of a binary classification model. The model is doing predicitons continuously, and these values are the sum of positive labels during a 10 hours period. As you can see, some of ...
- 101
1
vote
0
answers
59
views
Reference for spar parameter in R's implementation of the smoothing spline
In the R's implementation of the smoothing spline which is smooth.spline function, there is a parameter, called spar that ...
- 1,603
1
vote
0
answers
11
views
Smoothing of a surface [closed]
A land surveyor made a topographic map of a property in the hills by measuring the elevation at different points. The x-y coordinates are more or less "random", meaning no regular grid:
...
- 11
2
votes
3
answers
1k
views
What is the lag associated with Moving Average smoothing?
In a tutorial I came across this:
"Recall that the forecast value is: $\hat{y}_{t+1} = \frac{y_t + y_{t-1} + ... + y_{t-m+1}}{m}$
It's worth pondering that formula for a minute. While easy to ...
- 143
3
votes
2
answers
94
views
How is this seven day rolling average calculated?
Eric Topol posted the below on Twitter claiming that Europe is "turning COVID around" based on the trend in the past <7 days. However, while the case trend hitherto appears smooth, I was ...
- 53k