Questions tagged [smoothing]

Smoothing methods in data analysis, like splines or kernel smoothers, also regression smoothers like lowess.

Filter by
Sorted by
Tagged with
0 votes
0 answers
9 views

Smoothing temperature data using python

Hey there I have a datalogger which logs my temperature (internally and externally) as well as the humidity using a DHT22 sensor on a pizero wh. My current temperature graph (no smoothing) looks like ...
user avatar
  • 101
0 votes
0 answers
20 views

The correct pre-processing of time-series data when using LSTM

I have a time-series data that I would like to use for forecasting the data trend using LSTM. I followed https://towardsdatascience.com/the-complete-guide-to-time-series-analysis-and-forecasting-...
user avatar
2 votes
1 answer
33 views

What reasons beyond interpretability are there to use additive models over a complex, multivariate smoother?

Let's adopt for the second the notation from the R package mgcv. ...
user avatar
  • 1,063
1 vote
0 answers
8 views

Can do smoothing after making time-series data stationary

Can do smoothing after making time-series data stationary? And is it useful to do smoothing before making data stationary?
user avatar
0 votes
0 answers
13 views

Making a distribution rougher or smoother [duplicate]

When I have a given distribution, how can I transform it to make it appear rougher or smoother? I need to give the user the option of magnifying or reducing differences in the density plot, and I don'...
user avatar
4 votes
2 answers
67 views

Determining spline basis dimension using Wood's statistical test

In Simon Wood's book Generalized Additive Models (2nd ed.) on page 243, he describes the following procedure for checking that the basis dimension is too small: Fortunately informal checking that the ...
user avatar
  • 1,063
1 vote
1 answer
75 views

Change y-axis value in R GAM plot from smoothed to actual response variable

I know similar questions have been asked before, but I've looked through them all and I'm still confused, in part because the other examples seem to be more complicated, so thought I'd throw mine up. ...
user avatar
  • 25
0 votes
1 answer
30 views

GAM - setting K-values before or after testing the different models

I'm working with GAM and I'm testing different models with and without certain variables, and I need to set k-values for the different smoothers. Do I need to use the exact same k-values prior to ...
user avatar
  • 1
0 votes
1 answer
45 views

Why does Friedman's Supersmoother predict using $X$ instead of $y$?

I am working on friedman's supersmoother with python notebook: https://notebook.community/moreati/supersmoother/examples/Supersmoother. In this example, data is generated randomly: ...
user avatar
1 vote
0 answers
18 views

Intuition for bandwidth and degrees of freedom in kernel smoothers

For Kernel smoothers such as local polynomial regression smoothers (the Nadaraya-Watson smoother), we consider $y = m(x) + \epsilon$, for $m(x)$ some smooth function we are trying to estimate and $\...
user avatar
0 votes
0 answers
171 views

plot.gam partial effects plot: is there another function to plot more than one variable?

The package description of plot.gam describes under Warning: "The function can not deal with smooths of more than 2 variables!" My gam model has a base model that includes habitat variables, ...
user avatar
0 votes
1 answer
49 views

Splines free of Gibbs phenomenon for smoothing

I'm trying to do 3-dimensional smoothing of some observations we made. To simplify things, here's a visualisation of one slice of the data we have: Note that the observations are gridded and contain ...
user avatar
  • 122
1 vote
0 answers
22 views

Confidence Interval for Centered Moving Average of Autocorrelated Timeseries Data (Smoothing)

I want to compute the confidence intervals of moving average (MA) which I apply to a time series with both high-frequency and low-frequency terms. I'm interested in the slowly changing part only (...
user avatar
  • 21
2 votes
1 answer
103 views

Is the modeling strategy of GAM in MGCV equivalent to ridge regression when there are no smoothing terms?

According to GAM, it utilizes a penalized likelihood, which is maximized by penalized iteratively re-weighted least squares (P-IRLS), to obtain parameter estimations. The likelihood is defined as: ...
user avatar
0 votes
0 answers
17 views

Is there a smoothing technique taking account of covariates?

Hypothetical scenario: (Just to explain my needs. Maybe not meaningful in reality) Suppose we have recorded $\{y_{ij}\}_{i=1}^{n_j}$, the annual consumption expenditure per capita for $n_j$ households ...
user avatar
0 votes
1 answer
32 views

How are the penalized splines defined here?

Based on 'Semiparametric Regression with R' (https://link.springer.com/chapter/10.1007%2F978-1-4939-8853-2_1), a penalized spline $$ f(x)=\beta_{0}+\beta_{1} x+\sum_{k=1}^{K} u_{k}\left(x-\kappa_{k}\...
user avatar
0 votes
0 answers
22 views

Rolling average plus exponential weighted average - Crazy?

Hi I am trying to predict covid data with neural network models for a Uni project. The covid death data is not reported in Scotland so a 7 day rolling average is definitely needed. No smoothing: 7 ...
user avatar
0 votes
1 answer
69 views

How to show that the gradient of the smoothed surrogate loss function leads to perceptron update?

This is about the contents of section 1.2.1 and 1.2.1.1 of the book "Neural Networks and Deep Learning: A Textbook". The link to the sections is here. The question arises from the following ...
user avatar
  • 1
0 votes
0 answers
23 views

Kneser Ney Smoothing P continuation denominator clarification

When I first read the P continuation formula from Wikipedia $$ p_{KN}(w_{i})={\frac {|\{w':0<c(w',w_{i})\}|}{|\{(w',w''):0<c(w',w'')\}|}} $$ I got the impression on the denominator I only count ...
user avatar
  • 101
5 votes
1 answer
58 views

What is the name of this kind of smoothing?

Essentially, the code below produces something that is similar to a running mean, but instead of 0/1 weights on a window, it has gaussian weights centered at the x-position in consideration. I'm ...
user avatar
1 vote
1 answer
91 views

Online time-series smoothing algorithm for sparse data

I am working on building a real-time system for processing and aggregating somewhat sparse and irregular survey measurements (ranges from 0-100, usually on the order of 20-100 measurements). I am ...
user avatar
0 votes
0 answers
20 views

How can I use GAM for factor variables?

I have factor variables like season (Summer and Winter), Time of Day (Morning, Afternoon, Late morning and Evening)and Landscape feature (Agriculture, Grassland etc._. I want to see the effect of ...
user avatar
0 votes
1 answer
96 views

Number of knots in smoothing splines, residual plots and sample size?

I'm working with a dataset of 56 samples, so I am trying to keep the complexity of the regression model down. However, I have rather complex non linear relationships between some of the predictors and ...
user avatar
  • 3
0 votes
0 answers
85 views

Using Autoencoders to denoise timeseries

I've been trying to use Autoencoders to denoise time series using R. In my script, I create a function that receives the series to be denoised (as a vector) and the number of lagged observations ...
user avatar
6 votes
2 answers
163 views

Estimating growth rate from noisy data?

Let's say we want to estimate growth rate from noisy data. Due to noise, simple calculation will result in very poor estimates (calculating growth rates just exacerbates noise), so smoothing is needed....
user avatar
0 votes
1 answer
53 views

Smoothing sensor signals with response times - how to

I have a Sensor (e. g. for temperature) that has a response time ($t_{99}$ -> this is the time that the sensor needs to give an output of 99 % of the actual value) of let's say 10 seconds. This ...
user avatar
2 votes
0 answers
45 views

(How) Can you combine moving average and exponential smoothing filters to get smoother trends?

Goal: Have a machine perform smoothing on time series data set to have a smooth looking trend. The correctness of the trend is balanced by being optimal i.e., minimize the MSE (Mean Standard Error). ...
user avatar
  • 13.6k
0 votes
0 answers
131 views

How to choose smoothing parameter in RBF interpolation

I'm using Scipy RBF interpolator to create a function that moves in between points of my multi-dimensional dataset (4 dimensions). The interpolator has a smoothing parameter that I'd like to use to ...
user avatar
0 votes
0 answers
19 views

Constrained interpolation/smoothing of multi-dimensional time series

Consider an $N$ dimensional time series $x_i(t),~i\in\{0,1,\cdots, N-1\}$ where $x_i(t)$ is smooth. It turns out that for all $t$: $x_i(t)>x_{i-1}(t)$. The multi-dimensional series is sampled at ...
user avatar
  • 205
2 votes
0 answers
165 views

Creating a temporal soap film GAM with mgcv

I've been experimenting with using soap film smoothers in mgcv to estimate the densities of a species over space given aerial imagery of the species' extent and in ...
user avatar
3 votes
1 answer
159 views

When to use smoothing of the predictor in a time dependent incident dynamic ROC analysis, and is there any arguments not to use smoothing?

In the article “Estimation of incident dynamic AUC in practice”, Geloven et al. 2020 (https://www.sciencedirect.com/science/article/pii/S0167947320301869#b10) they describe: “In recent years, several ...
user avatar
1 vote
0 answers
29 views

How to use a temperature raster (e.g., PRISM) to constrain temperature values in a thin plate spline regression and interpolation

I have point data with temperature, latitude, longitude, and elevation. I am interpolating across space to the extent of those points, and have been using elevation as a covariate in the ...
user avatar
  • 113
4 votes
1 answer
247 views

xgboost demand model with a smooth effect for the price variable

Question The question is: how to smooth out kinks in individual demand curves in a GBDT model without underfitting on the price variable? Background We have some GBDTs demand models already in place (...
user avatar
  • 153
0 votes
2 answers
43 views

How can I fit a smoother to a 2-dimensional parametric curve (with R)? [closed]

I have a dataset of GPS traces of lat/lon and time for some routes (ex: NYC-Boston). Since I have multiple traces for each route, I would like to find the "average" trace, or some kind of ...
user avatar
  • 141
0 votes
0 answers
42 views

GAM: Difference between first and last observation

I'm using GAMs (bam() in mgcv, R 4.0.5) to investigate how two individuals talking together vary in their realizations of specific sounds during their conversation. I find GAMs useful because they ...
user avatar
0 votes
1 answer
566 views

Confidence Interval for Centered Moving Average of Timeseries Data (Smoothing)

I'm having trouble finding a good resource on this. I'm plotting some timeseries data over the last 200 years that has a clear trend, although there is also a lot of noise. I have smoothed the data ...
user avatar
  • 135
0 votes
1 answer
67 views

How to clean data to produce a smooth histogram

Is there any way to 'clean' a set of data to produce a smooth histogram, ie. without overrepresentative bins? Looking for solution in python3. At the moment I have a histogram with overcounted bins: ...
user avatar
0 votes
0 answers
243 views

Determine family in GAMs of negative values

I have these values (negative and positive) and I want to determine the nonlinear relationship between variable and predictor using generalized additive models (GAMs). ...
user avatar
  • 3
1 vote
0 answers
53 views

Estimating difference smooth effects for each level of a factor - WHY?

I am wondering why you would want to use a by variable smooth s(time, by= x)? (time is non-linear hence why I am using GAMs) I am using GAMs to explore if my ...
user avatar
  • 11
1 vote
2 answers
153 views

How to show that smoothing spline fit preserves the local regression part of the fit

We need to show that a smoothing spline of $y_i$ to $x_i$ retains the local regression part of the fit. For linear regression, this problem seems trivial because it is relatively easy to move from $...
user avatar
  • 27
0 votes
1 answer
667 views

What do the values for initialization method mean in statsmodels simple exponential smoothing?

I'm trying to use Statsmodels' simple exponential smoothing for time series analysis. There are various methods available for initializing the recursions (estimated, heuristic, known). Can someone ...
user avatar
0 votes
0 answers
30 views

Terminology for smoothing methods: kernel density estimation, kernel smoothing, etc

I'm using what I have been referring to as 'kernel density estimation' to estimate the rates of a series of variables a, b, c from noisy observations distributed in ...
user avatar
0 votes
0 answers
19 views

Is there a way to do such kind of smoothing for log likelihood?

Let's say we have a sequential decision making problem. At each step, we need to make a decision, and the decision made in this step will determine the possible actions for the next step. Now I have a ...
user avatar
  • 153
0 votes
1 answer
31 views

Moving average (running mean) - how to keep all observations in smoothed time-series?

Let's suppose I have a time-series of 100 daily values and I want to compute a 5-day moving average of this time series. I would do as follows: ...
user avatar
  • 284
2 votes
0 answers
78 views

Roughness penalty matrix for the Fourier basis functions

I am trying to understand how the roughness penalty matrix for the Fourier basis functions is calculated in R using the fda ...
user avatar
  • 257
0 votes
0 answers
65 views

How can I smooth the predictions of a supervised model by learning its trend?

This is the predictions of a binary classification model. The model is doing predicitons continuously, and these values are the sum of positive labels during a 10 hours period. As you can see, some of ...
user avatar
  • 101
1 vote
0 answers
59 views

Reference for spar parameter in R's implementation of the smoothing spline

In the R's implementation of the smoothing spline which is smooth.spline function, there is a parameter, called spar that ...
user avatar
  • 1,603
1 vote
0 answers
11 views

Smoothing of a surface [closed]

A land surveyor made a topographic map of a property in the hills by measuring the elevation at different points. The x-y coordinates are more or less "random", meaning no regular grid: ...
user avatar
  • 11
2 votes
3 answers
1k views

What is the lag associated with Moving Average smoothing?

In a tutorial I came across this: "Recall that the forecast value is: $\hat{y}_{t+1} = \frac{y_t + y_{t-1} + ... + y_{t-m+1}}{m}$ It's worth pondering that formula for a minute. While easy to ...
user avatar
3 votes
2 answers
94 views

How is this seven day rolling average calculated?

Eric Topol posted the below on Twitter claiming that Europe is "turning COVID around" based on the trend in the past <7 days. However, while the case trend hitherto appears smooth, I was ...
user avatar
  • 53k

1
2 3 4 5
8