Skip to main content

Questions tagged [smoothing]

Smoothing methods in data analysis, like splines or kernel smoothers, also regression smoothers like lowess.

Filter by
Sorted by
Tagged with
0 votes
0 answers
18 views

Adjust winrates based on number of games played using Bayesian estimator

I am sorting players of an online video games based on their winrate, but I'm trying to use the number of games they scored that winrate on. The idea behind it is that even though 9 wins for 1 losse ...
user25485418's user avatar
0 votes
0 answers
31 views

Meaning/Definition of Smoothing

I am self-studying Vector Generalized Linear/Additive Models (mostly) via Yee, T. W. (2015) "Vector Generalized Linear and Additive Models: With an Implementation in R." This book has many ...
TJM's user avatar
  • 153
0 votes
0 answers
12 views

Forward-Backward Algorithm for Autoregressive HMMs

I am currently studying HMMs, and covered the Forward-Backward Algorithms as well as the smoothing and filtering process. Recently, we were posed a question on Autoregressive HMMs which I've been ...
Kai's user avatar
  • 83
4 votes
1 answer
83 views

How to detect a plateau at the end of a short time series?

In exercise testing (spiroergometry), participants' oxygen uptake is measured continuously while they engange in physical exercise that progressively intensifies. The measurements are taken at regular ...
COOLSerdash's user avatar
2 votes
0 answers
21 views

Smoothing spatial data for privacy issues

I have a dataset with about 6000 observations related to municipalities. The structure of the dataset includes four columns: the municipality code, the number of businesses in the municipality, the ...
Luigi's user avatar
  • 327
0 votes
0 answers
46 views

How to create a GAM with interation term for a soap film smoother

I am interested in creating a GAM that contains a soap film smoother that can also vary in time. Below is an example using synthetic data. I start by creating data with two spatial signals, which are ...
Marc in the box's user avatar
3 votes
1 answer
119 views

GAM(M): "Shape" vs "wiggliness"?

I'm currently going over how to interpret certain terms of a generalized additive mixed model (GAMM). I'm having a hard time differentiating between "shape" and "curvature". ...
Nate's user avatar
  • 1,641
0 votes
0 answers
48 views

Aggregation of smoothing terms in GAMs

I have fitted a GAM model by using the R package mgcv via the expression model <- gam(formula, data, family = binomial()) ...
Jo R's user avatar
  • 331
1 vote
1 answer
62 views

Smoothing out target variable for spiky demand forecasting

I am trying to predict ambulance demand for the next hour, for a city area in the USA, based on previous demand, weather, large people gatherings, and similar spatio-temporal factors - using Machine ...
Nadir Bašić's user avatar
3 votes
1 answer
257 views

Significance testing on Generalized Additive Mixed Models (GAMMs) - mgcv::gam

I am wondering whether anyone has any insight on comparing the performance of two GAMMs? Specifically, I want to compare two models: A "nested model", one with a smoothed age term ...
PhelsumaFL's user avatar
2 votes
0 answers
47 views

When smoothing using a cubic regression factor smooth in R using mgcv’s gam model, how do I set the knots and k for each level of the factor smooth?

I’ve fit a Poisson generalized additive model (GAM) to predict the number of deaths given exposure using the R library mgcv and its ...
Colin's user avatar
  • 41
0 votes
0 answers
81 views

Smoothing a transition matrix

I have a dataset containing observations at time t and t+1 of ratings A (best) to E (worst) and Default. I want to use transition matrices to predict future ratings t+2, t+3, etc The transition matrix ...
Lucy Edwards's user avatar
6 votes
1 answer
172 views

mgcv gam P-value vs plotted confidence intervals

I am trying to figure out how to explain the apparent discrepancy between P-values and plotted confidence intervals in mgcv. See for example the plot below that comes from a model that considers the ...
dean's user avatar
  • 455
0 votes
0 answers
25 views

How to detect moments of state change in boolean time series?

Let $A = (X_1, ... X_n)$ be a series consisting of long periods of ones interrupted with shorter periods of zeros, like $$A = 111111111 00000001111111111100000011111111111111.$$ The plot below (left) ...
Brzoskwinia's user avatar
0 votes
0 answers
10 views

How does Kneser Ney estimate ngrams with BOS without dividing by zero?

The recursive formula for (unmodified) Kneser Ney smoothing is (per Jurafsky08 3.40) $P_{KN}(w_i | h) = \frac{\text{max}(c_{KN}(h\ w_i) - d, 0)}{\sum_v c_{KN}(h \ v )} + \lambda(h) P_{KN}(w_i | h)$ ...
user2740's user avatar
  • 1,386
0 votes
0 answers
16 views

Assessing probability that one set of measurements extends the other with Kalman smoother

I have two sets of N-dimensional measurements following each other with a certain time gap in between. Let's name those sets $A$ and $B$, respectively. All observations have constant Gaussian white ...
emerte's user avatar
  • 1
0 votes
0 answers
23 views

Why would you smooth a logical variable in a GAM?

I just used the gam.scope function in the R-package gam to create possible scopes of each explanatory variable of my model to ...
Lena Ortega's user avatar
4 votes
1 answer
142 views

Is it possible to use smooth functions as part of a nonlinear regression?

Background I am fitting nonlinear regressions with a single response and two predictors. I know the relationship between the response and each of the predictors, but I do not know how they interact. I ...
mkt's user avatar
  • 18.9k
0 votes
2 answers
64 views

Inferences with Filtered and Smoothed state estimates from a tracking problem

I am working on a synthetic tracking problem in a two-dimensional space, where the start and end positions are known and noisy measurements of the state variables are given at discrete time steps. As ...
Carlo Berger's user avatar
3 votes
2 answers
427 views

How does lowess handle gaps in time series?

I have been using the lowess smoother to calculate trends for time series data for a while now but until now my data was always without gaps. I now have to work with data where there are quite large ...
Krautsultan's user avatar
1 vote
1 answer
181 views

Understanding Identifiability problem of multiple smooths in GAMs/Additive models

I am a beginner into additive models and GAMs, but have good enough knowledge of linear regression. I was going through this wikipedia article to understand more, but can't seem to understand the ...
user101874's user avatar
1 vote
0 answers
188 views

Smoothing of GPS tracks - remove noise and stop-go clusters

I know there are several posts about this, but I could not find exactly what I need. I have GPS track data (from an underwater vehicle) for short intervals of 1 second (time-stamps on data). The data ...
Mona's user avatar
  • 11
1 vote
0 answers
99 views

Functional Data Analysis with Missing Data

I am trying to analyze a dataset containing several observations for n = 27. Unfortunately, not every participant answered the question every time, i.e., when plotting the graph it has holes in it. I ...
Soph's user avatar
  • 11
2 votes
0 answers
174 views

GAM partial response plot interpretation

I've made the gam in the code below (in R), but I'm struggling to interpret the results. Specifically, the partial response plots for all but one of the variables is linear, and the CI lines cross in ...
blitz1259's user avatar
0 votes
0 answers
63 views

Smoothing without losing data points

How would I smooth a series of data without losing the ends of a series? For example, if I have $i_1, i_2, \dots, i_n$, and I apply a moving average, I lose the start and end of the series ($i_1$ and $...
user389220's user avatar
1 vote
1 answer
329 views

Kernel Smoothing for Time Series data [closed]

I have generated a time series data set of measurements that are a bit noisy and I want to apply kernel smoothing to the data. My time series data is not regular however, meaning that the time ...
Jade131621's user avatar
4 votes
1 answer
592 views

How 'by' factor works with 'fs' random smooth in gam?

I've a large dataset including a response bmk, a continuous predictor delay, a group factor (...
denis's user avatar
  • 185
1 vote
0 answers
38 views

Is there a statistically sound method of smoothing a data series without removing the edges?

Recently I've been plotting a lot of data, and often I find myself using a moving average to smooth out values that oscillate or otherwise fluctuate a lot. However, the problem with this is that it ...
Outis Nemo's user avatar
0 votes
1 answer
79 views

Assessing differences in slope of a loess (or any smoother) fit

I have continuous data for x (for example "time") and y with several groups. I would like to show/test that the slope of groupC changes over time (is positive and rises) while groups A and B ...
ATpoint's user avatar
  • 29
3 votes
1 answer
351 views

Gam plots y-axis range is different for model that that from predict function

I got some idea about gam (generalized additive model) plot from a course of Noam Ross. However, I have several confusion and got stucked. I fit a logistic gam and can interpret by using ...
Ly T's user avatar
  • 41
3 votes
1 answer
68 views

The difference between smoothing problems in stochastic processes and statistics

I am taking a class and we are about to talk about Kalman filters and smoothing and I am trying to do some reading ahead. On the Wikipedia page for 'Smoothing (stochastic processes)' it says 'the ...
yungtoad's user avatar
  • 131
7 votes
1 answer
358 views

Spline basis explicitly including a linear term; basis functions generated by the default call to "s()" function of mgcv package

I'm curious about the basis functions generated by the call to "s()" function with default parameter values, but even more specifically I'm curious about a smoother for a single variable ...
UsDAnDreS's user avatar
  • 159
2 votes
1 answer
138 views

Why smoothCon gives different estimates of smooths than mgcv?

I am learning gam by myself. I tried to find posts that can help me to finding and understanding smooth functions i.e., basis functions from gam using mgcv. From a ...
user avatar
1 vote
1 answer
161 views

GAM: covariate specific fits do not appear to match approximate significance of smooth terms

I have fit a GAM with two continuous independent variables and one discrete covariate with two levels ("gray" and "brown"). All variables are scaled and the model runs fine: ...
mluerig's user avatar
  • 731
2 votes
1 answer
64 views

How is bandwidth and roughness penalty in nonparametric regression connected?

Nonparametric regression, or smoothing, concerns finding a function that smooths the data, and appears in additive models or generalized additive models. You could say we are trying to fit the model $$...
cgmil's user avatar
  • 1,373
3 votes
0 answers
152 views

When fitting a generalised additive model, how to choose how much to smooth?

When fitting a GAM, is there a rule (of thumb) for deciding if $k$ (max number of degrees of freedom for a spline) is large enough or not? How much should edf be below $k'$? And is that an absolute ...
user avatar
2 votes
0 answers
34 views

How do professional statisticians evaluate or mathematically justify choice of one smoothing method from another? [closed]

I would like to fit some economic data and evaluate it. Usually I used only regression techniques, but recently I found out about smoothing techniques which could be used to diminish noise in dataset ...
SCP_org's user avatar
  • 21
2 votes
2 answers
81 views

Curve quantification

I have some longitudinal measurement data of 15,000. I smoothed that data with B-spline smoothing and got the following curve. I then want to quantify this curve and extract features for clustering ...
NakataKoo's user avatar
4 votes
2 answers
346 views

Rank of smoothing matrix

The solution to the minimizing problem: $$RSS(f, \lambda) = \sum_{i=1}^n (y_i - f(x_i))^2 + \lambda \int (f''(t))^2 dt$$ is written in matrix form as $f = N\theta = \sum_{j = 1}^n N_j(x_i) \theta_j$ ...
Lejoon's user avatar
  • 175
0 votes
0 answers
550 views

Timeseries - How are the window/span size and 'frac' in LOWESS related?

Many software implementations of locally weighted scatterplot smoothing (LOWESS), such as those in Python and R, require a frac parameter as an input. I am confused as to how frac is mathematically ...
Kartik's user avatar
  • 105
3 votes
1 answer
1k views

Is it possible to regularize a covariance matrix?

I have many "parallel" time-series (about 100). They have relatively short history. I calculate a covariance matrix between these time-series. Now, I believe that the observed covariance ...
Roman's user avatar
  • 612
0 votes
0 answers
56 views

Hat matrix for penalized spline

Suppose the penalized spline given by equation $$ \hat{x}=(W+\lambda_{1}D^{T}D+2\lambda_{2})^{-1}\left[Wy+\lambda_{2}(A+B)\right], $$ where $\lambda_{1},\lambda_{2}$ are positive constants and $A,B$ ...
justik's user avatar
  • 101
6 votes
2 answers
1k views

Smoothing spline seems to fit too precisely?

I am trying to fit some time series data to a smoothing spline in R. However, it seems like the spline is fitting the data too perfectly, meaning overfitting. I was trying to figure out what settings ...
krishnab's user avatar
  • 1,522
3 votes
1 answer
2k views

How to decide which variables to smooth in GAM

When specifying it's formula GAM has s function for smoothing. Let's say I want to fit mtcars. I can do the following: Approach 1: ...
littleworth's user avatar
2 votes
0 answers
193 views

How to choose the smoothing method and factors that best capture the trend in data?

I have a time-series data that I will be using to forecast the future using machine learning algorithms such as LSTM. Prior to the forecast, I need to smooth the data to remove the noise and capture ...
Traveling Salesman's user avatar
2 votes
1 answer
114 views

What reasons beyond interpretability are there to use additive models over a complex, multivariate smoother?

Let's adopt for the second the notation from the R package mgcv. ...
cgmil's user avatar
  • 1,373
1 vote
0 answers
71 views

Can do smoothing after making time-series data stationary

Can do smoothing after making time-series data stationary? And is it useful to do smoothing before making data stationary?
Ruslan Pylypyuk's user avatar
4 votes
2 answers
119 views

Determining spline basis dimension using Wood's statistical test

In Simon Wood's book Generalized Additive Models (2nd ed.) on page 243, he describes the following procedure for checking that the basis dimension is too small: Fortunately informal checking that the ...
cgmil's user avatar
  • 1,373
1 vote
1 answer
955 views

Change y-axis value in R GAM plot from smoothed to actual response variable

I know similar questions have been asked before, but I've looked through them all and I'm still confused, in part because the other examples seem to be more complicated, so thought I'd throw mine up. ...
GRSB.1's user avatar
  • 25
1 vote
1 answer
204 views

GAM - setting K-values before or after testing the different models

I'm working with GAM and I'm testing different models with and without certain variables, and I need to set k-values for the different smoothers. Do I need to use the exact same k-values prior to ...
Lotte's user avatar
  • 11

1
2 3 4 5
8