Questions tagged [smoothing]
Smoothing methods in data analysis, like splines or kernel smoothers, also regression smoothers like lowess.
0
votes
0answers
7 views
Smoothing with non-regular observations
If we have data that changes continuously in time, and we sample this data at regular intervals - i.e. we get samples $x_0$, $x_1$, $\dots$, where the time $\Delta t$ between taking samples $x_i$ and $...
0
votes
1answer
38 views
Retrieving time series from a smoothed periodogram
If I were to smooth a periodogram and then filter out low level frequencies, how can I derive the filtered time series? For example, in the case of a non-smoothed periodogram: https://folk.uib.no/...
0
votes
1answer
29 views
Natural spline term in GAM
Is it advisable to use natural regression spline basis? I learned that in R the supported smoothers in gam are the lo, ...
0
votes
0answers
11 views
Time series regression with temporal smoothing of regressors
I'm trying to forecast a discrete time series $Y_t$ with many observations of a very noisy covariate $X_t$. I could simply use a large number of lags:
$$ \mathrm{E}[Y_t] = \beta_0 + \sum_{i=1}^k \...
0
votes
0answers
20 views
Wavelet smoothing or regression with scattered data
Suppose we have a data set $\{x_i, y_i\}_i$ where $x_i$ is a multi-dimensional tuple and scattered (not on a equally spaced regular lattice). How does one regress or smooth such a scattered data set ...
0
votes
0answers
27 views
Reproducing Holt-Winters analysis in the Cowpertwait-Metcalfe book
I am having trouble reproducing some output for some R code in the time series book by Cowpertwait and Metcalfe. There are quite a few typos in their code throughout the book, but in other cases I ...
0
votes
0answers
16 views
Good turing smoothing for unigram LM
I was wondering if it is at all possible to use good turing smoothing for unigram language model?
I know that this smoothing technique helps distribute the weights from most occurring words to less ...
1
vote
0answers
22 views
Transforming a categorical distribution up side down
I have the following categorical distribution:
$$
original = (2, 7, 3, 5, 0, 1, 4, 6, 3, 8, 8)\\
\sum original = 47
$$
I want to transform it to its upside down distribution step by step, while ...
0
votes
0answers
14 views
Computing a moving average when data points arrive one at a time
I am sure there's a cool python/numpy/pandas way to do this.
I am receiving one data point at a time. I would like to compute something like a moving average over the last n observations, even better ...
1
vote
1answer
42 views
Random effect in GAM - what are the smooth functions used?
In the GAM package in R created by Simon Wood there is a selection of the smooth function basis. I sort of understand the options such as bs='tp', bs='cr', etc.
But bs='re' seems odd... that does ...
0
votes
0answers
21 views
What do the outputs of the function dlmSmooth from R's dlm package mean?
In the vignette for the R package dlm: link on page 12, the author runs the function dlmSmooth to smooth the data and the function returns an object which is ...
0
votes
0answers
33 views
Fixed-Delay Kalman smoother with/without augmented measurements
There are several algorithms regarding fixed-lag Kalman smoothing.
In most cases, an augmented state vector is defined in which the elements are the current and delays of the original state vector.
...
0
votes
0answers
18 views
Smoothing of experimental data for PCA
I am applying PCA to a set of spectrophotometric measurements with the aim of differentiating two groups of substances. The many small wiggles on the right-hand side of the curves (the region above ~...
0
votes
1answer
38 views
Is a Kalman filter ever the optimal way to estimate a dynamic value given a full history of measurements?
I'm trying to get some intuition for Kalman filtering, and I conceived this toy example:
Say that I have a sensor that tracks a moving 1-dimensional target. Say that the measurements from the sensor ...
2
votes
4answers
175 views
Measure of smoothness
I have an image that has artefacts which I am using a specific process to remove. I want to show that the new image is improved by that process.
To compare the two images I am using data from a ...
6
votes
3answers
179 views
How to fit a robust step function to a time series?
I have a somewhat noisy time series that hovers around different levels.
For example, the following data:
I have the solid line data available, and I would like to obtain an estimate for the dashed ...
1
vote
0answers
29 views
Is there a filter function that performs similar to moving average but does not loose data?
I need to smooth a time series using a low pass filter. A simple moving average is working fine for me, however, using a moving average causes an inevitable loss of data a the beginning of the ...
1
vote
0answers
27 views
Expected Value of Cross Validation approximates Predictive Square Error
In the context of Smoothing Splines, Im trying to show that the expected value of the cross-validation can approximate the predictive square error. More specifically, I want to show that
$$E[(y_i - \...
1
vote
0answers
48 views
Natural Splines and Smoother Matrix
In the context of smoothing splines, one can show that the Reinsch form is given by:
$
\hat{y} = N (N^{T}N +\lambda \Omega)^{-1}N^{T} y
= (I+ \lambda K)^{-1}y
$
where (1) $K = (N^{T})^{-1}\Omega N^{-...
2
votes
1answer
49 views
How to smooth a curve by learning location and shape of 4 Gaussian kernels?
I have a data set of 365 daily curves $f_i(t)$ and want to smooth them by positioning four radial basis functions $r_i(t)$.
Each daily curve should then be approximated by
$$ \hat{f}_i(t) = w_{i1} ...
0
votes
0answers
52 views
How to prove that katz backing off smoothing technique is a valid probability distribution?
How to prove that katz backing off smoothing technique is a valid probability distribution? Take the example of bigram.
You can go through this link for katz back off model
https://en.wikipedia.org/...
1
vote
0answers
36 views
RKHS norm and Fourier transform link
In the notes here, it is stated that norms of some reproducing kernel Hilbert spaces can be written in terms of Fourier transforms, and this is often used to argue that a higher RKHS norm implies a ...
1
vote
1answer
25 views
Partial spline. Reference
I have a well done and perfectly working protocol to smooth my experimental data. I do the following:
I have a variable of size 1000. Iteratively I choose random 100 points and spline them using the ...
3
votes
1answer
59 views
What does “middle of the data” really mean?
I have some noisy time series data for different climate variables and I want to know overall if they are increasing or decreasing with time. From this Water Resources Statistics Guide, the LOWESS ...
2
votes
1answer
44 views
Show that solution to cubic smoothing spline reduces to regular least squares minimization as $\lambda$ approaches infinity
I am asked to show that the solution to a smoothing splines problem of the form
$$
\text{PRSS}(f,\lambda) = \sum_{i=1}^N\left[y_i-f(x_i)\right]^2 + \lambda \int f''(t)^2 dt,
$$
with
$$
f(x) = \sum_{j=...
0
votes
0answers
61 views
$\chi^2$-test for smoothing splines: degrees of freedom
Suppose we are given raw data, e.g. raw mortality rates $\widetilde{q_x}$, which are graduated by a smoothing (cubic natural) spline $S$. That is, we obtain smoothed rates by setting $q_x:=S(x).$
Let ...
1
vote
1answer
67 views
Generalized additive models - formula for basis functions
I'm trying to understand the basics of GAMs.
Wood's book "Generalized Additive Models: an introduction with R" introduces GAMs via a cubic spline basis $b_j(x)$ (see p. 122), where
$b_1(x)=1, b_2(x)=...
0
votes
0answers
24 views
How can I do monotonic smoothing in R? [duplicate]
I would like to create a monotonic smoothing function to transform vectors such that I reduce their spikes (i.e., noise) while maintaining their trend. The problem I am encountering with standard ...
0
votes
0answers
67 views
Generalized Additive Models: How to fit models with the LMS method by Cole?
I have some problems to understand the univariate generalized models that were proposed by Cole (1992).
My question is how the fitting procedure works described in the appendix. More specific, how to ...
5
votes
1answer
170 views
Functional Data Analysis and Splines Regression
I'm new to Functional Data Analysis so my question will'be very simple for an expert in this topic. Operatively, when I fit a model like this, in R :
model<- lm( y ~ bs( x, df = k, degree = l ) )...
0
votes
1answer
94 views
Improve estimate of zeros and how to deal with concurvity in GAM NB
I'm trying to model the number of deaths for each age, for 4 regions, across 5 years. I fitted several models, but end up with the following model:
...
0
votes
0answers
35 views
Back-Off with Good-Turing Discounting
How do I handle backing off when using Good-Turing discounting?
Let's say I want to score $p(A|B)$. If $A$ has been seen with the word $B$ as its history, I can compute the probability as $p(A|B) = \...
1
vote
1answer
344 views
Exponential Smoothing & Seasonality
I am relatively new to stats and forecasting and was hoping to tap the wisdom of this community in regards to a question.
One of my colleagues insists that exponential smoothing presumes seasonality. ...
0
votes
0answers
57 views
mgcv optimization: why optimize **log** smooth parameter?
In mgcv package, when optimizing smoothing parameters (in the outer iteration), one take the partial derivative of objective function with respect to $\rho = log(\lambda)$. My question is why not take ...
0
votes
0answers
27 views
Can you fit a non-parametric regression such that the first derivative will be equal to zero for some specific points?
Given two variables x and y, where y=f(x)+error, is it possible to estimate a non-parametric regression of y on x taking into account the fact that we know that f(x) should take maximum or minimum for ...
0
votes
0answers
9 views
accuracy conditional on feature values
I have a binary classification model and I would like to estimate the accuracy as a function of another variable.
To be clearer, I can compute the usual accuracy on the testset:
$$
acc = \frac{\...
0
votes
0answers
51 views
Significance testing of correlated time series
I have two time series (~1200 data points each) and I want to understand how the relationship between them has evolved over time. I took the correlation (Pearson's) over a rolling window and see a ...
1
vote
1answer
375 views
Find good smoothing spline factor
I'd like an automatic way to find the "best" smoothing factor s for a spline fit to a given set of data points. Here's a sample visualization of some data and the ...
3
votes
1answer
54 views
Exchangeability and data smoothing
If a non-i.i.d sequence of a continuous random variable that is exchangeable, is smoothed by taking rolling average, is the resulting sequence exchangeable? My intuition suggests that it is not. I'll ...
1
vote
0answers
93 views
Estimating signal-to-noise ratio after smoothing data and covariance estimates
I have data for two variables, let's say $A$ and $B$, which have distributions that are roughly centered at 0. I am computing a value, let's say $C = \sqrt{A^2 + B^2}$, from this data. I also have a ...
1
vote
3answers
202 views
Positive smoothing with the fda-package (Functional data analysis)
In the book Functional Data Analysis with R (Ramsay&Silverman) there is described the possibility to do the "positive smoothing" if it’s needed instead of the "normal smoothing".
In the books ...
0
votes
0answers
28 views
Detect/correct 2d slices outliers from 3d volume
I have to segment 3D volumes (240x240x155 pixels). I am doing this by segmenting 155 2D slices of 240x240 pixels with a CNN. At the end, I reconstruct my 3D volume by simply concatenating the ...
0
votes
1answer
50 views
STL function in R: is it possible to make it backward looking?
I would like to smooth modeling input time series data using the stl function in R.
Based on my understanding, the function performs a local regression using data ...
0
votes
0answers
50 views
transformation for binary and categorical independent variables
Friends,
I have a large dataset in which only Y and one of the independent variables are continuous. there are 12 binary independent variables and 2 other categorical independent variables (each with ...
8
votes
1answer
257 views
Probabilistic interpretation of Thin Plate Smoothing Splines
TLDR: Do thin plate regression splines have a probabilistic/Bayesian interpretation?
Given input-output pairs $(x_i,y_i)$, $i=1,...,n$; I want to estimate a function $f(\cdot)$ as follows
\begin{...
5
votes
1answer
208 views
GAM : smoothing splines
Here are 2 questions. Your expertise on those issues would be highly appreciated.
In the GAM approach, it makes sense to start with a highly flexible approach and then apply penalties to achieve the ...
3
votes
1answer
208 views
Invertibility in Reinsch form Derivation (Smoothing Splines)
In Element of Statistical Learning II, in the context of smoothong splines, $\pmb{S_{\lambda}}$ is defined as
$$
\pmb{S_{\lambda}} = \pmb{N}(\pmb{N}^T\pmb{N} + \lambda \pmb{\Omega_N})^{-1}\pmb{N}^T
$$
...
1
vote
2answers
92 views
some examples of filtering the noise out of a data set
I have a data set which measures 60 data points in a second (60Hz). Clearly, I do not really need all 60 points in one second since this only generates some noise.
ABOUT MY DATA:
So my data sets ...
0
votes
1answer
160 views
GAM automatic smoothness selection [closed]
I'm trying to use gam in MGCV to automatically select my level of smoothness using cross validation or some equivalent method for my generalized additive model. reading around online it looks like ...
1
vote
0answers
38 views
What is the density of $X$ under fixed design?
We observe an i.i.d. sample $(X_1, Y_1), \ldots (X_n, Y_n).$ Let $m(x) = E(Y|X=x),$ $\sigma^2(x) = \operatorname{Var}(Y|X=x)$ and let $f(\cdot)$ be the density of $X.$
Under some regularity ...
