Questions tagged [smoothing]

Smoothing methods in data analysis, like splines or kernel smoothers, also regression smoothers like lowess.

Filter by
Sorted by
Tagged with
0
votes
1answer
121 views

Smoothing a binned averages

I am trying to smooth some binned data. I have a discrete variable X which might best be thought of as time and a continuous variable Y. I want to know the average Y value for each value of X and this ...
1
vote
1answer
35 views

How to measure smoothness of inputs over outputs?

I know similar questions have been asked for time series data. But my question is a little bit different. Consider that we have input dataset $X \in R^{N \times M}$, where $M$ is the dimension of ...
1
vote
0answers
45 views

Smoothing 3D motion tracking data [closed]

I'm working on an application that tracks an object's position in 3D space through time. The hardware generates (x,y,z,t) data points, with a non-uniform sampling ...
2
votes
2answers
255 views

Smoothing histograms with kernel methods

I have a problem where I can receive as output, multidimensional counts in "histogram" form. I can also adjust the size of the bins I receive (i.e., many or few bins). I want to smooth the data and ...
2
votes
0answers
32 views

Using Hierarchical Bayesian Network for Data Smoothing

I have a dataset that contains the mean speed at which 50 car models move, along with the standard deviation (based on an underlying database of sampled car speeds). Using this mean and deviation, I ...
2
votes
1answer
171 views

How should I understand smoothing in functional data analysis from a modelling perspective (specifically for temperature data)?

The specifics of this question are that I am looking at daily maximum and minimum temperature from the GHCND data set obtained from NOAA's API and I view the temperatures observed at days in a year as ...
6
votes
2answers
277 views

Forcing smoothness of regression coefficients

I'm building regression models on spectral datasets: the predictors are the intensites of signal at the different frequencies. In this case the intensities at close frequency values are highly ...
1
vote
2answers
87 views

Smoothing time series with non-constant variance

I have a discrete time series $x(t)$ $ t = \{0,\Delta t,2\Delta t\dots\}$ in which every point comes with a confidence value $c(t)$ arising from the measurements. You may think of is as the variance ...
1
vote
2answers
89 views

What are the exact factors used in smooth.spline by R?

What exactly is optimized mathematically when I use: smooth.spline(x, y, lambda) in terms of the integrated second derivative? Is it $$\min_{f\in C^2} \sum_{i=1}...
0
votes
0answers
225 views

Demmler-Reinsch basis for smoothing splines

I have seen some papers about using the so-called Demmler-Reinsch basis for smoothing spline because it is a basis for natural spline space and also Sobolev space. For example, these papers: A ...
2
votes
0answers
407 views

Smoothing in a monotonically increasing manner

I have a number of curves that contain numbers from between 0 and 1. The curves should be monotonically increasing, but due to random noise, there may be some times where it is decreasing. Is there ...
0
votes
0answers
81 views

predicting X values from smooth.spline

I have an existing smooth.spline object, and I wish to estimate X values for a set of new Y values. I see that ...
0
votes
1answer
95 views

Questions re. fitting a polynomial: Runge's phenomenon solutions

I have data on hospital treatment times. I would like to fit a polynomial to the data using least-squares. In a previous question raised before I have already been advised against this, but ...
2
votes
2answers
117 views

Questions re. fitting a polynomial: smoothing, cross-validation, etc

I have data on hospital treatment times. I would like to fit a polynomial to the data. My data comes in 5 minute increments and it is very noisy. It looks like this: I can aggregate to a higher ...
1
vote
0answers
311 views

Peak detection and smoothing data

I'm working on smoothing data. But I want to retain the peak actual value and draw the actual value along with other smoothed data. And I use the Robust peak detection algorithm but it didn't work the ...
2
votes
1answer
62 views

How can I smooth data in 2D coordinates that has time-dependent error?

I have collected some GPS data from running over and around a hill many, many times. The hill itself is about 9-10 meters high compared to the ground around it, although when I collected data, my ...
1
vote
2answers
853 views

What is the difference between smoothing and decomposition in time series?

I am bit new to time series modelling, currently I am trying to understand some basics. What is the difference between smoothing and decomposition in time series. I have gone through many materials, I ...
0
votes
0answers
34 views

Matrix inversion in the smoothing splines [duplicate]

The question is about the matrix inversion in the smoothing splines. Given observations (y1, x1), ..., (yn, xn) and a choice of $\lambda \ge 0$, the smoothing spline estimator, $\hat{f}_{\lambda}$ is ...
0
votes
1answer
208 views

Should I do detrending or smoothing first?

Does it matter which one I perform first? If yes, why? Might be a simple question, yet I could not find an answer anywhere else.
3
votes
1answer
324 views

How does the penalized form of RSS (residual sum of squares) work?

In another word, how to reverse engineering the equation (5.9) by explain all the assumption and reasoning after the plus sign of (5.9) in Elements of Statistical Learning: $$\text{RSS}(f,\lambda) = \...
1
vote
0answers
110 views

What does alpha in smoothing stand for

I want to apply Gaussian smoothing to a dataset and came across the smth.gaussian function in R. That besides the numerical input data requires two parameters: ...
1
vote
1answer
92 views

How to generate mean curve of non-function?

I am currently working on curves generated in tensile tests of polymer specimens. Here, I try to generate a mean curve of five data sets generated at the same composition of the samples. Unfortunately,...
5
votes
2answers
366 views

Is there a name for a moving average when it is done not across time but some other variable?

The moving average is defined as A method of smoothing a time series to reduce the effects of random variation and reveal any underlying trend or seasonality. (Oxford Dictionary of Statistics, ed. ...
1
vote
1answer
67 views

Why 2nd derivative is “squared” to represent wigglyness in GAM?

In David Miller's presentation (here, slide 21), he drew 1st derivative and 2nd derivative of a function. Then he said (slide 22) that grey part can is : $ \int (\frac{\partial ^{2}f(x)}{\partial x^{...
1
vote
0answers
76 views

Smoothing with non-regular observations

If we have data that changes continuously in time, and we sample this data at regular intervals - i.e. we get samples $x_0$, $x_1$, $\dots$, where the time $\Delta t$ between taking samples $x_i$ and $...
0
votes
1answer
209 views

Retrieving time series from a smoothed periodogram

If I were to smooth a periodogram and then filter out low level frequencies, how can I derive the filtered time series? For example, in the case of a non-smoothed periodogram: https://folk.uib.no/...
1
vote
1answer
725 views

Natural spline term in GAM

Is it advisable to use natural regression spline basis? I learned that in R the supported smoothers in gam are the lo, ...
0
votes
0answers
38 views

Wavelet smoothing or regression with scattered data

Suppose we have a data set $\{x_i, y_i\}_i$ where $x_i$ is a multi-dimensional tuple and scattered (not on a equally spaced regular lattice). How does one regress or smooth such a scattered data set ...
0
votes
0answers
30 views

Reproducing Holt-Winters analysis in the Cowpertwait-Metcalfe book

I am having trouble reproducing some output for some R code in the time series book by Cowpertwait and Metcalfe. There are quite a few typos in their code throughout the book, but in other cases I ...
1
vote
0answers
28 views

Transforming a categorical distribution up side down

I have the following categorical distribution: $$ original = (2, 7, 3, 5, 0, 1, 4, 6, 3, 8, 8)\\ \sum original = 47 $$ I want to transform it to its upside down distribution step by step, while ...
1
vote
0answers
19 views

Computing a moving average when data points arrive one at a time

I am sure there's a cool python/numpy/pandas way to do this. I am receiving one data point at a time. I would like to compute something like a moving average over the last n observations, even better ...
1
vote
1answer
438 views

Random effect in GAM - what are the smooth functions used?

In the GAM package in R created by Simon Wood there is a selection of the smooth function basis. I sort of understand the options such as bs='tp', bs='cr', etc. But bs='re' seems odd... that does ...
1
vote
1answer
132 views

What do the outputs of the function dlmSmooth from R's dlm package mean?

In the vignette for the R package dlm: link on page 12, the author runs the function dlmSmooth to smooth the data and the function returns an object which is ...
1
vote
0answers
96 views

Smoothing of experimental data for PCA

I am applying PCA to a set of spectrophotometric measurements with the aim of differentiating two groups of substances. The many small wiggles on the right-hand side of the curves (the region above ~...
0
votes
1answer
120 views

Is a Kalman filter ever the optimal way to estimate a dynamic value given a full history of measurements?

I'm trying to get some intuition for Kalman filtering, and I conceived this toy example: Say that I have a sensor that tracks a moving 1-dimensional target. Say that the measurements from the sensor ...
4
votes
4answers
3k views

Measure of smoothness

I have an image that has artefacts which I am using a specific process to remove. I want to show that the new image is improved by that process. To compare the two images I am using data from a ...
7
votes
3answers
1k views

How to fit a robust step function to a time series?

I have a somewhat noisy time series that hovers around different levels. For example, the following data: I have the solid line data available, and I would like to obtain an estimate for the dashed ...
1
vote
0answers
168 views

Is there a filter function that performs similar to moving average but does not loose data?

I need to smooth a time series using a low pass filter. A simple moving average is working fine for me, however, using a moving average causes an inevitable loss of data a the beginning of the ...
1
vote
0answers
35 views

Expected Value of Cross Validation approximates Predictive Square Error

In the context of Smoothing Splines, Im trying to show that the expected value of the cross-validation can approximate the predictive square error. More specifically, I want to show that $$E[(y_i - \...
1
vote
0answers
139 views

Natural Splines and Smoother Matrix

In the context of smoothing splines, one can show that the Reinsch form is given by: $ \hat{y} = N (N^{T}N +\lambda \Omega)^{-1}N^{T} y = (I+ \lambda K)^{-1}y $ where (1) $K = (N^{T})^{-1}\Omega N^{-...
3
votes
1answer
79 views

How to smooth a curve by learning location and shape of 4 Gaussian kernels?

I have a data set of 365 daily curves $f_i(t)$ and want to smooth them by positioning four radial basis functions $r_i(t)$. Each daily curve should then be approximated by $$ \hat{f}_i(t) = w_{i1} ...
3
votes
0answers
249 views

RKHS norm and Fourier transform link

In the notes here, it is stated that norms of some reproducing kernel Hilbert spaces can be written in terms of Fourier transforms, and this is often used to argue that a higher RKHS norm implies a ...
1
vote
1answer
49 views

Partial spline. Reference

I have a well done and perfectly working protocol to smooth my experimental data. I do the following: I have a variable of size 1000. Iteratively I choose random 100 points and spline them using the ...
3
votes
1answer
74 views

What does “middle of the data” really mean?

I have some noisy time series data for different climate variables and I want to know overall if they are increasing or decreasing with time. From this Water Resources Statistics Guide, the LOWESS ...
2
votes
2answers
469 views

Show that solution to cubic smoothing spline reduces to regular least squares minimization as $\lambda$ approaches infinity

I am asked to show that the solution to a smoothing splines problem of the form $$ \text{PRSS}(f,\lambda) = \sum_{i=1}^N\left[y_i-f(x_i)\right]^2 + \lambda \int f''(t)^2 dt, $$ with $$ f(x) = \sum_{j=...
2
votes
1answer
263 views

Generalized additive models - formula for basis functions

I'm trying to understand the basics of GAMs. Wood's book "Generalized Additive Models: an introduction with R" introduces GAMs via a cubic spline basis $b_j(x)$ (see p. 122), where $b_1(x)=1, b_2(x)=...
5
votes
1answer
688 views

Functional Data Analysis and Splines Regression

I'm new to Functional Data Analysis so my question will'be very simple for an expert in this topic. Operatively, when I fit a model like this, in R : model<- lm( y ~ bs( x, df = k, degree = l ) )...
1
vote
1answer
247 views

Improve estimate of zeros and how to deal with concurvity in GAM NB

I'm trying to model the number of deaths for each age, for 4 regions, across 5 years. I fitted several models, but end up with the following model: ...
1
vote
1answer
576 views

Exponential Smoothing & Seasonality

I am relatively new to stats and forecasting and was hoping to tap the wisdom of this community in regards to a question. One of my colleagues insists that exponential smoothing presumes seasonality. ...
0
votes
0answers
89 views

mgcv optimization: why optimize **log** smooth parameter?

In mgcv package, when optimizing smoothing parameters (in the outer iteration), one take the partial derivative of objective function with respect to $\rho = log(\lambda)$. My question is why not take ...

1
2
3 4 5
7