# Questions tagged [smoothing]

Smoothing methods in data analysis, like splines or kernel smoothers, also regression smoothers like lowess.

40 questions
Filter by
Sorted by
Tagged with
108k views

### How to use Pearson correlation correctly with time series

I have 2 time-series (both smooth) that I would like to cross-correlate to see how correlated they are. I intend to use the Pearson correlation coefficient. Is this appropriate? My second question ...
2k views

### What is a density function?

I know about histograms and also know that if we connect the mid-points on the top of bars in a histogram we will get a frequency polygon. This polygon could then be 'smoothed' in a way that it ...
9k views

### How to smooth data and force monotonicity

I have some data which I would like to smooth so that the smoothed points are monotonically decreasing. My data sharply decreases and then begins to plateau. Here's an example using R ...
6k views

### Choosing a bandwidth for kernel density estimators

For univariate kernel density estimators (KDE), I use Silverman's rule for calculating $h$: \begin{equation} 0.9 \min(sd, IQR/1.34)\times n^{-0.2} \end{equation} What are the standard rules for ...
3k views

### Rationale for the use of Regressogram (Bin-Smooth)

I am taking a class in data mining and we have recently been introduced to bin-smoothing in regression analysis but i cannot seem to understand the usefulness of this method nor how the method works ...
9k views

### Alternate distance metrics for two time series

I have time-series data of different houses. Assume it is power consumption data. Now, I want to cluster the houses following similar power consumption pattern utmost. So, the various distance metrics ...
4k views

### How to smear a histogram

I was asked to perform a Gaussian smearing on the bins of an histogram. What does this mean?
293 views

### Smoothness of a surface

I am currently working on a model which takes two parameters and produces a measurement statistic. Think of it as Z = f(X,Y). Z is a matrix of my statistics and I am creating a surface plot of it in ...
277 views

### Forcing smoothness of regression coefficients

I'm building regression models on spectral datasets: the predictors are the intensites of signal at the different frequencies. In this case the intensities at close frequency values are highly ...
2k views

### Practical description of LOESS and smoothing splines?

I understand roughly that LOESS is a smoothing procedure that uses splines to interpolate data. However, I also know that outlier removal is a large problem with LOESS. I was trying to estimate a non-...
6k views

### When will a Kalman filter give better results than a simple moving average?

I recently implemented a Kalman filter on the simple example of measuring a particles position with a random velocity and acceleration. I found that Kalman filter worked well, but I then asked myself ...
3k views

### Knots in Smoothing Splines

In Introduction to Statistical Learning, there's this line under the section describing Smoothing Spline's tuning parameter $\lambda$: In fitting a smoothing spline, we do not need to select the ...
5k views

### Smoothing methods for gam in mgcv package?

I am currently working with gam models in the mgcv package and for me the smoothing methods are a bit confusing and I hope that you guys can help me to understand that better. So here is what I've ...
763 views

### Savitzky-Golay (aka Hodrick–Prescot or Whittaker-Henderson) vs. Kernel

Is there a clear analytic link from Kernel smoothing, particularly the Nadaraya–Watson estimator, (S-G, H-P, or W-H) smoothing filter? The "filter" is called by different names in different fields ...
4k views

### The Pros and Cons of Smoothing spline

I have a general question. Recently I just learnt Basis Expansion and Regularization. There are several interesting techniques including: cubic spline, natural spline, b-spline and smoothing spline. ...
12k views

### Smoothing - when to use it and when not to?

There is quite an old post on William Briggs' blog which looks at the pitfalls of smoothing data and carrying that smoothed data through to analysis. The key argument is namely: If, in a moment of ...
20k views

### Finding inflection points in R from smoothed data

I have some data that I smooth using loess. I'd like to find the inflection points of the smoothed line. Is this possible? I'm sure someone has made a fancy method ...
15k views

### How to tune smoothing in mgcv GAM model

I am trying to figure out how to control the smoothing parameters in an mgcv:gam model. I have a binomial variable I am trying to model as primarily a function of x and y coordinates on a fixed grid, ...
755 views

### Invertibility in Reinsch form Derivation (Smoothing Splines)

In Element of Statistical Learning II, in the context of smoothong splines, $\pmb{S_{\lambda}}$ is defined as $$\pmb{S_{\lambda}} = \pmb{N}(\pmb{N}^T\pmb{N} + \lambda \pmb{\Omega_N})^{-1}\pmb{N}^T$$ ...
705 views

### If variable kernel widths are often good for kernel regression, why are they generally not good for kernel density estimation?

This question is prompted by discussion elsewhere. Variable kernels are often used in local regression. For example, loess is widely used and works well as a regression smoother, and is based on a ...
11k views

### How to obtain the values used in plot.gam in mgcv?

I would like to find out the values (x, y) used in plotting plot(b, seWithMean=TRUE) in mgcv package. Does anyone know how I ...
1k views

### How to fit a robust step function to a time series?

I have a somewhat noisy time series that hovers around different levels. For example, the following data: I have the solid line data available, and I would like to obtain an estimate for the dashed ...
3k views

### Truncated power basis function and continuity in b-splines

I do not understand how adding a truncated power basis function leads to continuity in B-Splines. Could someone please provide a low level explanation?
5k views

### Spline fitting in R - how to force passing two data points?

I am using "smooth.spline" in R. Here is a snippet from the documentation: http://stat.ethz.ch/R-manual/R-patched/library/stats/html/smooth.spline.html smooth.spline {stats} R Documentation Fit a ...
65 views

### What's the best way to find outliers in the time-series, encountering that it is a real-world mechanical process (process continuity)?

What's the best way to find outliers in the time-series, encountering continuity? I attached two time-series that I'm interested to filter. One is less noisy, and one is a bit noisier. I'm mostly ...
3k views

### Variance of a smoothed AR(1) process

The query I have relates to calculating the variance of AR(1) processes that are smoothed with a simple moving average. So: In an AR(1) process of the form: $$X_t=c+\varphi X_{t-1}+\varepsilon_t,$$...
4k views

### scatterplot smoothing in r with big dataset: different methods

I have a large dataset (>300,000 rows) with two variables. y is binary and x is continuous & numeric. I'd like to plot y and add smooth curve against x. I understand that loess(y~x) is a solution, ...
601 views

### How can I robustly smooth my time series data?

I have data like the following image, where the x-axis is the absolute elapsed time in hours (think calendar days; this plot goes over ~2.5 years), and the y-axis is the manually entered uptime of a ...
157 views

Kernel based operations are common in a variety of applications, such as image processing (e.g., blurring), generating smoothed estimation maps, and so on. A common approach is to select four ...
1k views

### Holt Winters initialization using backcasting like SPSS

I'm trying to figure out how SPSS is initializing the multiplicative Holt Winters exponential smoothing model using backcasting. Thankfully IBM roughly described their way of doing so .. very roughly. ...
299 views

### AIC on Savitzky-Golay width

I want to use a Savitzky-Golay filter to smooth some data. There is a right width to use based on the data that it is smoothing. A number of papers basically use "eyeball norm" on the parameters but ...
1k views

### Plotting smoothed hazard ratio intervals for interaction terms

I don't know if it is possible. I am following Terry Therneau's Spline terms in a Cox model vignette available for survival package. In Section 3, Splines in an ...
324 views

### How does the penalized form of RSS (residual sum of squares) work?

In another word, how to reverse engineering the equation (5.9) by explain all the assumption and reasoning after the plus sign of (5.9) in Elements of Statistical Learning: \text{RSS}(f,\lambda) = \...
234 views

### How to explain certain patterns appearing after kernel averaging?

Having a 2D map filled uniformly by random values (Figure:top-left), the next maps are kernel averaged with a kernel of sizes ...
1k views

### Locally weighted regression vs. splines

What's the pros/cons of splines approaches compared to locally weighted regression approaches for the purposes of (a) scatter plot smoothing and (b) prediction? Obviously, in the case of prediction I ...
536 views

### Strange effective degrees of freedom (smoothness) selected for smooth component in GAM model with mgcv

Consider the following very simple example: ...
117 views

### Questions re. fitting a polynomial: smoothing, cross-validation, etc

I have data on hospital treatment times. I would like to fit a polynomial to the data. My data comes in 5 minute increments and it is very noisy. It looks like this: I can aggregate to a higher ...
119 views

### What is the lag associated with Moving Average smoothing?

In a tutorial I came across this: "Recall that the forecast value is: $\hat{y}_{t+1} = \frac{y_t + y_{t-1} + ... + y_{t-m+1}}{m}$ It's worth pondering that formula for a minute. While easy to ...