Questions tagged [smoothing]

Smoothing methods in data analysis, like splines or kernel smoothers, also regression smoothers like lowess.

Filter by
Sorted by
Tagged with
66
votes
3answers
108k views

How to use Pearson correlation correctly with time series

I have 2 time-series (both smooth) that I would like to cross-correlate to see how correlated they are. I intend to use the Pearson correlation coefficient. Is this appropriate? My second question ...
3
votes
1answer
2k views

What is a density function?

I know about histograms and also know that if we connect the mid-points on the top of bars in a histogram we will get a frequency polygon. This polygon could then be 'smoothed' in a way that it ...
19
votes
2answers
9k views

How to smooth data and force monotonicity

I have some data which I would like to smooth so that the smoothed points are monotonically decreasing. My data sharply decreases and then begins to plateau. Here's an example using R ...
29
votes
2answers
6k views

Choosing a bandwidth for kernel density estimators

For univariate kernel density estimators (KDE), I use Silverman's rule for calculating $h$: \begin{equation} 0.9 \min(sd, IQR/1.34)\times n^{-0.2} \end{equation} What are the standard rules for ...
4
votes
1answer
3k views

Rationale for the use of Regressogram (Bin-Smooth)

I am taking a class in data mining and we have recently been introduced to bin-smoothing in regression analysis but i cannot seem to understand the usefulness of this method nor how the method works ...
11
votes
2answers
9k views

Alternate distance metrics for two time series

I have time-series data of different houses. Assume it is power consumption data. Now, I want to cluster the houses following similar power consumption pattern utmost. So, the various distance metrics ...
7
votes
2answers
4k views

How to smear a histogram

I was asked to perform a Gaussian smearing on the bins of an histogram. What does this mean?
4
votes
1answer
293 views

Smoothness of a surface

I am currently working on a model which takes two parameters and produces a measurement statistic. Think of it as Z = f(X,Y). Z is a matrix of my statistics and I am creating a surface plot of it in ...
6
votes
2answers
277 views

Forcing smoothness of regression coefficients

I'm building regression models on spectral datasets: the predictors are the intensites of signal at the different frequencies. In this case the intensities at close frequency values are highly ...
5
votes
1answer
2k views

Practical description of LOESS and smoothing splines?

I understand roughly that LOESS is a smoothing procedure that uses splines to interpolate data. However, I also know that outlier removal is a large problem with LOESS. I was trying to estimate a non-...
15
votes
2answers
6k views

When will a Kalman filter give better results than a simple moving average?

I recently implemented a Kalman filter on the simple example of measuring a particles position with a random velocity and acceleration. I found that Kalman filter worked well, but I then asked myself ...
6
votes
2answers
3k views

Knots in Smoothing Splines

In Introduction to Statistical Learning, there's this line under the section describing Smoothing Spline's tuning parameter $\lambda$: In fitting a smoothing spline, we do not need to select the ...
6
votes
2answers
5k views

Smoothing methods for gam in mgcv package?

I am currently working with gam models in the mgcv package and for me the smoothing methods are a bit confusing and I hope that you guys can help me to understand that better. So here is what I've ...
3
votes
1answer
763 views

Savitzky-Golay (aka Hodrick–Prescot or Whittaker-Henderson) vs. Kernel

Is there a clear analytic link from Kernel smoothing, particularly the Nadaraya–Watson estimator, (S-G, H-P, or W-H) smoothing filter? The "filter" is called by different names in different fields ...
16
votes
1answer
4k views

The Pros and Cons of Smoothing spline

I have a general question. Recently I just learnt Basis Expansion and Regularization. There are several interesting techniques including: cubic spline, natural spline, b-spline and smoothing spline. ...
18
votes
2answers
12k views

Smoothing - when to use it and when not to?

There is quite an old post on William Briggs' blog which looks at the pitfalls of smoothing data and carrying that smoothed data through to analysis. The key argument is namely: If, in a moment of ...
16
votes
5answers
20k views

Finding inflection points in R from smoothed data

I have some data that I smooth using loess. I'd like to find the inflection points of the smoothed line. Is this possible? I'm sure someone has made a fancy method ...
17
votes
1answer
15k views

How to tune smoothing in mgcv GAM model

I am trying to figure out how to control the smoothing parameters in an mgcv:gam model. I have a binomial variable I am trying to model as primarily a function of x and y coordinates on a fixed grid, ...
6
votes
1answer
755 views

Invertibility in Reinsch form Derivation (Smoothing Splines)

In Element of Statistical Learning II, in the context of smoothong splines, $\pmb{S_{\lambda}}$ is defined as $$ \pmb{S_{\lambda}} = \pmb{N}(\pmb{N}^T\pmb{N} + \lambda \pmb{\Omega_N})^{-1}\pmb{N}^T $$ ...
17
votes
2answers
705 views

If variable kernel widths are often good for kernel regression, why are they generally not good for kernel density estimation?

This question is prompted by discussion elsewhere. Variable kernels are often used in local regression. For example, loess is widely used and works well as a regression smoother, and is based on a ...
12
votes
4answers
11k views

How to obtain the values used in plot.gam in mgcv?

I would like to find out the values (x, y) used in plotting plot(b, seWithMean=TRUE) in mgcv package. Does anyone know how I ...
7
votes
3answers
1k views

How to fit a robust step function to a time series?

I have a somewhat noisy time series that hovers around different levels. For example, the following data: I have the solid line data available, and I would like to obtain an estimate for the dashed ...
7
votes
2answers
3k views

Truncated power basis function and continuity in b-splines

I do not understand how adding a truncated power basis function leads to continuity in B-Splines. Could someone please provide a low level explanation?
6
votes
2answers
5k views

Spline fitting in R - how to force passing two data points?

I am using "smooth.spline" in R. Here is a snippet from the documentation: http://stat.ethz.ch/R-manual/R-patched/library/stats/html/smooth.spline.html smooth.spline {stats} R Documentation Fit a ...
1
vote
1answer
65 views

What's the best way to find outliers in the time-series, encountering that it is a real-world mechanical process (process continuity)?

What's the best way to find outliers in the time-series, encountering continuity? I attached two time-series that I'm interested to filter. One is less noisy, and one is a bit noisier. I'm mostly ...
8
votes
2answers
3k views

Variance of a smoothed AR(1) process

The query I have relates to calculating the variance of AR(1) processes that are smoothed with a simple moving average. So: In an AR(1) process of the form: $$ X_t=c+\varphi X_{t-1}+\varepsilon_t, $$...
5
votes
2answers
4k views

scatterplot smoothing in r with big dataset: different methods

I have a large dataset (>300,000 rows) with two variables. y is binary and x is continuous & numeric. I'd like to plot y and add smooth curve against x. I understand that loess(y~x) is a solution, ...
3
votes
1answer
601 views

How can I robustly smooth my time series data?

I have data like the following image, where the x-axis is the absolute elapsed time in hours (think calendar days; this plot goes over ~2.5 years), and the y-axis is the manually entered uptime of a ...
2
votes
0answers
157 views

About kernel based estimates

Kernel based operations are common in a variety of applications, such as image processing (e.g., blurring), generating smoothed estimation maps, and so on. A common approach is to select four ...
1
vote
0answers
1k views

Holt Winters initialization using backcasting like SPSS

I'm trying to figure out how SPSS is initializing the multiplicative Holt Winters exponential smoothing model using backcasting. Thankfully IBM roughly described their way of doing so .. very roughly. ...
3
votes
2answers
299 views

AIC on Savitzky-Golay width

I want to use a Savitzky-Golay filter to smooth some data. There is a right width to use based on the data that it is smoothing. A number of papers basically use "eyeball norm" on the parameters but ...
3
votes
2answers
1k views

Plotting smoothed hazard ratio intervals for interaction terms

I don't know if it is possible. I am following Terry Therneau's Spline terms in a Cox model vignette available for survival package. In Section 3, Splines in an ...
3
votes
1answer
324 views

How does the penalized form of RSS (residual sum of squares) work?

In another word, how to reverse engineering the equation (5.9) by explain all the assumption and reasoning after the plus sign of (5.9) in Elements of Statistical Learning: $$\text{RSS}(f,\lambda) = \...
2
votes
0answers
234 views

How to explain certain patterns appearing after kernel averaging?

Having a 2D map filled uniformly by random values (Figure:top-left), the next maps are kernel averaged with a kernel of sizes ...
2
votes
0answers
1k views

Locally weighted regression vs. splines

What's the pros/cons of splines approaches compared to locally weighted regression approaches for the purposes of (a) scatter plot smoothing and (b) prediction? Obviously, in the case of prediction I ...
2
votes
1answer
536 views

Strange effective degrees of freedom (smoothness) selected for smooth component in GAM model with mgcv

Consider the following very simple example: ...
2
votes
2answers
117 views

Questions re. fitting a polynomial: smoothing, cross-validation, etc

I have data on hospital treatment times. I would like to fit a polynomial to the data. My data comes in 5 minute increments and it is very noisy. It looks like this: I can aggregate to a higher ...
1
vote
3answers
119 views

What is the lag associated with Moving Average smoothing?

In a tutorial I came across this: "Recall that the forecast value is: $\hat{y}_{t+1} = \frac{y_t + y_{t-1} + ... + y_{t-m+1}}{m}$ It's worth pondering that formula for a minute. While easy to ...
1
vote
2answers
1k views

What is the effect of having skewed dependent variable on scatterplot result

The histogram of my dependent variable is as following: I draw the scatter plot of my dependent variable and independent variable, and the result is as following picture? I am wondering if the ...
1
vote
0answers
248 views

What precautions should one take when building a model in two stages?

My data has too many features to be estimated with a single predictive model (see How to model a stochastic trend in the response variable of a regression?). In consequence, I'm considering a time ...