The smoothing tag has no wiki summary.
0
votes
0answers
9 views
How to quantify mis-specification bias and compare against smoothing bias for a non-parametric estimate of a randomly allocated continuous treatment?
Suppose that there is a data-generating process
$$
y = \alpha + g(x) + \epsilon
$$
which is to say that an outcome is some function of $x$. Suppose that $x$ is randomly assigned, so ...
0
votes
0answers
20 views
Curve smoothness - local adjacency
I am looking for statistical measures of curve smoothness.
Time-series values {(1, 0.5), (4, -0.6), (200, 1.0)} where (time-unit, value) is linearly interpolated from one to the next.
The smoothest ...
4
votes
2answers
67 views
Kernel density estimator that doesn't collapse in the tails
I have iid data-points $x_1, \dots, x_n$, generated by an unknown density $f(x)$.
So far I have approximated $f(x)$ with a normal $N(\hat{\mu}, \hat{\sigma}^2 )$, where $\hat{\mu}$ and ...
2
votes
1answer
45 views
Mean squared error definition
I'm currently working through (part of) a textbook on non-parametric regression techniques. Regarding the choice of smoothing parameter the book starts out explaining the MSE which is defined as:
...
0
votes
0answers
20 views
Smoothing of log-distributed periodogram
I use the lomb-scargle periodogram to output information about chemical species in distinct time periods. This produces a distribution that is skewed heavily, with the majority of points and variance ...
1
vote
0answers
24 views
Differentiating experimental data
I'm trying to differentiate experimental data which are measured with time. However, the time increments are not equally spaced.
I first obtained a trend line for these data and then did a linear ...
3
votes
2answers
62 views
Smoothing algorithm for irregular time interval
I have various sets of irregular interval time series data to which I want to apply some sort of smoothing algorithm to produce a good fit.
I have attempted various methods which all were ...
1
vote
0answers
60 views
Is LOESS the appropriate way of visualizing my RT data?
I am conducting a Lexical Decision Task where my dependent random variable is Response Time (RT).
My experimental design consists of 5 blocks of a 100 trials each. In each block, 50% of trials ...
0
votes
2answers
59 views
2d interpolation method for coarsely sampled image
I'm looking for a general method for 2d interpolation of a coarsely sampled image. I'll use an example, taken from the scipy.interpolate (Python) page.
Say, I have this image, but instead of ...
2
votes
1answer
131 views
Rationale for the use of Regressogram (Bin-Smooth)
I am taking a class in data mining and we have recently been introduced to bin-smoothing in regression analysis but i cannot seem to understand the usefulness of this method nor how the method works ...
1
vote
0answers
23 views
Smoothing objective function
I have a current problem where i have a continuous error real function of n-variables, which is computationally expensive and (possibly) has multiple local minima (optima). I want to somehow smooth ...
0
votes
1answer
47 views
Computing lowess mean value
I am trying to reimplement lowess algorithm in java. I read in the matlab page explaining the lowess with the follwing steps:
Compute the regression weights for each data point in the span.
A ...
1
vote
0answers
55 views
Computing weighted standard deviation using lowess mean values
I have two questions:
First question:
I want to compute the weighted standard deviation with tri-cubic kernel. I am using lowess function in R to compute the weighted mean using tri-cubic ...
7
votes
5answers
388 views
Smoothing 2D data
The data consist of optical spectra (light intensity against frequency) recorded at varying times. The points were acquired on a regular grid in x (time) , y (frequency). In order to analyse the time ...
1
vote
0answers
77 views
Automatically choose a bandwidth in R
I have a function that I have added noise to. I want to test a few smoothers, but for each smoother I want to minimise MSE. I know the underlying function, so how can I make R automatically choose the ...
1
vote
0answers
36 views
Good-Turing smoothing when everything appears once
I would like to use Good-Turing smoothing to find the probability of the next type of fish I will catch. If I have caught the following already:
...
4
votes
2answers
104 views
Smoothing a time series of ratios
I have a time series of proportions, $x_t = \frac{a_t}{b_t}$ i.e.
$x_1 = 2 / 30, x_2 = 1/10$, ...
I want to smooth $x_t$. Should I apply a smoothing function directly to $x_t$, or should I smooth ...
1
vote
1answer
173 views
Process for Standardising and Normalising data
First: I'm not well versed in statistics terminology so please forgive me - I'll try to be as verbose as possible with my problem.
This is a problem which I've previously solved very naively. I'm ...
2
votes
0answers
70 views
Faster alternative to multivariate LOESS?
I want to make predictions by creating a smooth response surface to 2 variables. I get good results using R's loess() function, but with 10 million observations, it is far too slow. Are there any ...
2
votes
0answers
47 views
Smooth expectations outside the exponential family
At page 85-86 of Young and Smith "Essentials of Statistical Inference" there
is an interesting result. If $X$ is a r.v. distributed according to the exponential family and $\phi(x)$ is a bounded (but ...
1
vote
0answers
225 views
optimal choice of smooth.spline parameter?
I'm analyzing a time series (terms of trades) on which I want to perform a trend estimation by nonparametric methods like the above mentioned. By the way, I'm a total beginner with R and using the ...
2
votes
0answers
67 views
local polynomial regression standard errors
I am attempting to find a reference which explains how one computes standard errors for local polynomial regression? Specifically, in R one can use the loess ...
4
votes
1answer
92 views
Generalizing Add-one/Laplacian Smoothing
Let us assume we are estimating a proportion or rate of "hits". If we have $h$ hits and $m$ misses, the obvious estimator is
$\dfrac{h}{h + m}$
In order to avoid unreasonable estimations of $0$ or ...
2
votes
1answer
78 views
Smoothing a 2-by-2 contingency table
I am trying to implement a system for automatic document categorization, where each document of a corpus belongs to some class. I define the following contingency table for every class C and every ...
7
votes
1answer
387 views
When will a Kalman filter give better results than a simple moving average?
I recently implemented a Kalman filter on the simple example of measuring a particles position with a random velocity and acceleration. I found that Kalman filter worked well, but I then asked myself ...
2
votes
2answers
240 views
I have a very fuzzy data set, what can I do to 'smooth' it?
Can you suggest some pointers on how I can manipulate my data to be 'smoother'? Any algorithms, or techniques that would be useful in this aspect?
Updates:
In response to the comments asking for ...
1
vote
0answers
119 views
N-gram smoothing implementation in R
Is there an implementation of n-gram smoothing like Kneser-Ney in R?
I'm using the tm package.
3
votes
0answers
32 views
How to find 1-year population average with intermittent series data
I have performance review data and scores (ranging from 1 to 4) for employees of a company. I need to show the company average over the past year. However, the employees were only ever reviewed for a ...
1
vote
2answers
607 views
Spline fitting in R - how to force passing two data points?
I am using "smooth.spline" in R. Here is a snippet from the documentation:
http://stat.ethz.ch/R-manual/R-patched/library/stats/html/smooth.spline.html
smooth.spline {stats} R Documentation
Fit a ...
4
votes
3answers
200 views
Curve smoothing in the presence of non-gaussian uncertainty
What options are available for smoothing 2-dimensional real data for which the the ordinate points are real intervals of the form
$(x_j , [y_{j0} , y_{j1}])$
In my case, the data is vague because of ...
5
votes
1answer
331 views
State-of-the-art in smoothing splines
What is the state-of-the-art in the efficient computation of smoothing splines?
The algorithm I see mentioned most often is that of Reinsch, dating back to 1967. As I understand it, the most ...
0
votes
1answer
84 views
Numeric methods to estimate curve parameter from 2 input data with Gaussian noise
I have:
a function of the form
$S=M f(t, \theta)$
where $\theta$ is a variable, $t$ and $M$ are parameters.
two observations
$(\theta_1, S_1)$ and $(\theta_2, S_2)$
in which $S_1$ and ...
4
votes
1answer
167 views
SVD of a data matrix after smoothing
Let's say I have a (n x m) centered data matrix $A$ with SVD $A = U \Sigma V^{T}$.
For example, m=50 columns (measurements) that are spectra with n=100 different frequencies. The matrix is centered ...
1
vote
1answer
101 views
Monte Carlo test for comparing curvature of binomial response surfaces from effective degrees of freedom of GCV-fitted splines
I would like to compare the curvature of two response surfaces, each of the form:
binomial ~ continuous variables1-5
I think it would be appropriate to use the effective degrees of freedom of a ...
4
votes
2answers
149 views
How to select the bandwith of smoothed bootstrap for building confidence intervals?
I am interested in non-parametric methods for building confidence intervals for an estimator (e.g. the mean) using few samples (e.g. 10). I think I have read somewhere that smoothing the bootstrapped ...
4
votes
1answer
274 views
Negative weights in a moving average?
A number of well known moving averages, such as Spencer's 15 point MA, and Henderson moving averages have negative weights in the averages.
What does this mean in a conceptual sense? What information ...
2
votes
2answers
279 views
What to smooth fit a curve to data in order to identify an elbow?
I ran an experiment and obtained the points shown in black below.
I would like to smooth the curve or fit (something like the red curve) in order to identify the elbow.
The problem is that the ...
2
votes
1answer
87 views
Smoothing function for displaying stacked lines without the smoothing introducing crossings
I'm displaying time-series data as a "stacked line" or "stacked area" chart. (E.g. with percentage data, data points at 10%, 20% and 30% are displayed at 10%, 30% and 60% on the chart.) Unsmoothed ...
1
vote
1answer
194 views
What's the difference between dlmSmooth and dlmFilter in R's dlm package?
Could someone please explain what the difference is between the two, and perhaps avoid the worst statistical jargon?
I am currently using the dlm package to model ...
1
vote
1answer
294 views
Kalman smoothing of returns vs. prices with dlmSmooth in R's dlm package?
So I am using the R code behind Fig. 3.14 in Dynamic Linear Models With R (p. 124-5) to make a dynamic version of a simple pair trading model:
$$
Y = \alpha + \beta X.
$$
If I use log returns ...
0
votes
0answers
55 views
local ordinal regressions
I have found no tool to estimate an "oprobit"-like ordinal model using the local regression (local likelihood) framework. It's not built into R's locfit. What is a good way to proceed?
Example:
I ...
2
votes
0answers
161 views
How to explain certain patterns appearing after kernel averaging?
Having a 2D map filled uniformly by random values (Figure:top-left), the next maps are kernel averaged with a kernel of sizes ...
3
votes
1answer
200 views
Difference between ~lp() or simply ~ in R's locfit [closed]
(Please note the cross-post at http://stackoverflow.com/questions/7626347/difference-between-lp-or-simply-in-rs-locfit)
I am not sure I see the difference between different examples for local ...
6
votes
1answer
98 views
Smoothing when standard errors are known/estimated
I have a set of estimates of function values, along with estimates of their standard errors. To somewhat simplify matters: for $x$ running from $1$ up to $70$ in (integer) steps (in fact a parameter ...
3
votes
0answers
96 views
How to denoise a “Poissonous” time series
I have $N$ time series each of which can be modeled as $$y_{kt}=Ax_{kt}+b+\varepsilon_{kt}\quad(1\le k\le N,1\le t\le T),$$ where $x_{kt}\sim\text{Pois}(\lambda\Delta t)$ and $\varepsilon_{kt}\sim ...
4
votes
1answer
868 views
How to tune smoothing in mgcv GAM model
I am trying to figure out how to control the smoothing parameters in an mgcv:gam model.
I have a binomial variable I am trying to model as primarily a function of x and y coordinates on a fixed grid, ...
3
votes
1answer
151 views
Constrained kernel density estimation
Suppose you are trying to estimate the joint density $p(x,y)$ based on observed $(X,Y)$. However, you know that the marginal density $p(x)$ is uniform. How can you use this information to improve ...
3
votes
1answer
58 views
How can I even out a random distribution while minimising how far each data point is moved?
I have a software application which uses a queue and multiple processors to process those jobs. Jobs get re-run on a daily basis for customers, but we also have new customers signing up regularly.
...
1
vote
0answers
485 views
Is it OK to do additive smoothing before applying Pearson's chi-square test for independence?
I'm concerned about treating my data as gold, especially in areas of low data support, so I would like to apply additive smoothing. I'm then doing several things with this data, and one of them is ...
3
votes
0answers
42 views
Updating a set of estimated forecasts
Suppose I have some stochastic process $X_t$. At each time $t$, I receive an estimated probability distribution for $x_t$, followed by an observation $x_t$. After receiving a set of observations ...
