The tag has no wiki summary.

learn more… | top users | synonyms

1
vote
0answers
16 views

In Kneser-Ney smoothing, how are unseen words handled?

From what I have seen, the (second-order) Kneser-Ney smoothing formula is in some way or another given as $ \begin{align} P^2_{KN}(w_n|w_{n-1}) &= \frac{\max \left\{ C\left(w_{n-1}, w_n\right) - ...
1
vote
2answers
51 views

What is the effect of having skewed dependent variable on scatterplot result

The histogram of my dependent variable is as following: I draw the scatter plot of my dependent variable and independent variable, and the result is as following picture? I am wondering if the ...
1
vote
0answers
5 views

Smoothing strategies for features assuming values from countably infinite domains

I am in the midst of programming a simple Naive Bayes classifier as an exercise. It is supposed to perform word-sense disambiguation on natural language phrases, e.g. predicting the correct meaning of ...
1
vote
0answers
14 views

How to compare fit of discrete process with discrete underlying process?

I am basically looking for an equivalent to something like an $\mathbb{R}^2$ for a model on a dataset that is itself simply a collection of points. That is, if my data set is (trivial example): ...
0
votes
0answers
27 views

How to apply Gaussian kernel to smooth density of points on 2D (theoretically)

I have a set of discrete points on a 2D surface and need to build a heat map or a distribution of the density of the points. However, I also need to smooth out the density/distribution by applying ...
0
votes
0answers
13 views

In practice, how is the penalisation matrix for splines, created by smoothCon (package mgcv), specified?

Evaluating a smooth object with smoothCon provides, besides several other things, the ordinary "untouched" spline bases ...
2
votes
2answers
86 views

Which forecasting method for load profiles

I'm new to this forum and I'm quite new to forecasting. Currently I'm trying to learn the basics about exponential smoothing, ARIMA etc. Now I want to forecast the total energy consumption of a rather ...
0
votes
1answer
25 views

What R packages are available for nonparametric regression of two predictors?

I would like to fit a nonparametric regression model with two predictors. What R packages are available? I found 'loess' function in 'stats' package and 'gam' function in 'gam' package. Is there ...
0
votes
0answers
14 views

How are local and global plug-in bandwidths different in kernel smoothing regression?

I'm looking into an R package 'lokern.' It provides two bandwidths selectors, global and local plug-in bandwidth. I would like to understand the difference between two methods. My understanding is ...
1
vote
0answers
27 views

How can I smooth a set of discrete data points for the purpose of schedule planning?

Disclaimer: I do not have a background in statistics or the math behind filtering, save one long-time-ago college course. I have a well defined problem space. I am calculating hourly staffing ...
1
vote
1answer
160 views

Smoother lines for ggplot2

This question probably has a simple solution, still the thing is I've written a code to plot mortality in 2 different groups and that is, death in obese patients vs not obese. Now their are 2 groups ...
3
votes
3answers
79 views

In Naive Bayes, why bother with Laplacian smoothing when we have unknown words in the test set?

I was reading over Naive Bayes Classification today. I read, under the heading of Parameter Estimation with add 1 smoothing: "Let $c$ refer to a class (such as Positive or Negative), and let $w$ ...
0
votes
0answers
11 views

Invalid measurement data removing

I have a lot of data (temperatures and similar things) collected using automatic devices. Lots of these data is let's call it smooth: even when the temperature increases/decreases fast there are few ...
3
votes
1answer
69 views

scatterplot smoothing in r with big dataset: different methods

I have a large dataset (>300,000 rows) with two variables. y is binary and x is continuous & numeric. I'd like to plot y and add smooth curve against x. I understand that loess(y~x) is a solution, ...
4
votes
1answer
74 views

Smoothing dirty data?

I'm trying to do analysis on revenue data and due to circumstances beyond my pay grade, there are large spikes in the data around the time financial reports are due, followed by corresponding low ...
0
votes
0answers
27 views

How to compare GCV diagnostic plots of a spline object

I am trying to analyse diagnostic plots of a spline object, I am using the package Fields from R. I am lost with the GCV function plot, since I can't find any guidelines of what to look for when ...
3
votes
1answer
37 views

How can I estimate the tail of a distribution with a truncated distribution?

The broadband speed data I'm working with have all values over 30Mbps placed into a >30 category. The distribution is thus truncated. This leads to the final column in the histogram below being a ...
2
votes
2answers
279 views

Kalman Filtering, Smoothing and Parameter Estimation for State Space Models in R and C#

I am in the process of writing an open source State Space Analysis suite in C# (for fun). I have implemented a number of different Kalman-Based Filters (Kalman Filter, Information Filter and the ...
1
vote
0answers
52 views

Supersmoother algorithm - tutorial?

I came across a reference to a "Supersmoother" algorithm that was applied to prices in a timeseries. Does anyone know of a tutorial in Python or whether Pandas has such a capability?
1
vote
1answer
93 views

Kernel density estimator function explanation needed

I'm studying about kernel density estimation and from wikipedia I get this formula: $$\hat{f}_h(x, h) = \frac{1}{n}\sum^{n}_{i=1}K_h(x-x_i) = \frac{1}{nh}\sum_{i=1}^nK(\frac{x-x_i}{h}).$$ I think ...
0
votes
0answers
53 views

How to smooth time-series NDVI data using polynomial regression

I have a time-series NDVI image. The image has 26 bands. 26 bands mean that images taken in 8-day time interval and counted in Julian days (97 to 297). For example; first band of the image is NDVI ...
7
votes
2answers
272 views

Selection of k knots in regression smoothing spline equivalent to k categorical variables?

I'm working on a predictive cost model where the patient's age (an integer quantity measured in years) is one of the predictor variables. A strong nonlinear relationship between age and risk of a ...
1
vote
1answer
58 views

R - How to smooth a conversion of weekly sales into daily sales?

I have weekly sales figures, and would like to convert them into daily sales figures, making a simple hypothesis that there are 7 days with equal sales "power". Let's imagine that I have: ...
1
vote
0answers
27 views

Line smoothness/waviness

I am working on a project which requires me to watch video of athletes and measure the straightness/smoothness/waviness, whatever term is acceptable, of their spine. Dividing the spine into segments ...
2
votes
0answers
63 views

Modeling binary outcomes - inaccurate model when using logistic regression?

I am trying to model the probability of a binary outcome with the independent variable being an hour variable. I understand that linear regression is not the correct method for this type of task (I ...
6
votes
1answer
70 views

Kernel smoothing for Edgeworth expansion

Suppose I have an estimator which includes an indicator function in the objective function, then the objective function is not smooth. But if I want to approximate the behavior of this estimator in ...
0
votes
0answers
26 views

What is the difference between the metric window width and Nearest-neighbor's window in Kernel Smoothing methods?

I'm learning Kernel smoothing methods. I didn't really get the difference between the metric window width and Nearest-neighbor's window. For me both seem the same. Can anybody explain it to me? for ...
1
vote
0answers
48 views

How to set initial values for Holt-Winters backcasting?

I'm trying to figure out how to set initial seasonal values for backcasting of the multiplicative Holt Winters. As pointed out in this thread: Holt Winters initialization using backcasting like SPSS, ...
1
vote
0answers
57 views

smoothing nodes values on a graph given adjacency matrix

I am currently looking for a method to smooth values on a graph (composed of vertices and edges). For example I have a graph with a set of nodes V and I want to be able to smooth it. I could have ...
1
vote
1answer
100 views

What is a density function?

I know about histograms and also know that if we connect the mid-points on the top of bars in a histogram we will get a frequency polygon. This polygon could then be 'smoothed' in a way that it ...
5
votes
2answers
463 views

Laplace smoothing and Dirichlet prior

On the wikipedia article of Laplace smoothing (or additive smoothing), it is said that from a Bayesian point of view, this corresponds to the expected value of the posterior distribution, using a ...
0
votes
1answer
240 views

Using Moving-average smoothing in forecast package [closed]

I tried to use the non-centred moving average, that means just using past values by setting the option centre = FALSE, but unfortunately you get the centred results. Can anyone detect the failure ...
3
votes
0answers
195 views

confidence band around a smoothed function

I am using earth packageearth: Multivariate Adaptive Regression Spline Models regression to get a constant piecewise approximation of my data. I want to plot a band ...
1
vote
1answer
136 views

LOESS smoothing fit

Here are 3 questions about the LOESS smoothing fit. ...
0
votes
0answers
54 views

LOOCV for smoothing spline

The smoothing spline problem is $$ RSS(\theta,\lambda)=(y-N\theta)^T(y-N\theta)+\lambda\theta^T\Omega_N\theta $$ where $\{N\}_{ij}=N_j(x_i)$, $N_j(x)$ are an $N$-dimensional set of basis functions. ...
1
vote
2answers
80 views

Bayesian inference on possibly-non-linear effects

In my field, it is occasionally the case that we want to evaluate the degree to which some variable, Y, might be influenced by another variable, X, where X is measured across a range of continuous ...
1
vote
0answers
26 views

Data smoothing with derivative constraints

Which existing methods are capable of performing a smooth curve interpolation with constraints on derivative? What I need it to smooth my evenly spaced (with few missing points) data to get a smooth ...
4
votes
4answers
734 views

Finding inflection points in R from smoothed data

I have some data that I smooth using loess. I'd like to find the inflection points of the smoothed line. Is this possible? I'm sure someone has made a fancy method ...
0
votes
1answer
437 views

Formula that R “predict” function uses to calculate intervals exponential smoothing

I have been trying to figure out the exact formula that the R "predict" function uses to calculate prediction intervals for simple exponential smoothing. The prediction interval formula seems to vary ...
1
vote
2answers
166 views

How to explain smoothing functions in the logistic regression model

I fitted a logistic regression model with some smoothing functions, and the software made beautiful plots for them. Here is one example: My main concern is that there is no reference level in the ...
5
votes
2answers
330 views

How to smear a histogram

I was asked to perform a Gaussian smearing on the bins of an histogram. What does this mean?
5
votes
2answers
240 views

Variance of a smoothed AR(1) process

The query I have relates to calculating the variance of AR(1) processes that are smoothed with a simple moving average. So: In an AR(1) process of the form: $$ X_t=c+\varphi X_{t-1}+\varepsilon_t, ...
3
votes
2answers
93 views

What is smoothing in gaussian processes

I have been hearing this frequently that gaussian processes is a smoothing operation. I didn't get what they mean by that. Any clarifications guys?
1
vote
0answers
257 views

Holt Winters initialization using backcasting like SPSS

I'm trying to figure out how SPSS is initializing the multiplicative Holt Winters exponential smoothing model using backcasting. Thankfully IBM roughly described their way of doing so .. very roughly. ...
0
votes
1answer
424 views

Does the Holt-Winters algorithm for exponential smoothing in time series modelling require the normality assumption in residuals?

I'm working on a project to compare different approaches to time series modeling. In the model selection process, we perform residual analysis for the fitted models. For regression, we need to check ...
3
votes
0answers
170 views

Laplace smoothing parameter choice for Markov chain transitions

Let $Y_{t}$ be the state of the process at time $t$, ${\bf P}$ be the transition matrix then: $$ {\bf P}_{ij} = P(Y_{t} = j | Y_{t-1} = i) $$ Since this is a Markov chain, this probability depends ...
0
votes
2answers
124 views

Smoothing algorithm for saturating function

I have a noisy readout of a curve that is monotonically increasing or decreasing for a narrow range of points and then quickly saturates. I don't know exactly where the saturation point is, but from ...
1
vote
1answer
100 views

How to filter these extremely bad data points?

I'm looking at data for my company, and basically we have some periods over the last year where the data was not uploaded correctly. In this figure, "mu" is the value of interest, and duration is a ...
0
votes
1answer
102 views

How to quantify mis-specification bias and compare against smoothing bias for a non-parametric estimate of a randomly allocated continuous treatment?

Suppose that there is a data-generating process $$ y = \alpha + g(x) + \epsilon $$ which is to say that an outcome is some function of $x$. Suppose that $x$ is randomly assigned, so ...
0
votes
0answers
43 views

Curve smoothness - local adjacency

I am looking for statistical measures of curve smoothness. Time-series values {(1, 0.5), (4, -0.6), (200, 1.0)} where (time-unit, value) is linearly interpolated from one to the next. The smoothest ...