72

Pearson correlation is used to look at correlation between series ... but being time series the correlation is looked at across different lags -- the cross-correlation function. The cross-correlation is impacted by dependence within-series, so in many cases the within-series dependence should be removed first. So to use this correlation, rather than ...


17

You can do this using penalised splines with monotonicity constraints via the mono.con() and pcls() functions in the mgcv package. There's a little fiddling about to do because these functions aren't as user friendly as gam(), but the steps are shown below, based mostly on the example from ?pcls, modified to suit the sample data you gave: df <- data....


16

Exponential Smoothing is a classic technique used in noncausal time series forecasting. As long as you only use it in straightforward forecasting and don't use in-sample smoothed fits as an input to another data mining or statistical algorithm, Briggs' critique does not apply. (Accordingly, I am skeptical about using it "to produce smoothed data for ...


13

The recent scam package by Natalya Pya and based on the paper "Shape constrained additive models" by Pya & Wood (2015) can make part of the process mentioned in Gavin's excellent answer much easier. library(scam) con <- scam(y ~ s(x, k = k, bs = "mpd"), data = df) plot(con) There are a number of bs functions you can use - in the above I used mpd for ...


12

Rather than use smooth.spline() in the stats package, there is a function cobs() in the cobs package that allows you to do exactly the sort of thing you want. COBS stands for Constrained B-splines. Possible constraints include going through specific points, setting derivatives to specified values, monotonicity (increasing or decreasing), concavity, convexity,...


12

From the perspective of using R to find the inflections in the smoothed curve, you just need to find those places in the smoothed y values where the change in y switches sign. infl <- c(FALSE, diff(diff(out)>0)!=0) Then you can add points to the graph where these inflections occur. points(xl[infl ], out[infl ], col="blue") From the perspective of ...


11

So we're saying "Y is related to X", but we don't know the form. We want to estimate $E(Y|X=x)$. But at any observation, we have noise, and we need to be able to estimate it between observations. One (fairly naive) way we can approximate it is to assume that while the average changes, that it doesn't change too rapidly (i.e. that it's 'slowly varying' in a ...


11

There appears to be a difference in terminology (as so often is the case with a discipline used in so many areas), so I'm not 100% sure, but I think they're referring to kernel density estimation, with a Gaussian kernel, but performed on binned data. [Edit: if someone familiar with how the term "Gaussian smearing" is used in physics - and how it would apply ...


10

Sure. This is essentially the observation that the Dirichlet distribution is a conjugate prior for the multinomial distribution. This means they have the same functional form. The article mentions it, but I'll just emphasize that this follows from the multinomial sampling model. So, getting down to it... The observation is about the posterior, so let's ...


10

Great question. I believe that the answer to the question you ask - "is the penalized smoothing spline equivalent to running a ridge regression or lasso" - is yes. There are a number of sources out there that can provide commentary & perspective. One place that you may want to start with is this PDF link. As is noted in the notes: "Fitting a smoothing ...


9

mgcv uses a thin plate spline basis as the default basis for it's smooth terms. To be honest it likely makes little difference in many applications which of these you choose, though in some situations or with very large data set sizes, other basis types might be used to good effect. Thin plate splines tend to have better RMSE performance than the other three ...


8

Why your first thoughts led you astray: When you take the SVD of a matrix, $U$ and $V$ are unitary (orthogonal). So, while it is true that $SA = SU \Sigma V^{T}$, that is not (generally) the SVD of $SA$. Only if $S$ is unitary (which in the case of a smoothing matrix, it's not) would it be true that $U' = SU$. Is there any elegant, symbolic way of relating ...


8

There is no normality assumption in fitting an exponential smoothing model. Even if maximum likelihood estimation is used with a Gaussian likelihood, the estimates will still be good under almost all residual distributions. There is also no normality assumption when producing point forecasts from an exponential smoothing model. However, there is often a ...


8

To start with the title question, the density function is (for a continuous random variable) a function that describes the relative probability of the values taken by the variable (it is not probability! the word relative is important, since the probabilities are all zero); loosely it's proportional to the probability of being within a very small interval ...


8

Having read the book myself, this statement refers to a regularized model (edit: I guess all smoothing splines are like that), where every point is a knot, but you are regularizing by adding $\lambda \int g''(x)^2$ to the loss function, so you are punishing a "wiggly function", as expressed in a large absolute second derivative. I believe they also point ...


8

But, in my opinion wouldn't that be overfitting? No. Your equation explains it all. $$\underbrace{\sum_{i=1}^n(y_i-g(x_i))^2}_\text{residual squares}+\underbrace{\lambda\int g''(t)^2dt}_\text{roughness penalty}$$ The second part $\lambda\int g''(t)^2dt$ is often called a roughness penalty, and $\lambda$ - roughness coefficient. The idea here is that first ...


7

The "uniform" in uniform prior doesn't just mean that hits and misses are equally likely. It means that you assume that you have a probability measure on the rates $[0,1]$ and this measure is the uniform measure. For example, it means the chance the true rate is between $0.9$ and $1.0$ is $0.1$. There are other measures on $[0,1]$ which have mean value $1/2$....


7

There is a function in the TeachingDemos package called loess.demo that helps with the understanding of loess models, reading the documentation and running that function a few times may help with your understanding. 1) One way to get an R-square value is to square the correlation between the original y-values and the predicted y-values at the same point (...


7

If your entire data follows the pattern from your sample here you could try polynomial regression method with a second order (quadratic) polynomial function. See the example below: ggplot(allmortality,aes(x=ar_year)) + stat_smooth(aes(y = t1_all_estimate, group=1, colour="Obese" ), method=lm, formula = y ~ poly(x,2), level=0.95) + #tweak the ...


7

As you will see pointed out elsewhere that tf-idf is discussed, there is no universally agreed single formula for computing tf-idf or even (as in your question) idf. The purpose of the $+ 1$ is to accomplish one of two objectives: a) to avoid division by zero, as when a term appears in no documents, even though this would not happen in a strictly "bag of ...


7

Sounds like you just need to adjust the smoothing parameters (sometimes called bandwidth) to your liking. Either of these methods should be able to be tuned appropriately. Moving average can be adjusted to account for more points (increasing smoothness). Similarly, lowess smoothers have a smoothing parameter to increase or decrease smoothness.


7

Here is an example of a 2-dimensional Kalman filter that may be useful to you. It is in Python. The state vector is consists of four variables: position in the x0-direction, position in the x1-direction, velocity in the x0-direction, and velocity in the x1-direction. See the commented line "x: initial state 4-tuple of location and velocity: (x0, x1, ...


7

Smoothing splines have all the knots (knots at each point), but then regularizes (shrinks the coefficients/smooths the fit) by adding a roughness penalty term (integrated squared second derivative times a smoothing parameter/tuning parameter) to the least squares criterion. In one way, it's sort of analogous to a kind of "weighted" ridge regression, if you'...


7

A simple, robust method to handle such noise is to compute medians. A rolling median over a short window will detect all but the smallest jumps, while medians of the response within intervals between detected jumps will robustly estimate their levels. (You may replace this latter estimate by any robust estimate that is unaffected by the outliers.) You ...


6

Let us rewrite $x_t, x_{t-1}, \dots, x_{t-K+1}$ in terms of $x_{t-K}$ $$x_t=c\left(1+\varphi+\dots+\varphi^{K-1}\right)+\varepsilon_t+\varphi\varepsilon_{t-1}+\dots+\varphi^{K-1}\varepsilon_{t-K+1}+\varphi^Kx_{t-K}$$ $$x_{t-1}=c\left(1+\varphi+\dots+\varphi^{K-2}\right)+\varepsilon_{t-1}+\varphi\varepsilon_{t-2}+\dots+\varphi^{K-2}\varepsilon_{t-K+1}+\...


6

The requester is probably referring to a kernel density smoother as Glen_b notes, but "smearing" is evocative of the shadowgram featured in the book Visual Statistics. To address the issue of choosing the right bin or kernel width, the shadowgram is a overlay of many different width choices.


6

I don't know what estimator you are considering but what you propose has certainly been done before. Horowitz (1998) investigates whether the bootstrap can be used for asymptotic refinements of median regression. He faces the same problem as you given that the objective function has an embedded indicator $$\widehat{\beta} = min_{\beta} \frac{1}{n} \sum^{n}_{...


6

Dan Jurafsky has published a chapter on N-Gram models which talks a bit about this problem: At the termination of the recursion, unigrams are interpolated with the uniform distribution: $ \begin{align} P_{KN}(w) = \frac{\max(c_{KN}(w)-d,0)}{\sum_{w'}c_{KN}(w')}+\lambda(\epsilon)\frac{1}{|V|} \end{align} $ If we want to include an unknown word &...


6

To complete the answer of Glen_b and his/her example on random walks, if you really want to use Pearson correlation on this kind of time series $(S_t)_{1 \leq t \leq T}$, you should first differentiate them, then work out the correlation coefficient on the increments ($X_t = S_t - S_{t-1}$) which are (in the case of random walks) independent and identically ...


6

[In the later discussion of LOESS here I attempt to describe LOWESS and its implementation in the R function lowess as well as outline some of the modifications made for the function loess (though some details that don't seem to be directly relevant to your questions are omitted).] In particular: with smoothing splines, how do we choose the number and ...


Only top voted, non community-wiki answers of a minimum length are eligible