# Moving estimators for nonstationary time series, like loglikelihood: l_T=sum_{t<T} a^{t-T} ln(rho(x_t))?

While in standard ("static") e.g. ML estimation we assume that all values are from a distribution of the same parameters, in practice we often have nonstationary time series: in which these parameters can evolve in time.

It is usually considered by using sophisticated models like GARCH conditioning sigma with recent errors and sigmas, or Kalman filters - they assume some arbitrary hidden mechanism.

I have recently worked on a simpler and more agnostic way: use moving estimator, like loglikelihood with exponentially weakening weights of recent values: $$\theta_T=\arg\max_\theta l_T\qquad \textrm{for}\qquad l_T= \sum_{t intended to estimate local parameters, separately on each position. We don't assume any hidden mechanism, only shift the estimator.

For example it turns out that EPD (exponential power distribution) family $$\rho(x) \propto \exp(-|x|^\kappa)$$, which covers Gaussian ($$\kappa=2$$) and Laplace ($$\kappa=1$$) distributions, can have cheaply made such moving estimator (plots below), getting much better loglikelihood for daily log-returns of Dow Jones companies (100 years DJIA, 10 years individual), even exceeding GARCH: https://arxiv.org/pdf/2003.02149 - just using the $$(\sigma_{T+1})^\kappa=\eta (\sigma_{T})^\kappa+(1-\eta)|x-\mu|^\kappa$$ formula: replacing estimator as average with moving estimator as exponential moving average:

I have also MSE moving estimator for adaptive least squares linear regression: page 4 of https://arxiv.org/pdf/1906.03238 - can be used to get adaptive AR without Kalman filter, also analogous approach for adaptive estimation of joint distribution with polynomials: https://arxiv.org/pdf/1807.04119

Are such moving estimators considered in literature?

What applications they might be useful for?

• Hi: that's an interesting idea. Rather than considering parameteric distributions, it might be interesting to see how it performs compared ro simple exponential smoothing itself. In other words, take some dgp ( ses assumes no trend and no seasonality. it's just estimating the mean ) and then A) estimate the $\alpha$ parameter using exponential smoothing and B) estimate the parameter by exponential smoothing the likelihood and see if one is better than the other. neat idea. Note that just because I've never seen what you're doing before DOES NOT MEAN that it hasn't been considered before. – mlofton Mar 19 '20 at 11:53
• Thanks, looks very natural: don't assume model just shift estimator, and has great performance, but it seems nobody have heard of such approaches, I have tried asking e.g. here: redd.it/fkl2ee I don't like DGP as the best model for them is the generating one, but can test on some real data, collaborate. – Jarek Duda Mar 19 '20 at 13:13
• Hi: I don't have time to collaborate but the offer is appreciated. I think you should test on a DGP whose mean is changing a little and nothing else. Say like a random walk plus noise. For this model, it is well known that an ARIMA(0,1,1) is optimal but an ARIMA(0,1,1) is equivalent to simple exponential smoothing (SES ). So, if your exponential likelihood idea can beat that case, then you know that you've got something ! I doubt it can ( because of the optimality of exponential smoothing for a random walk + noise ) but it would be useful to see how close it gets to the SES estimates. – mlofton Mar 20 '20 at 5:12
• Such moving estimator is optimal in ML/MSE sense - shifting window with weakening weights. Testing on generated data makes sense for guessed arbitrary models like GARCH, but this one is agnostic. For a given generator we can find the optimal estimator, but it is not interesting or valuable. A method is valuable if it works well on the real data - I prefer to focus on, and with these approaches I beat standard methods in objective evaluation criteria, like loglikelihood in cross validation for static models, or using only previous values for adaptive models. – Jarek Duda Mar 20 '20 at 7:48
• Sounds good. If I had more time, I would look at what you're doing more closely. Unfortunately, I don't. To me, the notion of exponentially smoothing the likelihood is interesting but I wonder if there's not some equivalence between smoothing the likelihood and smoothing the parameter estimate itself ? I guess that's more of a mathematical questoon and math is not my thing. But, intuitively, if you're smoothing the likelihood and the likelihood is a function of the parameter estimate , then one would think that smoothing the estimate would be equivalent in some way ? Just food for thought. – mlofton Mar 21 '20 at 8:34