# Intuition behind Bayesian statistic and MCMC when applied to time series models

Even though I've studied Probability and Statistics, my knowledge of Bayes stops at the famous formula. So, when it comes to modern Bayesian computational methods, I'm just able to understand answers to questions like this one, but I'm still missing the most important point: playing around with some Bayesian tools, I've noticed that models parameters are gradually updated as new information is disclosed.

Let me explain better what I would like to understand: when we're dealing with time series analysis and prediction, we often have a huge sample but we also have the task to select a proper window width to estimate our model parameters; this is because we are burdened by the usual trade-off between long and short term memory, for which:

• using a large sample would probably mean a very stable estimate but unable to adapt quickly to structural changes in the time series;
• using a small sample would probably mean an unstable estimate but able to adapt to structural changes.

The simplest example of this trade-off is a moving average of the time series, but consider more convoluted examples like a function which estimates the parameters of a simple $$AR(1)$$ model and the application of such function through a rolling window: the autocorrelation parameter's time series, $$\rho$$, is going to have lower variance in the larger window width case than in the smaller one.

(In the figure below I've produced an example by estimating $$\rho$$ from a rolling window $$AR(1)$$ model on a time series to show that the values obtained can be very different according to the window width).

After some trials with MCMC tools and Bayesian statistic applied to models parameters, I've seen things like this:

If the sample is a time series, my intuition suggests me that the density of beta in the figure above comes from a parameter that varies over time. This "variation over time" should simply be a time series of beta that is gradually updated while the prior distribution is plugged into the Bayesian tool. Then parameters distribution is sampled from MCMC instead of being represented by a single value coming out from the solution to some minimization problem (I'm thinking about OLS). From this point of view, Kalman filter is a Bayesian tool when used to solve dynamic linear models, because of the mechanism that updates the values of the parameters as soon as new information is disclosed.

My questions:

1. do Bayesian statistics tools behave like a rolling/expanding window parameter estimation when applied to a time series?
2. Do Bayesian statistics behave like the Kalman filter when applied to a time series?
3. If 1 and/or 2 are at least partially true, can Bayesian statistics and MCMC be considered tools that solve the aforementioned trade-off of time series analysis window width?
4. Could you please provide a simple example in R for this problem: show the time-varying parameters that describe the mean and the standard deviation of your favourite time series as Bayesian statistics would yield them by using MCMC. You can use any package and any time series you want.

You seem to be confusing some things, so let me clarify. First of all, Bayesian models do not have any special connection to time-series. Same as for non-Bayesian models, using Bayesian framework you can define possibly infinite number of different statistical models.

Sure, you can first estimate the model for $$n$$ time-series points, and then update it for the $$n+1$$'th point, but the results would be the same as if you estimated the model on $$n+1$$ datapoints all at once (this is so-called Bayesian updating). Of course you can compare the estimate for the parameter from $$n$$ points, to the one calculated using $$n+1$$ points, but again, you can do this for non-Bayesian models as well. So there is no special "autoregressive" property. Moreover, from technical reasons, Bayesian updating is not simple to be done for complicated models that require MCMC sampling, so in practice you would usually just calculate the model on $$n+1$$ datapoints all at once, rather then updating it sequentially.

As about "variability over time" of $$\beta$$, let me use an example. Imagine a trivial model

$$x_{t+1} = \mathcal{N}(x_t\beta, \sigma)$$

with known $$\sigma$$. In non-Bayesian setting, you could estimate the $$\beta$$ parameter by maximizing the likelihood function. The only major difference in Bayesian setting would be that you would assume a prior distribution for $$\beta$$, and maximize (or sample from) posterior $$\propto$$ likelihood $$\times$$ prior. What follows, by assuming $$\beta$$ to be a random variable, setting prior for it, and plugging it to Bayes theorem, you would end up with a posterior distribution. Posterior distribution helps you to quantify how uncertain are you about the $$\beta$$ parameter, it does not have any special connection to time series or changes over time. In the above model, we do not assume $$\beta$$ to vary over time, so the model does not learn anything specific about such variability. Updating your model sequentially and looking at the changes of the estimates over time would only show you how your estimates become more precise with growing sample size.

So Bayesian models do not need to behave like rolling/expanding window parameter estimation. You can implement rolling average model in Bayesian setting, but you can also implement a number of other time series models. You can plug in all the data at once and nothing would "roll". On another hand, as described earlier, you could estimate the non-Bayesian models for $$n$$, $$n+1$$, $$n+2$$ etc. points as well.

Bayesian models also do not solve any problems for time-series. In some cases, with Bayesian model, if you already have model estimated on $$n$$ points, it could be easier to "update" the model for the new $$n+1$$'th point, but this is not always straightforward.