2
$\begingroup$

I am working on regression model with time-varying coefficient, which looks like enter image description here

For simplicity, I assume that $\alpha_t$, $\beta_t$, and $\gamma_t$ follow random walk. I have already learned how to estimate the coefficient $\alpha_t$, $\beta_t$, and $\gamma_t$ using Kalman Filter and these three coefficients are time series. But how can I say the estimates are good? Is there any statistical test widely used that could help me to do so for Kalman Filter estimates? Like t test or F test in traditional regression maybe?

Add to the question: After I receive several response, I would like to clarify what "good" mean. What I want is, say in classic OLS, we have statistic tests to show whether the estimate is significant in statistic term. So, I am wondering is there a widely accepted paradigm that could provide such statistical significance for the estimates (time-series in our case).

$\endgroup$
  • 1
    $\begingroup$ t or F tests in traditional regression don't really tell you whether "the estimates are good". $\endgroup$ – Glen_b Jul 15 '15 at 3:34
  • 1
    $\begingroup$ "good" is an ambiguous term. Good as in 1) are my model assumptions correct? 2) is my model predicting better than a certain benchmark (i.e. where you assume $\alpha_t,\beta_t,\gamma_t$ to be constant, equal to zero or something else,3) do my parameters actually follow a random walk, and/or is there a better model, or 4) something else entirely. $\endgroup$ – Zachary Blumenfeld Jul 15 '15 at 6:30
  • $\begingroup$ 1) will lead toward diagnostic procedures, 2) would involve using an F-tests or something similar like a likelihood ratio. It would not be in-feasible to construct a joint F-test and/or t-tests, but it may be difficult given the random walk structure. A likelihood ratio test with the null hypothesis being some simpler nested model would be easier. $\endgroup$ – Zachary Blumenfeld Jul 15 '15 at 7:07
  • $\begingroup$ The Kalman filter gives you estimates of $\alpha_t$, $\beta_t$, etc. and also of their variances, so you can check whether your time-varying coefficients stay away from zero, if that is what you are after. $\endgroup$ – F. Tusell Jul 15 '15 at 7:30
  • $\begingroup$ @Tusell, how to test time-varying coefficients stay away from zero? you mean test whether the mean of the coefficients (given the estimated coefficients are time series) are different from 0 using t test? $\endgroup$ – Tracy Yang Jul 15 '15 at 12:44
2
$\begingroup$

So I guess I will answer your question in two parts. First, hypothesis or "significance" testing and second assessing model fit.

Hypothesis Testing

Your model can be written as $$Sales_t=\alpha_t A_t + \beta_t W_t + \gamma_tA_tW_t+v_t$$ $$ \alpha_t=\phi_{\alpha}\alpha_{t-1}+\varepsilon_{t,\alpha}$$ $$ \beta_t=\phi_{\beta}\beta_{t-1}+\varepsilon_{t,\beta}$$ $$ \gamma_t=\phi_{\gamma}\gamma_{t-1}+\varepsilon_{t,\gamma}$$ $$ (\varepsilon_{t,\alpha},\varepsilon_{t,\beta},\varepsilon_{t,\gamma})\sim \mathcal{N}(\mathbf{0},\Sigma)$$ $$v_t \sim N(0,\omega_t)$$ Where, because you are assuming a random walk, $\phi_{\alpha}=\phi_{\beta}=\phi_{\gamma}=1$.

In many Kalman filter models (in most I have seen), the parameters, $\phi_{\alpha},\phi_{\beta}, \phi_{\gamma}$ are estimated with the model (not fixed a priori) and hypothesis testing is usually carried out on those parameter estimates as opposed to the state estimates $(\hat \alpha_t,\hat \beta_t, \hat \gamma_t)$. Your case is different, you would like to test hypotheses of the form $$ H_0: \alpha_t=\beta_t=\gamma_t=0 \mathrm{\;for\;all\;}t $$

$$ H_a: \alpha_t \neq 0 \mathrm{\;or\;} \beta_t \neq 0 \mathrm{\;or\;} \gamma_t \neq 0 \mathrm{\;for\;some\;}t $$ or something similar where you are only interested in a smaller subset of state variables and time periods. This test is certainly very awkward and I would argue that it is not not logically consistent with the way the model was estimated.

The states are not parameter estimates in the same sense as ordinary least squared regression coefficients of which you are more familiar with testing. If this were true you would have no degrees of freedom left and would not be able to estimate your model. Rather, the states are more akin to error terms in the sense that they are incidental; they result as consequence of other parameters such as the $\phi$'s and the parametric structure of the model. As such, the "standard errors" associated with the states do not approach zero as the sample size increases, which is the primary logic behind frequentest hypothesis testing.

That said, just like we would know the distribution of our error terms in a OLS regression, $\epsilon_t \sim N(0,\sigma)$, we know the distribution of our state variables for each $t$. i.e. $\alpha_t \sim N(\hat \alpha_t, \sigma_{t,a})$ where $\hat \alpha_t$ and $\sigma_{t,a}$ are given by the Kalman filter. We also know the co-variance structure of the states for each $t$, i.e $cov(\alpha_t,\beta_t)$ in fact we assume the states are multivariate normal. So we can calculate probabilities such as $Pr(\gamma_t<0)$ for a single $t$ which can be useful.

You can technically use the above information to calculate z-test statistics for hypotheses of the form $H_0: \gamma_t=0\;\;H_a: \gamma_t \neq 0$. But what I am telling you is that that wouldn't make any sense since the states are incidental and will remain random variables until the end of time as $t \rightarrow \infty$

All the above aside, we can still test a hypothesis of the form $$ H_0: \alpha_t, \beta_t, \gamma_t \mathrm{\;all\;random\;walks} $$

$$ H_a: \alpha_t = c_1 \mathrm{\;and\;} \beta_t = c_2 \mathrm{\;and\;} \gamma_t=c_3 \mathrm{\;for\;all\;}t $$

where $c_1,c_2,$ and $c_3$ can be any set of constant numbers.

This is because $H_a$ is nested inside $H_0$, i.e. it is the special case of the random walk model where we assume initial states $(c_1,c_2,c_3)$ and $\Sigma=\mathbf{O}$. As such, you can use the likelihood ratio to test the above hypothesis. The likelihood ratio test statistic is $$ \lambda = -2\ln\bigg(\frac{L_0}{L_a} \bigg) $$ where $L_0$ and $L_a$ are the likelihoods of the null and alternative models respectively. $\lambda$ is asymptotically distributed chi-squared with degrees of freedom equal to the number of restrictions in the alternative. In this case, the degrees of freedom would be 6, the number of parameters necessary to estimate $\Sigma$. The p-value comes from plugging $\lambda$ directly into the chi-squared cdf.

Assessing Model Fit

As I stated in my comments this is a matter of taste. Model fit methods can be separated into two major groups, in-sample and out-of-sample.

The most popular in-sample methods are AIC and BIC. You can still used $R^2$ for the model if you like. The reason why a lot of professionals would not use $R^2$ in lieu of AIC or BIC is because $R^2$ does not consider the estimated variance for each prediction. In other words, the Kalman filter estimates $$ Sales_t \stackrel{iid}{\sim}N(\widehat{Sales_t},\Omega_t) $$

$R^2$ only takes into account the point estimates $\widehat{Sales_t}$. In actuality, the $\Omega_t$'s are very important as we care deeply about whether we are over or under estimating the variance/error in our predictions. AIC and BIC do take $\Omega_t$ into account as they are likelihood based.

In short, I may used $R^2$ in certain presentations and reports depending on my audience because more people know about it and it's easier to interpret. But for my own internal purposes of in-sample model evaluation I would use AIC or BIC.

There are many of out-of-sample fit statistics. A few popular ones are MAPE, MSPE and predictive likelihood. For reasoned mentioned above I prefer predictive likelihood, but again there are many variants of these techniques and alot of it comes down to your personal tastes.

$\endgroup$
  • $\begingroup$ Thanks so much for the detailed illustration. Could you please kindly provide some references with respect to AIC/BIC (better if it is used after apply Kalman Filter) and likelihood ratio for testing random walk ? $\endgroup$ – Tracy Yang Jul 20 '15 at 6:42

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.