9
$\begingroup$

I have a sample of $(Y_{i,t},X_{1,i,t},X_{2,i,t})$ for $i=1,\dots,N$ and $t=1,\dots,T$. I want to figure out which data generating process (DGP) it comes from, DGP1 or DGP2.

DGP1: $$ Y_{i,t}=\lambda_0+\lambda_1\beta_{1,i}+\varepsilon_{i,t} \tag{1} $$ with $\lambda_0=0$ and $\varepsilon_{\cdot,t} \stackrel{\text{i.i.d.}}{\sim} N(\mathbf{0},\Sigma)$. $\beta_{1,i}$ is a latent variable that is the slope coefficient from $$ Y_{i,t}=\beta_0+\beta_{1,i} X_{1,i,t}+u_{i,t}. \tag{i} $$

DGP2: $$ Y_{i,t}=\tilde\lambda_0+\tilde\lambda_1\tilde\beta_{1,i}+\tilde\lambda_2\tilde\beta_{2,i}+\tilde\varepsilon_{i,t} \tag{2} $$ with $\tilde\lambda_0=0$ and $\tilde\varepsilon_{\cdot,t} \stackrel{\text{i.i.d.}}{\sim} N(\mathbf{0},\tilde\Sigma)$. $\tilde\beta_{1,i}$ and $\tilde\beta_{2,i}$ are latent variables that are the slope coefficients from $$ Y_{i,t}=\tilde\beta_0+\tilde\beta_{1,i} X_{1,i,t}+\tilde\beta_{2,i} X_{2,i,t}+\tilde u_{i,t}. \tag{ii} $$

Question: How could I construct a test of $H_0$: DGP1 is the truth against $H_1$: DGP2 is the truth? Ideally, the answer would also cover the case where $X_{1,i,t}$ and $X_{2,i,t}$ are vector-valued random variables. I am also not very comfortable assuming the errors to be multivariate Gaussian, but hopefully we can get rid of the assumption asymptotically.

The background is multifactor asset pricing models and the more specific, finance-oriented question I have is here. The setup is not identical but similar. There, we could perhaps fit $(1)$ and $(2)$ (without the restriction $\lambda_0=0$) and compare the statistic $\boldsymbol{\hat\lambda}_0^{\top} \text{Cov}(\boldsymbol{\hat\lambda}_0,\boldsymbol{\hat\lambda}_0^{\top})^{-1}\boldsymbol{\hat\lambda}_0\sim\chi^2_{?}$ with $\boldsymbol{\hat{\tilde{\lambda}}}_0^{\top} \text{Cov}(\boldsymbol{\hat{\tilde{\lambda}}}_0,\boldsymbol{\hat{\tilde{\lambda}}}_0^{\top})^{-1}\boldsymbol{\hat{\tilde{\lambda}}}_0\sim\chi^2_{??}$, and perhaps that would $\sim\chi^2_{???}$?

Update: For the bounty, I expect a detailed answer that would be straightforward to implement. Thank you!

$\endgroup$
2
  • $\begingroup$ Just bringing in different perspectives: Are you bound to testing? Can't you check for predictive performance, e.g. with leave one out Cross-Validation? Then you could do forward or backward selection of features. Also sometimes dropping variables may not be beneficial for generalization (e.g. in the case of omitted variable bias). Maybe this gives you another perspective or you may find this comment off-topic $\endgroup$
    – Ggjj11
    Commented Dec 3, 2023 at 21:01
  • 2
    $\begingroup$ @Ggjj11, your suggestions are logical, but I am bound to testing. $\endgroup$ Commented Dec 4, 2023 at 6:18

1 Answer 1

0
$\begingroup$

Benavoli et al (2018) argue for a Bayesian approach to model comparisons, based on performance on cross-validation folds. While the article emphasizes the advantages of Bayesian estimation of null-hypothesis significance testing (refering to the Kruschke (2013) paper Bayesian estimation supersedes the t test), classical approaches that lead to p-values are discussed that could be implemented to placate a reviewer who wants a p-value. Again, such approaches are based on performance in cross-validation folds.

REFERENCES

Benavoli, Alessio, et al. "Time for a change: a tutorial for comparing multiple classifiers through Bayesian analysis." The Journal of Machine Learning Research 18.1 (2017): 2653-2688.

Kruschke, John K. "Bayesian estimation supersedes the t test." Journal of Experimental Psychology: General 142.2 (2013): 573.

$\endgroup$
1
  • $\begingroup$ This is interesting but sounds more complicated than it needs to be. Time series cross validation can be a pain with small sample size (which is my case). The approach also reminds me of Diebold's criticism of the abuse of the Diebold-Mariano test. Diebold (2015) makes a point that in-sample analysis is preferred to out-of-sample one when testing models against each other. I think John Cochrane (referenced in the link in my post) envisions a fairly straightforward solution, as I try to hint at in the "background" paragraph. $\endgroup$ Commented Jan 3 at 14:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.