I would like to perform model selection given a range of $k$ models $\mathcal{M}_1, \mathcal{M}_2, ..., \mathcal{M}_k$, each with some prior probability $f(\mathcal{M}_1), \dots, f(\mathcal{M}_k).$

I have a probability distribution for the random variable $X$ given each $\mathcal{M}_i$ and a vector of unknown parameters $\boldsymbol{\theta} \in \mathbb{R}^{m}$, where $m > 1$. This is to say that I know and can compute the probability mass function $f(x|\boldsymbol{\theta},\mathcal{M}_i)$. I do not have this in closed form but I can evaluate it.

The parameter vector $\boldsymbol{\theta}$ is independent of $\mathcal{M}_i$, that is to say $f( \boldsymbol{\theta}|\mathcal{M}_i) = f(\boldsymbol{\theta})$ for each model.

My data is in the form $\mathcal{D} = x \in \mathbb{N}$. Unfortunately, $\boldsymbol{\theta}$ is unidentifiable (via any MLE/MCMC method) from my single data point $x$.

My question is, how (if at possible) can I perform model selection (or any kind of hypothesis test) given my single data point $x$?

My idea was to use Monte Carlo Integration to find: $$f(\mathcal{D}|\mathcal{M}_k) = \int_\Theta f(\mathcal{D}|\boldsymbol{\theta},\mathcal{M}_k) f(\boldsymbol{\theta}) d \boldsymbol{\theta} \approx \frac{1}{N} \sum_{i=1}^{N} f(\mathcal{D}|\boldsymbol{\theta}^{(i)},\mathcal{M}_k), $$ where $\boldsymbol{\theta}^{(i)}$ for $i = 1, ... , N$ are iid samples from $f(\boldsymbol{\theta})$. Then I can compute $$f(\mathcal{M}_i|\mathcal{D}) = \frac{f(\mathcal{D}|\mathcal{M}_i) f(\mathcal{M}_i) }{\sum_j f(\mathcal{D}|\mathcal{M}_j) f(\mathcal{M}_j)},$$ for each model $i =1, \dots, k$ and then either pick the model which maximises this posterior likelihood or use this to compute Bayes Factors.

Is this a standard technique used? Or am I missing something? How would this relate to hypothesis testing? Thanks!

  • $\begingroup$ The (proper) prior removes the identifiability issue and makes the marginal likelihood well defined. From there it is indeed a correct Bayesian approach to the problem. With a single integer observation the support for one model versus another will proceed mostly from the corresponding priors. $\endgroup$ – Xi'an Mar 30 '19 at 10:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.