5
$\begingroup$

I have a model which fits data from repeated surveys: at time $t$, a number $n_t$ respondents is asked a question and can give one of $K$ answers ($k=1, ..., K$). This is repeated $T$ times ($t = 1, ..., T$). The model treats each survey response as independent (the respondents are randomly selected every year) and calculates the log likelihood of obtaining a given number of people choosing answer $k$ at time $t$ ($n_{kt}$) for $1 \le k \le K$ and $1 \le t \le T$, i.e. $\ell = \sum_{t=1}^T n_t \sum_{k=1}^K p_{kt} \log \hat{p}_{kt}$, where $p_{kt} := n_{kt} / n_t$ and $\hat{p}_{kt}$ is the probability predicted by the model.

My question is: what is my sample size if I have $N = \sum_{t=1}^T n_t$ surveys? Is it $N$ (number of surveys) or $T (K - 1) + T$ (number of independent quantities ``seen'' by the model)? It is important to me because I do not know what to use in Bayesian Information Criterion formula when comparing different model variants.

$\endgroup$

1 Answer 1

1
$\begingroup$

The number of independent points seen. Though I'm thinking of it as 'the number of predictions made'.

The mathematical reason for this is due to how the BIC arises. Specifically, the posterior is approximated near the MLE by a Gaussian. Then the marginal likelihood turns out to be approximated by

$$\ln p(D) \approx \ln p(D|\theta^*) - \frac{1}{2} \ln \left|\nabla^2 p(D|\theta)|_{\theta = \theta^*}\right|$$

where $D$ is the data, $\theta^*$ is the MLE, and the second term basically gives the 'curvature' near the MLE. That term is further approximated as

$$\nabla^2 p(D|\theta) = \prod_i \nabla^2p(D_i|\theta) \approx \prod_i\hat H = \hat H^N$$

for some fixed matrix $\hat H$. There's some more work to do to get the BIC, but this is where the $N$ in the formula comes from - from decomposing the probability of the data $D$ into a product of the probabilities of each datapoint.

$\endgroup$
5
  • $\begingroup$ But the probability of data is the product of $N$ (number of surveys) probabilities, not of $TK$ probabilities. $\endgroup$
    – quant_dev
    Apr 6, 2015 at 17:31
  • $\begingroup$ Sorry, I misunderstood your model! You're right. $\endgroup$
    – Andy Jones
    Apr 6, 2015 at 17:39
  • 1
    $\begingroup$ Is the real answer "whatever your log-likelihood scales linearly with"? $\endgroup$
    – quant_dev
    Apr 6, 2015 at 20:07
  • $\begingroup$ I guess? I'd be more comfortable with 'whatever number goes up by 1 when you predict a new thing' $\endgroup$
    – Andy Jones
    Apr 7, 2015 at 20:03
  • 1
    $\begingroup$ And that comes back to the original question: what is a "new thing" in this context - a new response, or a new batch of responses for a new year? $\endgroup$
    – quant_dev
    Apr 9, 2015 at 7:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.