Sample size when fitting categorical survey data

Question

I have a model which fits data from repeated surveys: at time $t$, a number $n_t$ respondents is asked a question and can give one of $K$ answers ($k=1, ..., K$). This is repeated $T$ times ($t = 1, ..., T$). The model treats each survey response as independent (the respondents are randomly selected every year) and calculates the log likelihood of obtaining a given number of people choosing answer $k$ at time $t$ ($n_{kt}$) for $1 \le k \le K$ and $1 \le t \le T$, i.e. $\ell = \sum_{t=1}^T n_t \sum_{k=1}^K p_{kt} \log \hat{p}_{kt}$, where $p_{kt} := n_{kt} / n_t$ and $\hat{p}_{kt}$ is the probability predicted by the model.

My question is: what is my sample size if I have $N = \sum_{t=1}^T n_t$ surveys? Is it $N$ (number of surveys) or $T (K - 1) + T$ (number of independent quantities ``seen'' by the model)? It is important to me because I do not know what to use in Bayesian Information Criterion formula when comparing different model variants.

Answer 1

The number of independent points seen. Though I'm thinking of it as 'the number of predictions made'.

The mathematical reason for this is due to how the BIC arises. Specifically, the posterior is approximated near the MLE by a Gaussian. Then the marginal likelihood turns out to be approximated by

$$\ln p(D) \approx \ln p(D|\theta^*) - \frac{1}{2} \ln \left|\nabla^2 p(D|\theta)|_{\theta = \theta^*}\right|$$

where $D$ is the data, $\theta^*$ is the MLE, and the second term basically gives the 'curvature' near the MLE. That term is further approximated as

$$\nabla^2 p(D|\theta) = \prod_i \nabla^2p(D_i|\theta) \approx \prod_i\hat H = \hat H^N$$

for some fixed matrix $\hat H$. There's some more work to do to get the BIC, but this is where the $N$ in the formula comes from - from decomposing the probability of the data $D$ into a product of the probabilities of each datapoint.