# Sample size when fitting categorical survey data

I have a model which fits data from repeated surveys: at time $t$, a number $n_t$ respondents is asked a question and can give one of $K$ answers ($k=1, ..., K$). This is repeated $T$ times ($t = 1, ..., T$). The model treats each survey response as independent (the respondents are randomly selected every year) and calculates the log likelihood of obtaining a given number of people choosing answer $k$ at time $t$ ($n_{kt}$) for $1 \le k \le K$ and $1 \le t \le T$, i.e. $\ell = \sum_{t=1}^T n_t \sum_{k=1}^K p_{kt} \log \hat{p}_{kt}$, where $p_{kt} := n_{kt} / n_t$ and $\hat{p}_{kt}$ is the probability predicted by the model.

My question is: what is my sample size if I have $N = \sum_{t=1}^T n_t$ surveys? Is it $N$ (number of surveys) or $T (K - 1) + T$ (number of independent quantities seen'' by the model)? It is important to me because I do not know what to use in Bayesian Information Criterion formula when comparing different model variants.

The number of independent points seen. Though I'm thinking of it as 'the number of predictions made'.

The mathematical reason for this is due to how the BIC arises. Specifically, the posterior is approximated near the MLE by a Gaussian. Then the marginal likelihood turns out to be approximated by

$$\ln p(D) \approx \ln p(D|\theta^*) - \frac{1}{2} \ln \left|\nabla^2 p(D|\theta)|_{\theta = \theta^*}\right|$$

where $D$ is the data, $\theta^*$ is the MLE, and the second term basically gives the 'curvature' near the MLE. That term is further approximated as

$$\nabla^2 p(D|\theta) = \prod_i \nabla^2p(D_i|\theta) \approx \prod_i\hat H = \hat H^N$$

for some fixed matrix $\hat H$. There's some more work to do to get the BIC, but this is where the $N$ in the formula comes from - from decomposing the probability of the data $D$ into a product of the probabilities of each datapoint.

• But the probability of data is the product of $N$ (number of surveys) probabilities, not of $TK$ probabilities. Commented Apr 6, 2015 at 17:31
• Sorry, I misunderstood your model! You're right. Commented Apr 6, 2015 at 17:39
• Is the real answer "whatever your log-likelihood scales linearly with"? Commented Apr 6, 2015 at 20:07
• I guess? I'd be more comfortable with 'whatever number goes up by 1 when you predict a new thing' Commented Apr 7, 2015 at 20:03
• And that comes back to the original question: what is a "new thing" in this context - a new response, or a new batch of responses for a new year? Commented Apr 9, 2015 at 7:57