Difference between empirical and marginal reliability of an IRT model I am using the mirt library in R to fit an instrument (binary responses) comprising two dimensions. In the mirt documentation are mentioned two types of reliability. I would like to ask what is the difference between empirical (marginal) reliability ("empirical_rxx") and marginal reliability ("marginal_rxx"). How are these coefficients are interpreted, in terms of what they represent? And finally is this an appropriate way to report reliability within the IRT framework?
 A: The functions mentioned provide two approaches to computing the so-called 'marginal' reliability of a test; one based on assuming the exact probability density function of the latent trait distribution, $g(\theta)$, while the other approach obtains an estimate, $\hat{g}(\theta)$, through the use of the $N$ post-hoc point estimates from the collection of all $\theta_i$ terms. Both attempt to quantify the overall reliability attained by a given test, but do so using different assumptions. 
Starting with assuming $g(\theta)$, there are many such expressions which have appeared for estimating marginal reliability, one of which is of the form
$$\rho_{xx^\prime} = \int \frac{\mathcal{I}(\theta)}{\mathcal{I}(\theta) + 1}g(\theta),$$where the $+1$ appears by assuming the latent traits are scaled to have a variance of 1 (typically by product of marginal ML estimation). The estimate of this quantity is therefore
$$\hat{\rho}_{xx^\prime} = \int \frac{\mathcal{\hat{I}}(\theta)}{\mathcal{\hat{I}}(\theta) + 1}g(\theta),$$ where the $\mathcal{\hat{I}}(\theta)$ estimates of the Fisher information are formed from the estimated item parameters (hence, are subject to sampling error). Notice again that $g(\theta)$ is assumed to be known (e.g., may follow a Gaussian p.d.f.). If $g(\theta)$ is correctly specified, and the IRT parameters are unbiased, then the resulting estimate will be a good estimator for $\rho_{xx^\prime}$. In the marginal_rxx() function in mirt $g(\theta)$ is assumed to be a standard normal distribution (assuming the model was itself estimated with $\theta \sim \mathcal{N}(0,1)$), which may or may not be reasonable for the population at hand. This can be changed if the front-end user has a better idea of the shape of $g(\theta)$.
The second approach to estimating $\rho_{xx^\prime}$ (i.e., the empirical_rxx() function) obtains the estimate $g(\theta)$ after collecting all $\hat{\theta}_i$ for $i=1,2,\ldots,N$ in the sample. Of course, this means that one has to provide an estimator for $\hat{\theta}$, which take on many different flavors (ML, MAP, EAP, EAP for sum scores, WLE, etc), but however the values are estimated the equations follow the form
$$\hat{\rho}_{xx^\prime} = \frac{\widehat{VAR}(\hat{\theta})}{\widehat{VAR}(\hat{\theta}) + \widehat{SE}(\hat{\theta})^2},$$
where $\widehat{SE}(\hat{\theta})^2$ is obtained by averaging across the $N$ standard errors of the respective $\hat{\theta}_i$ terms and squaring. This has the usual "true-score / (true-score + error)" form. 
For example, if the estimator chosen to obtain the $\hat{\theta}$ terms is ML then no prior distribution of $g(\theta)$ is assumed post model estimation, and the distribution of $\theta$ becomes a type of "empirical" estimate of the underlying $\theta$ distribution. This has good and bad properties since the aggregate results of point estimates are known to be sub-optimal, but at the same time this approach seems slightly more realistic than assuming the $g(\theta)$ is known a priori. As well, this approach also depends on which estimator is selected to obtain $\hat{\theta}$ predictions, which can slightly alter the resulting estimate (see here for a brief comment on this phenomenon), and again how accurately the estimated item parameters are in the same way as the marginal reliability estimator above. 
Which is better, marginal or empirical, is a matter of context, but suffice it to say they both are trying to compute the marginal reliability of the test, just in different ways.
