Covariance of the empirical probability mass function Suppose a discrete random variable $Y$ takes $k$ levels of different values $y_1,y_2,...,y_k$. Let $P(Y=y_k):=p_k$.
Suppose we have $n$ i.i.d. samples of $Y$, my question is:

*

*How can we compute the ePMF? What's the MLE estimator $\hat{\mathbf{p}}$ of the distribution parameter $\mathbf{p}:=[p_1,p_2,...,p_k]$?


*What's the co-variance matrix of $\hat{\mathbf{p}}$? How to get an consistent estimator $\hat{\Sigma}$ of the co-variance matrix $\Sigma$ in the asymptotic normality  $n^{\frac{1}{2}}(\hat{\mathbf{p}}-\mathbf{p})\to N(0,\Sigma)$?
 A: After reading some references, I think I have figured out my questions above.
Parameters of the discrete distribution can be estimated by MLE of multinomial distribution.
Let the random variable $X_i$ denote the number of $Y$ takes value $i$ in the $n$ samples ($\sum_i^k X_i =n$). Then, the random vector $\mathbf{X}:=[X_1,X_2,...,X_k]$ follows a multinomial distribution with parameters $n, \mathbf{p}$.
Likelihood function for the multinomial distribution is $L(p_1,...,p_k,n|x_1,...,x_k)=C_{n}^{x_1,...,x_k}\prod_i^k p_i^{x_i}$. So, some calculus gives the MLE estimator of $p_i$ is $\hat{p_i}=\frac{X_i}{n}$, and thus $\hat{\mathbf{p}}=\frac{\mathbf{X}}{n}$. [Thm. 14.5].
Co-variance of the estimator
So, we can either directed compute $\mathrm{Cov}(\mathbf{p})$ by:

*

*$\mathrm{Cov}(X_i,X_j)=-np_ip_j$, $\mathrm{Var}(X_i)=np_i(1-p_i)$. [Thm 5.9.2 in Degroot and Schervish.]

Or by computing the fisher infomation matrix. Both of them give the same result:
$\Sigma=\begin{bmatrix}
p_1(1-p_1) &  -p_1p_2& ... & -p_1p_k\\ 
-p_1p_2 &  p_2(1-p_2))& ... & -p_2p_k\\ 
 ...&  ...& ... & ...\\ 
 -p_1p_k&  -p_2p_k& ... & p_k(1-p_k)
\end{bmatrix}$.
So, a consistent estimator $\hat{\Sigma}$ can be obtained by replacing the $p_i$ with $\hat{p_i}$.
