Conceptual explanation of Maximum Likelihood Estimation

Question

Given a generic time-series $$y_{t+1}= \alpha y_{t} + \Sigma_{t+1}^{1/2}\varepsilon_{t+1} \quad \text{with} \quad \varepsilon_{t+1}=N(0, I)$$

where $\Sigma_{t+1}^{1/2}$ indicates the conditional covariance matrix as usual.

From the numerical point of view when we estimate the conditional mean $m_{t+1}= \alpha y_{t}$ and conditional covariance $\Sigma_{t+1}$, we find those simultaneous estimates that make the log-likelihood maximum (subject to $\Sigma_{t+1}$ beign positive definite), i.e. we maximize

\begin{aligned} \ell(\mathbf{ \mu_{(t)}, \Sigma_{(t)} \mid x_{(t)} }) & = \sum_{t=1}^m \text{C} - \frac{m}{2} \log |\Sigma_{(t)}| - \frac{1}{2} \mathbf{(x_{(t)} - \mu_{(t)})^T \Sigma^{-1}_{(t)} (x_{(t)} - \mu_{(t)}) } \end{aligned}

My question is: why is it so conceptually? Especially when it comes to maximizing it with respect to the parameters in $\Sigma_{t}$?

Q1) is it correct to say that, when we are maximizing that log-likelihood, we are conceptually trying to find those $\mu_t$ and $\Sigma_t$ such that the standardized innovations of the series $\Sigma_t^{1/2}(y_t-\mu_t)$ will be distributed as close as possible as $N(0,I)$ like the innovations $\epsilon_{t}$? I.e. we are trying to find those parameters that will make the conditional distribution of the standardized returns as close as possible to the unconditional distribution of the innovation $\varepsilon_t=N(0,I)$.

Q2) So when we maximize the log-likelihood with respect to $\Sigma_t$ we are tring to find those parameters in $\Sigma_{t}$ such that the unconditional covariance matrix of $\Sigma_t^{1/2}(y_t-\mu_t)$ estimated in the sample will be as close as possible to the unit matrix $I$?

Q3) Will the log-likelihood be at its maximum with respect to $\Sigma_{t}$ given $\mu_t$ when the estimated $\Sigma_{t}$ are such that $\sum_{t=1}^m \Sigma_t=\sum_{t=1}^m ( \Sigma_t^{1/2}(y_t-\mu_t) ) (\Sigma_t^{1/2}(y_t-\mu_t))^{T}=Im$ or very close to $Im$?

Answer 1

I just want to complement your last answer like this (I will use your notation).

If you assume that the true process is $y_{t}=A_{t}^{1/2}\epsilon_{t}$, and want to show that $A$ can be retrieved as a ML estimate, you can consider the following simplified approach, that provides some intuitive understanding of the problem.

If you assume that the true process is $y_{t}=A_{t}^{1/2}\epsilon_{t}$, then, instead of writing the likelihood for y $L(y|X)$, let's consider the standardized residuals of $y$ that are standardized with the Cholesky square root of the positive definite unknown variance to be estimated $\Sigma_{t}$. Denote such standardized residuals with the unknown variance as $y_{t}^{*}=\Sigma_{-1/2}y$. Notice that, in doing so, we are de facto assuming that the unknown covariance matrix admits a Cholesky decomposition which is also pd and non-singular, therefore we are assuming that the true variance matrix to be estimated is pd. Now let's make some thoughts on the -log likelihood for the st residuals $-\log p(y^{*}|X)$

$$ -\log p(y^{*}|X) = \sum_{t=1}^{N} \frac{1}{2}y^{*T}_{t}V_{t}^{*-1}y^{*}_{t} + \frac{1}{2}\log|V^{*}_{t}|+\frac{n}{2}\log2\pi$$

Notice that the unknown of interest is $\Sigma_{t}$, which is stored into $y^{*}_{t}=\Sigma_{t}^{-1/2}y_{t}$, as we are assuming to represent the standardized residuals scaled by the unknown true variance. Then, since we are scaling by the true (albeit still unknown variance), we can say that the value of the conditional variance of such standardized residuals must be constant and equal to the unit matrix $I$ for each t ($V_{t}^{*}=I$ for each t). So we re-write the -log lik with constant $V^{*}$ as

$$ -\log p(y^{*}|X) = \sum_{t=1}^{N} \frac{1}{2}y^{*T}_{t}V^{*-1}y^{*}_{t} + \frac{1}{2}\log|V^{*}|+\frac{n}{2}\log2\pi$$

and take the first-order condition for maximization by setting to 0 the gradient with respect to the constant V (remember that the gradient of the sum is the sum of gradients):

$$ \sum_{t=1}^{N} V^{*} - y^{*}_{t}y^{*T}_{t} =0 \rightarrow NV^{*} = \sum_{t=1}^{N} y^{*}_{t}y^{*T}_{t} \rightarrow NV^{*} = \sum_{t=1}^{N} y^{*}_{t}y^{*T}_{t} $$

Since we are assuming to standardize the $y$ with the true (but still unknown) variance of the process, then we can also assume to know that the true value of $V^{*}$ is the matrix $I$. And we also know that, in light of the asymptotic consistency of MLE, if the sample size N tends to infinite (in practice it is large enough), the MLE sample estimate of the parameter $V^{*}$ converges in probability to the true value $I$. Therefore, as long as the sample size is large enough, we can expect that the optimal value that solves the first order condition is $V^{*}_{t}=I$ for each t (solving the condition for $V^{*}$ would yield the true value of the parameters for large enough samples). Since the first order condition holds at the optimum, then we can set $V^{*}=I$ and restate the first-order condition to make it a function of the unknown $\Sigma_{t}$, and finally solve for the unknowns: set $V^{*}=I$ and make the first order condition a function of the unknown $\Sigma_{t}$ by substituting $y^{*}_{t}=\Sigma_{t}^{-1/2}y_{t}=\Sigma_{t}^{-1/2}A_{t}^{1/2}e_{t}$. For $V^{*}=I$. You get the following restated first order condition:

$$ \sum_{t=1}^{N} V^{*} - y^{*}_{t}y^{*T}_{t} =0 \rightarrow NI = \sum_{t=1}^{N} y^{*}_{t}y^{*T}_{t} \rightarrow NI = \sum_{t=1}^{N} \Sigma_{t}^{-1/2}A_{t}^{1/2}e_{t} e_{t}^{T}A_{t}^{1/2T}\Sigma_{t}^{-1/2T}$$

which, for N large enough, will hold for $\Sigma_{t}^{-1/2}A_{t}^{1/2}=I \rightarrow \Sigma_{t}=A_{t}$. Indeed, for large enough N, $\sum_{t=1}^{N} e_{t} e_{t}^{T}$ converges to the matrix $N*I$ where $N$ is the sample size and we can proxy $\sum_{t=1}^{N} e_{t} e_{t}^{T} = NI$. Therefore the first-order condition is satisfied setting $\Sigma_{t}=A_{t}$ for each t. To prove it, just substitute and verify the following:

$$ NI = \sum_{t=1}^{N} \Sigma_{t}^{-1/2}A_{t}^{1/2}e_{t} e_{t}^{T}A_{t}^{1/2T}\Sigma_{t}^{-1/2T} \rightarrow NI = \sum_{t=1}^{N} Ie_{t} e_{t}^{T}I= NI $$

Answer 2

Here I post a solution for Q3, where we firstly assume that the true process is known and we show that taking $\Sigma_{t}=A_{t}$ (where $A_{t}$ is the "true" covariance matrix holding in the population) will maximize the likelihood in a sample drawn from that process.

Initially, during a first-stage estimation, we make the hypothesis that the unknown stochastic process is of the kind $y_{t}=\mu_{t}+\Sigma_{t}^{1/2}\epsilon_{t}$ with iid normally distributed random error $\epsilon_{t}=N(0,I)$ and positive definite $\Sigma_{t}$ for every t. And we estimate the parameters and choose the best specification of the model such that the estimated process represents the true process holding in the population. Suppose now that we already know the true values of the unknown parameters $\mu_{t}$ and $\Sigma_{t}$ desribing the process, because for example we have performed a first-stage estimation of the unknown parameters and we have correctly specified the model, so that now we can assume that we know the true process holding in the population and the true value of the unknown parameters as we have permed appropriate inference on that during the first stage. Assume indeed that the true process is such that $\Sigma_{t}=A_{t}$ with $A_{t}$ pd for each t. Suppose also that, for simplicity of the notation, $\mu_{t}=0$. So we know that the true process from which we observe a sample of lenght N is $y_{t}=A_{t}^{1/2}\epsilon_{t}$. We want to show that chosing $\Sigma_{t}=A_{t}$ maximizes the sample likelihood with respect to $\Sigma_{t}$ for each t in a sample drawn from that correctly-specified process. Or, analogously, it minimizes the additive inverse of the log likelihood.

Notice that, if the true process is the one mentioned above, then $ y_{t}y_{t}^{T}=A_{t}^{1/2}\epsilon_{t}\epsilon_{t}^{T}A^{1/2T}$ which implies $E_{t-1}(y_{t}y_{t}^{T})=A_{t}$, therefore we can write $y_{t}y_{t}^{T}=A_{t}+U_{t}$ where $U_{t}$ is a white noise (see for example this reference pag.5-6) such that $E_{t-1}(U_{t})=0$ and $E(U_{t})=0$. Therefore the –log likelihood to be minimized wrt $\Sigma_{t}$ can be re-written as:

$$-\log p(y|X) = \sum_{t=1}^{N} \frac{1}{2}y_{t}^T\Sigma_{t}^{-1}y_{t} + \frac{1}{2}\log|\Sigma_{t}|+\frac{n}{2}\log2\pi = \sum_{t=1}^{N} \frac{1}{2}\log|\Sigma_{t}|+\frac{n}{2}\log2\pi + \frac{1}{2}Tr(\Sigma_{t}^{-1}y_{t} y_{t}^{T})$$

Substituting $y_{t}y_{t}^{T}=A_{t}+U_{t}$ $$-\log p(y|X) = \sum_{t=1}^{N} \frac{1}{2}\log|\Sigma_{t}|+\frac{n}{2}\log2\pi + \frac{1}{2}Tr(\Sigma_{t}^{-1} (A_{t}+U_{t}))$$

Notice that $Tr(\Sigma_{t}^{-1} (A_{t}+U_{t})) = Tr(\Sigma_{t}^{-1} A_{t}+\Sigma_{t}^{-1} U_{t}) = Tr(\Sigma_{t}^{-1} A_{t})+Tr(\Sigma_{t}^{-1} U_{t})$. Notice also that, since Tr is a linear operator and commutes with expectations (see for example this), then the following holds $E(Tr(\Sigma_{t}^{-1}U_{t}))= Tr(E(\Sigma_{t}^{-1})E(U_{t}))$. As long as we have correctly specified the process at the first stage and now we are sampling from that true process, then $ E(U_{t})=0$ approximately holds in the sample (i.e. $ 1/N \sum_{t=1}^{N} U_{t} =0 \rightarrow \sum_{t=1}^{N} Tr(\Sigma_{t}^{-1} U_{t})=Tr(\sum_{t=1}^{N} \Sigma_{t}^{-1} U_{t})=Tr(0)=0$ with some neglectable rounding in a large well-behaved sample), then we can write a proxy for the -log likelihood as: $$-\log p(y|X) = \sum_{t=1}^{N} \frac{1}{2}\log|\Sigma_{t}|+\frac{n}{2}\log2\pi + \frac{1}{2}Tr(\Sigma_{t}^{-1} A_{t})$$

Taking the gradient wrt a generic $\Sigma_{t}$ and setting it equal to 0, we get: $$ \Sigma_{t}=A_{t}$$ Where $A_{t}$ is pd by assumption. Notice also that taking and additional derivation, the Hessian is $A_{t}$ which is pd by assumption, so we have found a minimum. Thus we show that taking $\Sigma_{t}$ equal to the true value of the parameter holding in the population maximizes the likelihood (or minimizes the negative log marginal likelihood) in a sample drawn from that process/population. Notice also that we do not even need to impose the constraint $det(\Sigma_{t})>0$ to show that the choice of $A_{t}$ is the choice of $\Sigma_{t}$ that minimizes the –log lik subject to the requirement that $\Sigma_{t}$ is pd for every t, if we perform some simplification on the expression of the sample likelihood (which is valid as long as the model is correctly specified and we are taking a value of $\Sigma_{t}=A_{t}$ which matches the true value in the population).