When using ordinary least square (OLS) method, there are certain residual diagnostics that need to be performed. In a similar manner, what diagnostics should be performed when using Maximum Likelihood Estimation (MLE) when fitting a model?
$\begingroup$
$\endgroup$
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
7
A typical testable assumption would be that the errors of the model are i.i.d. Then the likelihood of the vector (corresponding to a sample) of the dependent variable is the product of the individual likelihoods of each element in the vector, or equivalently, the log-likelihood of the vector of the dependent variable is the sum of the individual log-likelihoods of each element in the vector. This feature is typically used in formulating the likelihood function.
You can test the assumption using residuals. People usually look for violations of independence (though it is also possible to inspect the "identically distributed" part of the i.i.d. assumption, too). Thus they seek patterns in the residuals revealing some sort of dependence. E.g. in a time series setting, these could be nonzero autocorrelation, autoregressive conditional heteroskedasticity, level shifts and changes of the unconditional variance, among other. Tests such as Ljung-Box, Breusch-Godfrey, ARCH-LM and various structural breaks tests are used for detecting these types of dependence.
Then there are some regularity conditions (see Wikipedia) to ensure nice behavior such as consistency and asymptotic normality of the MLE, but I do not know if any of them are tested using residuals.
answered Jan 12, 2022 at 15:35
-
$\begingroup$ Great explanation! In one of the papers I came across which uses the EM algorithm for Dynamic Factor Model estimation, the errors were allowed to follow an AR(1) process. Is this possible? $\endgroup$ Commented Jan 12, 2022 at 17:13
-
$\begingroup$ @Geek_Tech, if it is in a paper, it is probably possible. I do not immediately see a problem with it, but some more context might be needed. But is this related to this thread? Might it be better suited for a new thread perhaps? $\endgroup$ Commented Jan 12, 2022 at 17:31
-
1$\begingroup$ Question is relevant because the answer is simply incorrect, MLE doesn't assume i.i.d, it assumes whatever the likelihood that is optimised assumes. If the likelihood assumes i.i.d., then this is the assumption, but if the likelihood is AR1, the MLE for the AR1 process will of course not assume i.i.d, see e.g. rdocumentation.org/packages/lmeNB/versions/1.2/topics/… $\endgroup$ Commented Jan 18, 2022 at 8:52
-
1$\begingroup$ Yes, when you condition on $u_{t-1}$ the remaining error is iid, but this is a property of AR1 and not MLE. In many correlations structures, you can condition on something such that the remaining variance is independent. This doesn't change the point that MLE assumes that the data is distributed according to the assumptions in the likelihood, which are by no means restricted to i.i.d. $\endgroup$ Commented Jan 18, 2022 at 11:16
-
1$\begingroup$ @FlorianHartig, I have said nothing about how the data is distributed. I have only said the model's errors are i.i.d. The confusion between assumptions on data vs. on errors is common among beginners, but I have avoided it. I agree that the data is distributed according to the assumptions in the likelihood, but the question is what those assumptions are. My answer names one of them. $\endgroup$ Commented Jan 18, 2022 at 11:48
$\begingroup$
$\endgroup$
3
When performing MLE, model parameters $\Theta$ are optimised according to the likelihood $p(D|\Theta)$, where $p(D|\Theta)$ is calculated according to the assumed likelihood structure -> data-generating process. There is a large large number of such processes (and consequently likelihoods -> models) available, including likelihoods for which residuals are expected to be heteroskedastic and / or correlated (see e.g. R package nlme).
If you want to test if your observed data / residuals fit to your model assumptions for a given likelihood, the general idea is that they should behave as you would expect, given your likelihood. For example, if you have a likelihood that assumes residual autocorrelation, you would expect your naive model residuals to have the same property.
How to best check this in practice depends a lot on which model we are talking about. For GLMMs in R, the R package DHARMa (disclaimer: I am the developer) may be a good starting point.
-
$\begingroup$ In my case the used algorithm is EM algorithm. If we assume that the error terms follow an AR(1) process, is there a specific method to test the fit of the model through residual diagnostics? $\endgroup$ Commented Jan 18, 2022 at 14:44
-
1$\begingroup$ I would suggest opening up a new question, asking for how to check the assumptions of an AR1 process (or check if that hasn't been asked before). $\endgroup$ Commented Jan 18, 2022 at 15:43
-