1
$\begingroup$

I'm reading Agresti - Categorical Data Analysis and it says

Consider two models, $M_0$ with fitted values $\hat{\mu}_0$ and $M_1$ with fitted values $\hat{\mu}_1$ with $M_0$ a special case of $M_1$. A smaller set of parameter values satisfies $M_0$ than satisfies $M_1$. Maximizing the log likelihood over a smaller space cannot yield a larger maximum. Thus $L(\hat{\mu}_0;y) \leq L(\hat{\mu}_1;y)$

But this is not like say $L(\theta)\leq L(\hat{\theta}), \forall \theta$ if $\hat{\theta}$ is the MLE. Because in that quote, the dimensions are different. Maybe the intuition is correct: the fit is more "likely" if I use more parameters to adjust the data. But I'd like a mathematical explanation of that quote.

Thanks

$\endgroup$
1
  • 1
    $\begingroup$ Could you explain what you mean by "dimensions are different"? And what kind of mathematical explanation do you need beyond the one already presented in the quotation? All it says is that when you limit the possible values of the argument $\theta$ you cannot arrive at a better solution--and that scarcely needs additional explanation. $\endgroup$
    – whuber
    Jan 28, 2019 at 20:31

2 Answers 2

1
$\begingroup$

This is only a statement of the training error.

As $M_0$ is a special case of $M_1$, each choice of parameter $\mu$ valid in $M_0$ is also valid in $M_1$(or better: has an equivalent model in $M_1$). Therefore if $\mu_0$ is the optimal solution, it is also contained in $M_1$, therefore $\mu_1=\mu_0$. If $\mu_0$ is not optimal, there might exist a parameter $\nu$ in $M_1$ with larger likelihood on the dataset and therefore $\mu_1=\nu$. This does not say anything about the generalisation error.

$\endgroup$
1
$\begingroup$

This statement is only correct if the fitted values are obtained as MLEs over the stated sets of parameter values. To see the implication in this case, suppose that $\hat{\mu}_0$ and $\hat{\mu}_1$ are MLEs over $M_0$ and $M_1$ respectively. (For simplicity of notation, I will take these as sets of parameters.) Then since $M_0 \subseteq M_1$ we have:

$$L_y(\hat{\mu}_0) = \underset{\mu \in M_0}{\max} L_y(\mu) \leqslant \max \Big( \underset{\mu \in M_0}{\max} L_y(\mu), \underset{\mu \in M_1-M_0}{\max} L_y(\mu) \Big) = \underset{\mu \in M_1}{\max} L_y(\mu) = L_y(\hat{\mu}_1).$$

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.