For data $ (y_{1},x_{1}),...(y_{n},x_{n})$, consider the polynomial regression $y_{i} = g(x_{i}) + \epsilon_{i} $ where $\epsilon_{i}$ are iid gaussian errors $N(0,\sigma^{2})$, and $g(x) = a_{0} + a_{1}x +...+a_{p}x^{p}$ is a polynomial of order $p$ and coefficients $\boldsymbol{\hat{\theta}} = \{a_{0},...,a_{p}\}$. I wish to determine the the MDL term $L(y|\boldsymbol{\hat{\theta}}) = - \log(P(y|\boldsymbol{\hat{\theta}}))$. In (Lee, 2001) the result is given as $L(y|\boldsymbol{\hat{\theta}}) = \frac{n}{2}\log(\frac{RSS_{p}}{n})$ where $RSS_{p} = ‎‎\sum_{i=1}^{n}\{y_{i}-(a_{0} + a_{1}x_{i} +...+a_{p}x_{i}^{p})\}^{2}$. However, my work leads to a different result:

$P(y|\boldsymbol{\hat{\theta}}) = \large\prod_{i=1}^{n}\left(\frac{1}{\sqrt{2\pi}\sigma}\right)\exp\left(\frac{-(y_{i}-g(x_{i}))^{2}}{2\sigma^{2}}\right)$ $=\left(\frac{1}{\sqrt{2\pi}\sigma}\right)^{n}\exp\left(\frac{-\sum_{i=1}^{n}(y_{i}-g(x_{i}))^{2}}{2\sigma^{2}}\right)$

$L(y|\boldsymbol{\hat{\theta}}) = - \log(P(y|\boldsymbol{\hat{\theta}})) = \frac{1}{2\sigma^{2}}RSS_{p} + \frac{n}{2}\log(2\pi\sigma^{2})$

How can I reconcile these two expressions? Am I wrong or is there some algebra I can do to show they are equivalent?

Update: I may have figured it out. If you assume $ P(y|\boldsymbol{\hat{\theta}}) = \left(\frac{1}{RMSE}\right)^{n}$ where RMSE is the root mean square error, then you get the result in (Lee, 2001).

Lee, Thomas CM. "An Introduction to Coding Theory and the Two‐Part Minimum Description Length Principle." International statistical review 69, no. 2 (2001): 169-183.



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.