1
$\begingroup$

Ordinary least square solution of linear regression can be derived from the assumption of normally distributed residuals: $$ e_i=y_i-\hat{y_i}\\ e_i\sim N(0, \sigma^2) $$

What I don't quite understand from reading a lot of derivations is how we go from $MLE=\arg \max_{\theta}\sum{log(p_{\theta}^i)}$ to the sum of logarithms of PDFs of normal distribution. PDFs are called "probabilities" everywhere I read (e.g. https://machinelearningmastery.com/linear-regression-with-maximum-likelihood-estimation), which is incorrect. PDFs can be > 1. Only integrals of PDFs can be interpreted as probabilities of X falling into the integration interval.

Do I understand correctly, that the proper derivation should introduce a small interval $\epsilon$, so that we could first rewrite probabilities as:

$$ p_{\theta,\epsilon}^i=\int_{y_i-\epsilon}^{y_i+\epsilon}PDF(N(\hat{y^i}, \sigma^2))dy $$

Then we could assume that with small enough $\epsilon$ $$ p_{\theta,\epsilon}^i\approx 2*\epsilon*[\frac 1 {\sqrt {2*\pi}*\sigma} e^{-(\hat{y_\theta^i} - y_i)^2/\sigma^2}] $$

Now MLE can finally be approximated with $$ \require{cancel} MLE=\arg\max_{\theta}\sum log(2*\epsilon*[\frac 1 {\sqrt {2*\pi}*\sigma} e^{-(\hat{y_\theta^i} - y_i)^2/\sigma^2}])\\ =\arg\max_{\theta}\sum \cancel{log(2*\epsilon)} + \cancel{log(\frac 1 {\sqrt {2*\pi}*\sigma})} -(\hat{y_\theta^i} - y_i)^2/\sigma^2\\ =\arg\min_{\theta}\sum (\hat{y_\theta^i} - y_i)^2 $$

And the last expression is the OLS.

$\endgroup$
2
  • 1
    $\begingroup$ you would be quicker to open a textbook than write all that. you are incorrect you just need the pdf directly. see eg en.wikipedia.org/wiki/Maximum_likelihood_estimation $\endgroup$
    – seanv507
    Aug 5, 2022 at 13:10
  • 1
    $\begingroup$ Welcome to CV! You're correct about the sloppy language that conflates a probability density with a probability. Concerning the introduction of a "small interval," please see stats.stackexchange.com/a/397166/919. Then read the other answers in that thread (especially Ben's post) to see why that small interval does not need to be introduced. $\endgroup$
    – whuber
    Aug 5, 2022 at 13:51

1 Answer 1

1
$\begingroup$

You seem to mix two issues: first, how to interpret maximum likelihood estimation and second the derivation of OLS as an ML estimator.

Regarding the first point: Leaving mathematical rigour aside and thinking about the likelyhood about the joint probabilities for all possible small intervals can be useful. Let $\mathbf{y}=\{y_1, y_2..., y_n\}$ the data we observe. For each datapoint, we want to choose an estimator which maximises the probability to observe data in a small neighbourhood of $y_1$ and in a small neighbourhood of $y_2$ and in a small neighbourhood of $y_3$ and so on... We impose that these probabilites are compatible with the assumption of all $e_i=y_i-\hat{y}_i$ being drawn form a normally distributed random variable.

Regardingthe second point, i.e. estimation I agree with seanv507: Following wikipedia notation you are maximising the likelihood function which is the product of individual densities:

\begin{equation} f_n(\mathbf{y,\theta})=\Pi_{k=1}^n f_k^{univar}(y_k;\theta) \end{equation} plugging in univariate normal densities, taking logs and simplifying along the lines you do gives you the desired result.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.