First of all, statistics is just not my thing ... yet (I hope!)

I'm having a hard time finding out the log-likelihood equation:

Given $Y \rightarrow \mathcal{N}(\mu_1,\sigma_1)$ (observation) and $\hat{Y} \rightarrow \mathcal{N}(\mu_2,\sigma_2)$ (prediction) because of our big sample, $\approx 2500$ values), we have

$$Y+(-\hat{Y}) \rightarrow \mathcal{N}(\mu,\sigma) \quad \text{where} \quad \mu = \mu_1 - \mu_2 = 0 \quad \text{and}\quad \sigma = \sqrt{\sigma_1^2 + \sigma_2^2}$$

Therefore, the probability density function $f$ of our normal distribution can be writen as follows:

\begin{equation} f(y-\hat{y}\, |\,0,\sigma) = \dfrac{1}{\sigma\sqrt{2\pi}} \ \exp\left(-\frac{\left(y-\hat{y} \right)^2}{2\sigma^2}\right) \end{equation}

The likelihood forumla is the following with $x_i=y_i-\hat{y}_i$: $$ \begin{align} \mathcal{L}(\theta\, ; \, x_1,...,x_n) &= \prod_{i=1}^n f(x_i\, |\,\theta)\\ & = \frac{1}{\sigma^n\left(\sqrt{2\pi}\right)^n}\, \exp \left(-\frac{1}{2\sigma^2}\sum_{i=1}^n x_i^2\right) \end{align} $$

The log-likelihood is therefore given by: $$ \ln\left(\mathcal{L}\left(\theta\, ; \, x_1,...,x_n\right)\right) = -\frac{n}{2}\left[\ln\left(\sigma\right)+\ln\left(2\pi\right)+ \frac{1}{n\sigma^2}\sum_{i=1}^{n} x_i^2\right] $$


Standardizing data using $z_i = \dfrac{y_i-\mu_1^i}{\sigma_1^i}$ and $\hat{z}_i = \dfrac{\hat{y}_i-\mu_2^i}{\sigma_2^i}$, we have:

$$ \begin{align} &Z - \hat{Z} \rightarrow \mathcal{N}(0,1)\\ &f(z-\hat{z}\, |\, 0,1) = \dfrac{1}{\sqrt{2\pi}} \ \exp\left(-\frac{\left(z-\hat{z}\right)^2}{2}\right)\\ &\mathcal{L}(\theta\, ; \, x_1,...,x_n) = \frac{1}{\left(\sqrt{2\pi}\right)^n}\, \exp \left(-\frac{1}{2}\sum_{i=1}^n x_i^2\right) \quad \text{with $x_i = z_i - \hat{z}_i$}\\ &\ln\left(\mathcal{L}\left(\theta\, ; \, x_1,...,x_n\right)\right) = -\frac{n}{2}\left[\ln\left(2\pi\right)+ \frac{1}{n}\sum_{i=1}^{n} x_i^2\right] \end{align} $$

What am I doing wrong? Because, when computing AIC, the only likelihood formula used is the one below:

$$ \ln\left(\mathcal{L}\left(\theta\, ; \, x_1,...,x_n\right)\right) = -\frac{n}{2}\left[1 + \ln\left(2\pi\right)+ \ln\left(\frac{1}{n}\sum_{i=1}^{n} x_i^2\right)\right] $$

This forumla seems to be coming from the log-normal distribution, although I didn't achieve to demonstrate it neighter. I'm not asking for the entire proof, but just a hint on how to do it the right way.


My second question is: When I'm saing 'variables' below, I'm refering to the independent ones. The dependant one is and will always remain the same accross the models.

Is it statistically or mathematically possible to do a multivariate linear regression using linear AND non linear variables if those non linear variables seem to improve the model? If the answer is YES, can the AIC be used on a model combining linear and non linear variables?

Example: $Y$ is the dependent variable and $A_1,A_2,A_3,A_4$ are the independent variables.

The model to be found will be of the form: $\hat{Y} = \beta_0 + \sum_{i=1}^{4} \beta_i A_i$

Now, plotting Y(A_i) is giving the following results:

$Y$ vs $A_1$ or $A_2$ seems linear.

$Y$ vs $A_3$ or $A_4$ doesn't seem to be linear

And the models to be tested are as follows:

$\mathcal{M}_1$ includes $A_1$

$\mathcal{M}_2$ includes $A_1$ and $A_2$

$\mathcal{M}_3$ includes $A_1$ and $A_3$

$\mathcal{M}_4$ includes $A_1$ and $A_2$ and $A_4$

$\mathcal{M}_5$ includes $A_1$ and $A_2$ and $A_3$ and $A_4$

Can the AIC be used to compare those 5 models? (I know that AIC can be used on non-nested models).

Last question: Can somebody enumerate the hypothesis that must be checked in order to use the AIC? I've read a lot about the AIC but I can't make up my mind about what is mandatory and what is not. The hypothesis I've seen until now are the following:

1) The most complete model (containing all the variable that we want to test) must be tested against the dependent variable values. If $\chi^2$ test is not significant at the $95^th$%, we cannot use the AIC.

2) Each independent variables must be tested against the dependent variable, alone. Only the one seeming to be linear must be retained.

3) Predictive variables must be independent and normally distributed.

  • $\begingroup$ Well, it seems that I cannot edit my message. First of all, Hello everybody! $\endgroup$ – juh Aug 27 '15 at 10:53
  • $\begingroup$ That sounds odd -- if you can comment you should be able to edit. ... re your question: if $E(Y)=\mu$ and $E(\hat{Y})=\mu$, why do you think $E(Y-\hat{Y})=\mu$? $\endgroup$ – Glen_b Aug 27 '15 at 13:13
  • $\begingroup$ Now I can edit. Well, my mistake. I forgot indexing \mu and \sigma in the first place. $\endgroup$ – juh Aug 27 '15 at 14:18
  • $\begingroup$ But you didn't fix it! You still have: $f(y-\hat{y}\, |\, \mu,\sigma) = \dfrac{1}{\sigma\sqrt{2\pi}} \ \exp\left(-\frac{\left((y-\hat{y}) - \mu \right)^2}{2\sigma^2}\right)$, so you still seem to think that $\mu-\mu=\mu$. $\endgroup$ – Glen_b Aug 27 '15 at 16:22
  • $\begingroup$ I'm not sure I'm following you. I don't see where $\mu = \mu - \mu$. After adding the indexes to $\mu$ and $\sigma$, $\mu = \mu_1 - \mu_2$ as corrected in my first post. Where is my mistake? (I'm all new to this) $\endgroup$ – juh Aug 27 '15 at 18:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.