0
$\begingroup$

I have a dataset that has 40 experimental observations of cells' activity, $n=40$, I tested several models using each of these samples. The model can only explain one cell at a time due to variability between the cells. Hence, for each model I have 40 values of log-likelihood and 40 parameter sets for each model.

I thought that this is how I should calculate the AIC:

$AIC = -2\times \sum_{i=1}^n log\mathcal{L}_i + 2\times k $

where $k$ is a number of model parameters.

But because the models do not explain all the data with one set of parameters, and each cell ends up with their own best parameter set, I was wondering I should be dividing the log likelihood by $n$? I.e.:

$AIC = -2\times \frac{1}{n} \sum_{i=1}^n log\mathcal{L}_i + 2\times k $

$\endgroup$
  • $\begingroup$ How do you get the second formula? $\endgroup$ – Glen_b Jan 8 at 3:02
  • $\begingroup$ Not sure if I understand the question? If you are asking what made me think it's the second formula it was this: AIC = n*log(residual sum of squares/n) + 2K which is written here: pypi.org/project/RegscorePy $\endgroup$ – Ale Jan 8 at 7:51
  • 1
    $\begingroup$ That's not the same as the formula in your question though. $\endgroup$ – Glen_b Jan 8 at 14:21
  • $\begingroup$ I was wondering if you could explain why? I thought 'residual sum of squares' in the formula I linked would be equivalent to the sum of the log likelihood values of all my samples? $\endgroup$ – Ale Jan 8 at 15:14
  • $\begingroup$ Ok, sorry, I only just noticed that the formula in the link is different to my second formula... But I am still confused which one of the ones I wrote I should be using, if I should be dividing the sum of log(L) by sample number or not, I was wondering if you have any suggestions? $\endgroup$ – Ale Jan 8 at 15:41
0
$\begingroup$

The AIC is given explicitly in (for example) Akaike, 1974[1] (including in the abstract) as:

$^{-2 \log(\text{maximum likelihood}) + 2(\text{number of independently adjusted parameters within the model})}$

when you have independence of observations, this becomes your first form.

If you adjust AIC by shifting it, nothing of consequence changes (as long as the same shift is applied to every such term that is compared).

If you scale the entire AIC, that still allows less than or greater than comparisons but it's no longer an adjusted likelihood. People sometimes divide the entire AIC by $n$, which is like an average adjusted likelihood (and there can sometimes be good reasons to do this but it's not strictly AIC)

I see no justification whatever for dividing the first term by $n$ but leaving the second term alone; that changes the relative impact of the two terms and you are no longer doing what Akaike was doing.

In your case you're fitting multiple models to different samples. If you treat the samples as independent of each other, the original Akaike formulation works as is for this collection of models as long as you add the log-likelihoods and the parameters for each model. i.e. it's a model with as many observations as the total number of observations (assuming no overlap; these are supposed to be independent) and the number of parameters is the total number of parameters.

If you then decide to scale to some kind of average, you can do so (but as I mentioned earlier, it's no longer strictly AIC)

[1] Akaike, H. (1974),
"A new look at the statistical model identification",
IEEE Transactions on Automatic Control, 19 (6): 716–723, doi:10.1109/TAC.1974.1100705, MR 0423716.

$\endgroup$
  • $\begingroup$ Thank you so much for taking the time to write this. I was just wondering the bit where you say "and the number of parameters is the total number of parameters", do you mean the second term in the equation should be $-2\times k \times n$ ? $\endgroup$ – Ale Jan 10 at 0:23
  • $\begingroup$ Your notation is a little confusing to me so I have avoided it. You would add the number of free parameters in each model as long as they're all separate (no common parameters across those models). If you have a parameter $\theta_1$ in model A and another in model B, that's actually two parameters for the combined data $\theta_{1A}$ and $\theta_{1B}$. For example if I collect data for males and data for females and fit a pair of simple regression models, I have six free parameters ($\beta_{0F},\beta_{1F},\sigma_F,\beta_{0M},\beta_{1M},\sigma_M$). $\endgroup$ – Glen_b Jan 10 at 0:37
  • $\begingroup$ But if I combine them into a single model (using indicators / dummies for sex and the interaction of that with the predictor / IV), I still have the four parameters for the two lines but only one $\sigma$ parameter, making 5 in all. In each case you just count how many "independently adjusted" parameters you estimate, like Akaike said in that little bit I quoted. $\endgroup$ – Glen_b Jan 10 at 0:40
  • $\begingroup$ Ok, thank you so much, I think I might be slowly getting it. I'm just going to write this again just so that I am sure I am understanding what you're saying: $\endgroup$ – Ale Jan 10 at 1:11
  • $\begingroup$ In my case I am recording data from 40 cells, in the end I would like to determine which model is the best at describing all the cells. So let's say I want to test two models using AIC, let's say both models have 5 parameters. I fit Model1 and Model2 to each of my 40 samples, so each sample has unique parameters, so I end up independently adjusting 40x5 parameters? So the penalising term becomes 2x40x5? and the first term in the equation will be the sum of the log likelihoods of all 40 unique parameter sets? $\endgroup$ – Ale Jan 10 at 1:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.