1
$\begingroup$

I am having a problem understanding how to estimate support for models when the number of samples is different. The scenario is as follows:

Suppose the log-likelihood of models $M_1$ and $M_2$ of some data $\mathbf{x}$ are $\ell_1$ and $\ell_2$ respectively, and $\ell_1 > \ell_2$, and that both models have the same number of estimated parameters. Then, $M_2$ is $\exp(\ell_2 - \ell_1)$ times as probable as the first model to minimise information loss from Wikipedia (and discussed here).

Now consider that $\mathbf{z}$ is the dataset consisting of replicating the observations in $\mathbf{x}$ twice. Then, models $M_1$ and $M_2$ have log-likelihoods equal to $2\ell_1$ and $2\ell_2$ respectively, and $M_2$ is $\exp(2(\ell_2 - \ell_1))$ times as probable as the first model to minimise information loss.

This gives relative probability different to the one first calculated. How can this be so?

Improve this question
$\endgroup$
1
  • $\begingroup$ Replicating the samples is just a convenient way of showing the mathematical calculation. we can assume we know the true generation process, then draw twice as many samples. $\endgroup$
    – Alex
    Commented Aug 8, 2017 at 11:51

1 Answer 1

Reset to default
1
$\begingroup$

To answer your main question, when there is twice as much data that has arisen completely independently and that supports a particular model more than another one, then it should be unsurprising that the better supported model is favored even more. Of course, when you just use a set of data twice, that is not the same thing as the duplicate of the original data occuring independently.

In terms of details: The quantity is proportional to that probability, not a probability itself (for a start it is not bounded).

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks, I corrected the wording in my question as you are right. However, there is no factor of 0.5 as the AIC has the term $2\ell$ in it. $\endgroup$
    – Alex
    Commented Aug 8, 2017 at 5:57
  • $\begingroup$ I agree broadly that the better supported model should be favored even more. However, I cannot reconcile this with the apparent fact that the relative ability of models to minimise information loss is determined by how much data you have. Surely this is an absolute number unaffected by the amount of data. Unless there is something I am missing from reading this quote in the wiki link: " Suppose that the data is generated by some unknown process f. We consider two candidate models to represent f: g1 and g2. If we knew f, then we could find the information lost from using g1 to represent f..." $\endgroup$
    – Alex
    Commented Aug 8, 2017 at 6:08
  • $\begingroup$ Not sure what the problem is. One of two candidate distributions will normally be closer to the true distribution in terms of the Kullback–Leibler divergence. There is a "truth" - if you knew the true distribution exactly, you could exactly calculate how well each candidate approximates it. And then there is what we can infer from data. The more data you have the better you can tell, which one of the two candidates it is likely to be to be closer to the true data generating model in terms of Kullback–Leibler divergence. $\endgroup$
    – Björn
    Commented Aug 8, 2017 at 7:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.