Does the size of the training dataset affect the AIC? I'm conducting an experiment in which I want to compare the performances of 2 models. Both trained using the same algorithm (Logistic regression).
I split the data ($n=10000$) I have into 3 parts, train1 ($n_1=5000$) and train2 ($n_2=3000$) and test ($n_3=2000$).
I explicitly made sure no observation from both training sets falls into the testing set.
I built 2 models $m_1$ using train1 and $m_2$ using train2. And tested them on the testing set.
I repeated this 100 times. I always find that the model with larger training observations has the highest AIC which is somehow counter-intuitive for me.
Any explication to why this might occur? Does training size affect the AIC?
 A: I think there might be an issue of a numbers game, where the larger number of observations is messing around with the likelihood function and artificially boosting the AIC. Have you re-ran your analyses by setting train1 and train2 to be 4,000? Better yet, have you ran both models on train1 and train2 and measured the AIC on each?
If you go to the AIC formula, you get that
$$AIC = 2k - 2 ln(\hat{L}),$$
where

*

*$\hat{L}$ is the likelihood

*k = number of parameters estimated

then that since $$\hat{L}=\prod_i^N\text{individual likelihoods}$$, then if all the likelihoods are exactly the same, taking a larger product will change the value.
For example $0.99^5 \approx .95$ whereas $0.99^{10}\approx 0.90$. In your case, the difference of 2000 observations in the sample, might be confounding the actual AIC results.
AIC is meant to evaluate different models (i.e., different nested models or variable combinations) on the same dataset and not across datasets. My recommendation would be to calculate the AIC for both model1 and model2  first on the same dataset. And then repeat that on different subsamples (or maybe implement a cross-validation sampling-scheme) to make sure then that the AIC difference is not a function of the sample.
Hope this helps.
