Usually, when a difference of a statistic is discussed, that discussion is presented in the context of a significance of that difference. When self-entropy, i.e., information content, is examined, especially, but not only when non-nested models are compared, we use the lower value of the AIC, AICc, BIC or other information content index to suggest what the better model is. However, more generally, entropy is case-wise, i.e., data-wise, variable.

Question With what certainty do we know, based on comparative information content indices from a particular data set, that that lower index value properly suggests the correct model more generally for a less limited data set?

I feel that non-nested model comparison of information content is not always relevant in all circumstances, for example see this Q/A. However, I really would appreciate any insight into the variability of comparison of information content for either nested or non-nested models in the context of subset data.

Usually, when a difference of a statistic is discussed, that discussion is presented in the context of a significance of that difference. When self-entropy, i.e., information content, is examined, especially, but not only when non-nested models are compared, we use the lower value of the AIC, AICc, BIC or other information content index to suggest what the better model is. However, more generally, entropy is case-wise, i.e., data-wise, variable.

Question With what certainty do we know, based on comparative information content indices from a particular data set, that that lower index value properly suggests the correct model more generally for a less limited data set?

I feel that non-nested model comparison of information content is not always relevant in all circumstances, for example see this Q/A. Nesting is when all of the models tested can be derived by eliminating parameters from a parent model. Non-nesting is when the models contain parameters that are not in a set with subset(s) format.

I really would appreciate any insight into the variability of comparison of information content for either nested or non-nested models in the context of subset data.