When performing AIC (or BIC) model selection between competing models, one performs individual fits for all models (on the same training data set) and then compares their AIC (or BIC etc) values. If the difference between the lowest AIC value for some model is greater than 10 we can say that the model selection is conclusive. There is a great deal of discussion on the topic in Burnham and Anderson.
My problem: I am fitting two parametric models of different functional form (same number of parameters, same type of response), say $y = \rho_1(r|\theta_1)$, and $y = \rho_2(r|\theta_2)$ that give fits of similar quality and their AICc difference is on the order of $\Delta \mathrm{AICc}\sim 2$ (which is not conclusive). Now, I know that one of these models is the "true" one (in the sense that it was the model from which the data were created), and this comes out to have the best AICc value. Since both models have same number of parameters, it is essentially their mean square error loss that makes the selection. However if I use a sample of test data (after training), the difference of their (average) square error loss climbs up to $\sim 10$.
My questions:
a. Is it possible to use AICc with test data (i.e. evaluate square error loss on test data, but the parameters of the model were obtained from a training data set) and perform model selection? My intuition says yes, but I have no theoretical justification for this. In addition I have no reference cut off value for conclusive model selection if I use test data set.
b. Is there some kind of error based model selection on test data that can be conclusive (in the sense that AICc is conclusive when the difference $\Delta \mathrm{AICc} > 10$)?
PS. I understand that "all models are wrong", in the sense that they are approximations of the data realization and perhaps the very meaning of conclusive model selection doesn't make much sense.