Which statistical test to compare same model with different parameters? I have two datasets on people buying apples based on weight and price. One dataset in 2019 the other in 2020.
I estimate a logit model with Utility = betaWeight * weight + betaPrice * price.
Training on 2019 I obtain Model2019 -> betaWeight = 1, betaPrice = 2
Training on 2020 I obtain Model2020 -> betaWeight = 1, betaPrice = 4
I want to have a statistical test, saying the model of 2020 is better than the one of 2019 to explain the data in 2020. Basically, the difference in the estimated parameters is not due to a sampling error, but it is real.
If I compute the likelihood for predicting data of 2020:
L0 = Null log likelihood = -3500
L2019 = Log likelihood using Model2019 = -1100
L2020 = Log likelihood using Model2020 = -1020
Should I use the likelihood ratio test? Akaike test?
Thanks
 A: To test that the model built on one data set is better performing than the model built on the other data set:
Let predictions of model 1 be called $y_i$ and predictions of model 2 be called $z_i$. We want to show that $z_i$ is usually more correct than $y_i$. Define a score metric for every sample and run a paired t-test to determine that $score(z_i) > score(y_i)$.
See here
What you do is:
Let $d_i = (score(z_i) - score(y_i))$ and $d$ be the average of $d_i$ over all samples. Let $s_d$ be the standard deviation over $d_i$.
let $t = \frac{d}{\frac{s_d}{\sqrt{n}}}$
This test statistic follows a t-distribution with $n-1$ degrees of freedom.
You mentioned the problem is classification. A common loss function for individual samples in classification problems are 0-1 loss or brier score.
A: I have an answer which I am not sure about, but I would like to get the idea out in the open.
Use M19 to denote the model estimated on 2019 data and M20 for the one estimated on 2020 data.
Use P19 to denote the predictions of M19 for the 2020.
Use F20 to denote the fitted values of M20 on the 2020 data.
Use L19 to denote the losses from prediction errors corresponding to P19. Specifically, the losses are defined as negative log-likelihood of each observation. If you are fine with evaluating the models by their log-likelihoods, such losses will work fine.
Now use the Diebold-Mariano test with a twist. The ordinary Diebold-Mariano test compares two sets of forecast losses arising from two sets of forecasts of the same set of target values. So it can compare L19 with L20. However, we do not have L20 because we do not have P20, only F20. Here we can employ AIC to help us. We know that AIC estimates twice the negative log-likelihood of observations in the hypothetical case of the model having been fit on another dataset (of the same size and generated by the same data generating process) and used to predict the dataset at hand. Replacing the actual negative log-likelihood of M20 with half the AIC will provide an estimate of L20. From here on, you can compare L19 with the estimated L20 using the Diebold-Mariano test.
I can see two problems with this approach:

*

*We do not have the actual L20, only its estimate. My inutition is that this should not be a big problem in large samples, but I could be wrong.

*The negative log-likelihood may not be your favorite loss. If so, my approach is not directly applicable.

