Say, I have created 3 models that evaluates the amount of harmful pollution. It uses a variety of factors (location, time of year, etc.) to determine this.

I have a bunch of data for 10+ Cities from the 3 models over several years.

For example,

City A:

  • Jan/01/2012: 0.3334g
  • Jan/02/2012: 0.3335g
  • Jan/03/2012: 0.3335g
  • ...
  • Jan/01/2015: 0.5002g

City B:

  • Jan/01/2012: 0.3333g
  • Jan/02/2012: 0.3330g
  • Jan/03/2012: 0.3332g
  • ...
  • Jan/01/2015: 0.4997g

City C:

  • Jan/01/2012: 0.3331g
  • Jan/02/2012: 0.3332g
  • Jan/03/2012: 0.3000g
  • ...
  • Jan/01/2015: 0.5100g

And so on for City D, E, F, G...

What's the best statistical test to determine if one model is 'better' or more accurate than another? For example, for the Jan/01/2015 date, the City C model appears to be off more than the others. Can we use a test to say, with confidence, that there is a statistically significant difference between the models, or something akin to that?

The difficulty lies in the fact that there is no reference for comparison (i.e. the 'amount of harmful pollution' is simply a made up statistic).

Note: In general, the pollution has been trending upwards (but of course, it could trend down, or not all).

Additional question: What if I have more than 3 models to test?

  • $\begingroup$ Are your models nested or not ? $\endgroup$
    – Riff
    Jun 17 '16 at 7:29
  • $\begingroup$ I'm not sure what you mean, but I believe they are NOT nested. That is, the models do not depend on the other models. The figures are independently calculated using different formulas and methods. $\endgroup$
    – deelie
    Jun 17 '16 at 7:49
  • $\begingroup$ Okay. Just to be safe, roughly models are nested if one of the models uses a subset of variables of another model. For example, Y ~ A + B is nested in Y ~ A + B + C. $\endgroup$
    – Riff
    Jun 17 '16 at 7:55
  • $\begingroup$ In that case, they would all be roughly nested. For example Model A has inputs from variables x1,x2,x3,x4,x9 .... While model B has inputs from x1,x2,x4,x5,x6,x7... Model C has inputs from variables x2,x3,x5,x7,x8. There is no particular pattern. $\endgroup$
    – deelie
    Jun 17 '16 at 7:58
  • 1
    $\begingroup$ If they are not nested then LRT (likelihood ratio test) is not a good idea but you can still compare their AIC or BIC (Akaike Information Criterion & Bayesian Information Criterion) and choose the ones that minimizes it (en.wikipedia.org/wiki/Akaike_information_criterion), what you can also do is compare their RMSE (Root Mean Square Error) [en.wikipedia.org/wiki/Root-mean-square_deviation] which translates how good fares a model in predicting new values. $\endgroup$
    – Riff
    Jun 17 '16 at 8:00

Okay so considering the comments, your models appears to be nested you have the following methods to asses the 'best' model.

Likelihood Ratio Test

The basis for that test is to compare the likelihood of two models while taking into account that using more parameters will almost always increase likelihood. To put it simply it is a matter of checking if it is worth to use more variables.

So let's say you have models $\mathcal{M}_1$ & $\mathcal{M}_2$ such that $\mathcal{M}_1 \subset \mathcal{M}_2$ meaning that $\mathcal{M}_2$ is using all the variables $\mathcal{M}_1$ uses plus some other variables. You can have $\mathcal{L}_2$ & $\mathcal{L}_2$ the likelihood of each model.

The test statistic is then $\mathcal{D} = -2\ln\left ( \frac{\mathcal{L}_1}{\mathcal{L}_2} \right )$

Since $\mathcal{M}_1 \subset \mathcal{M}_2$, $\mathcal{L}_2 \geq \mathcal{L}_1$ because the model with more parameters will always fit at least as much.

Under the null hypothesis that $\mathcal{M}_2$ is not better than $\mathcal{M}_1$ then $\mathcal{D}$ follows approximately a $\chi^2$ distribution of degrees of freedom $df_{\mathcal{M}_2} - df_{\mathcal{M}_1}$

Information Criterion

Information Criterion are always valid (whereas LRT is only for nested models) and is basically a penalization of the model likelihood by the number of parameters it has. There are two major IC for model selection: Akaike's and Bayes'. Their usual expression are as follows:

$AIC = 2k - 2\ln(\mathcal{L})$ & $BIC = - 2\ln(\mathcal{L}) + k \ln(n)$

Where $\mathcal{L}$ is the model likelihood, $k$ the number of parameters used by the model and $n$ the number of observations.

You want the model that minimizes your information criterion ($AIC$ & $BIC$ mean nothing on their own, there needs to be a set of $IC$ to be compared).

Which one to use is a matter of preference I would say. $BIC$ is known to perform better when $n$ is much larger than $n$ and $AIC$ penalizes less strongly than $BIC$.

Root Mean Square Error

The RMSE works only for models with quantitative output. It is a measure of how well your model is at predicting using new values. To get a valid measure of your RMSE you need to use it with cross-validation meaning that you separate your data in multiple parts that will be used to train the model and then estimate its prediction 'power'.

Once you have predicted the values of your testing dataset then you can compute the RMSE.

$RMSE = \sqrt{\frac{1}{n}\sum \limits_{i = 1}^n (\hat{y}_i - y_i)^2} $

With $\hat{y}_i$ being the predicted value for observation $i$ and $y_i$ being the actual value of observation $i$.

You would then want to choose the model that has the best prediction so the lowest RMS. The advantage of RMSE is that it gives you, by itself, a measure of your model worth (prediction-wise).

Note that only the LRT is truly only valid for nested models and that there exists lots of other ways to select a model but these are the most often used. Hope it helps,


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.