Assume we have two models for the same task.

One with 1K parameters and one with 1M and they achieve accuracy as follows

Model Params Accuracy
small 1 000 0.8
big 1 000 000 0.9

I would like to compare them is terms of price/time necessary to achieve given accuracy.

Is there some kind of an established metric to compare two models with regard to their size?

Obviously accuracy divided by number of params would be biased towards small models.

  • 2
    $\begingroup$ Why can't you compare them based on price/time directly? $\endgroup$
    – Firebug
    Commented Jun 6, 2022 at 13:41
  • 7
    $\begingroup$ You have model comparisons like using the Akaike Information Criterion (AIC) or Bayesian Information Criterion (BIC). These include the number of parameters. However, machine learning models are often generated using training and validation and the comparison is done by analysing a testing performance. In this latter comparison the number of parameters doesn't really matter. This makes it unclear what you are exactly thinking about. Why would you like to incorporate the parameters? What sort of comparison are you making exactly? $\endgroup$ Commented Jun 6, 2022 at 13:45
  • $\begingroup$ Maybe each one has been trained on a different setup - number of CPUs, GPUs, RAM size,... All these parameters would have to somehow be factored into the price/time. Also sometimes those parameters aren't even known, e.g. if a research paper doesn't provide them. $\endgroup$ Commented Jun 6, 2022 at 13:46
  • 2
    $\begingroup$ Welcome to Cross Validated! I can't overemphasize how important the questions by @SextusEmpiricus are to answer thoroughly. $\endgroup$
    – Dave
    Commented Jun 6, 2022 at 13:47
  • $\begingroup$ If those parameters to determine a price/time: performance ratio are unknown, then maybe there is little point in trying to come up with a decent comparison. $\endgroup$ Commented Jun 6, 2022 at 13:48

2 Answers 2


A lot of the classical metrics such as AIC and BIC are made for classical statistical models rather than ML models. In addition classical theory suggest that validation performance is a convex function of model complexity. This is a consequence of the classic variance bias trade-off.

We know this not to be true anymore: deep learning models are so strongly overparameterised that they should perform horribly but they actually perform well. What we observe is a doubly deep descent: with model complexity the validation performance improves, worsens and improves again. This behaviour is not very well understood (to the best of my knowledge). The field of „statistical learning theory“ does tackle this problem, but I am not aware of any metrics that are practically usable as a judgement of model complexity. One such attempt is Rademacher Complexity, which describes how well the model can fit to random data (roughly speaking) thus indicating its potential to overfit. More theory comes from PAC Bayesian theory about generalisation as function of model complexity. However, all of these are very theoretical measures of complexity and I have never seen them applied anywhere.

TL;DR I don’t think there are practical measures of complexity for modern models that are based on statistical theory - equivalent to AIC and BIC.

When it comes to practical cost as a consequence of complexity, there are multiple measures which boil down to: inference latency or model performance as function of training time on a certain device with a certain package. Reading through some recent papers on big, big models will give you an idea of what exactly to calculate.

  • 2
    $\begingroup$ We know this not to be true anymore It is still true. It is only not applicable in every situation. For large models with many parameters it makes indeeed little sense to use such metrics. But we should not regard these novel models, like deep-learning, as being over-parametrised in the classical sense. Effectively the dimension/size of the model space is much smaller than the number of parameters suggests. There is no overfitting because measures are taken to prevent this. $\endgroup$ Commented Jun 7, 2022 at 20:26

It's hard to see how you can compare two models in terms of CPUs, GPUs, RAM, ... if the only information you have is the number of parameters. You can run the same model on hardware with very different cost and running time (different number and type of instances for example) while the accuracy remains the same.

In practice you would distinguish between optimizing metrics and satisficing metrics. Your budget can determine the satisficing metrics price and time. Among models with acceptable cost and running time, you will pick the best model in terms of the optimising metric.

See Andrew Ng's Machine Learning Yearning book.


Information critera such as AIC and BIC are defined as likelihood + penalty. Basically, if the model isn't fitted by maximizing a likelihood function, information criteria are not applicable.

Akaike Information Criteria applied on Random Forest

Picking appropriate infrastructure for a given model architecture is not trivial. Training and prediction usually have different constraints and so require different infrastructure. The book Learn Amazon SageMaker has practical advice about optimizing cost and performance. Obviously, it's specific to AWS.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.