Assume we have two models for the same task.
One with 1K parameters and one with 1M and they achieve accuracy as follows
| Model | Params | Accuracy |
|---|---|---|
| small | 1 000 | 0.8 |
| big | 1 000 000 | 0.9 |
I would like to compare them is terms of price/time necessary to achieve given accuracy.
Is there some kind of an established metric to compare two models with regard to their size?
Obviously accuracy divided by number of params would be biased towards small models.
A lot of the classical metrics such as AIC and BIC are made for classical statistical models rather than ML models. In addition classical theory suggest that validation performance is a convex function of model complexity. This is a consequence of the classic variance bias trade-off.
We know this not to be true anymore: deep learning models are so strongly overparameterised that they should perform horribly but they actually perform well. What we observe is a doubly deep descent: with model complexity the validation performance improves, worsens and improves again. This behaviour is not very well understood (to the best of my knowledge). The field of „statistical learning theory“ does tackle this problem, but I am not aware of any metrics that are practically usable as a judgement of model complexity. One such attempt is Rademacher Complexity, which describes how well the model can fit to random data (roughly speaking) thus indicating its potential to overfit. More theory comes from PAC Bayesian theory about generalisation as function of model complexity. However, all of these are very theoretical measures of complexity and I have never seen them applied anywhere.
TL;DR I don’t think there are practical measures of complexity for modern models that are based on statistical theory - equivalent to AIC and BIC.
When it comes to practical cost as a consequence of complexity, there are multiple measures which boil down to: inference latency or model performance as function of training time on a certain device with a certain package. Reading through some recent papers on big, big models will give you an idea of what exactly to calculate.
It's hard to see how you can compare two models in terms of CPUs, GPUs, RAM, ... if the only information you have is the number of parameters. You can run the same model on hardware with very different cost and running time (different number and type of instances for example) while the accuracy remains the same.
In practice you would distinguish between optimizing metrics and satisficing metrics. Your budget can determine the satisficing metrics price and time. Among models with acceptable cost and running time, you will pick the best model in terms of the optimising metric.
See Andrew Ng's Machine Learning Yearning book.
Updates
Information critera such as AIC and BIC are defined as likelihood + penalty. Basically, if the model isn't fitted by maximizing a likelihood function, information criteria are not applicable.
Akaike Information Criteria applied on Random Forest
Picking appropriate infrastructure for a given model architecture is not trivial. Training and prediction usually have different constraints and so require different infrastructure. The book Learn Amazon SageMaker has practical advice about optimizing cost and performance. Obviously, it's specific to AWS.
