1
$\begingroup$

Reading many papers of deep-learning, I realized that the number of parameters and the computation costs are not exactly the same thing. When I read papers, people usually measure and compare model complexities based on the number of parameters. But I think computational cost should also be considered alongside with the number of parameters.

Taking a residual CNN network (ResNet) for example, this model reduces spatial dimension very fast and reduces computational cost very much by adopting stride=2 in the first conv layer and then max pool. Therefore, the spatial feature dimension becomes 1/4 of the input in the third layer.

Here, I did a simple experiment. Because the residual CNN is fully connected network, input size does not matter. I changed the residual network by removing maxpool in the second layer so that the spatial dimension is now reduced to 1/2 instead of 1/4. And I trained it with imagenet. And it looks like the performance is better in this case. It is straight forward because the computation cost for feature extraction becomes larger. But in this case, the number of parameters is the same. I think it is one example showing the model's capacity is also depending on computational cost not just the number of parameters.

Another example is, people sometimes use bilinear pooling at the end of the CNN for classification task. And this bilinear pooling is very expensive in computation cost but requires no additional parameters. When a bilinear pooling is attached to a conventional CNN, total number of parameter does not increase but computation cost does. (http://vis-www.cs.umass.edu/bcnn/docs/bcnn_iccv15.pdf)

In summary, my question is, "why many papers just compare models just based on the number of parameters? I think they also should consider computation cost". Could anyone give me any thoughts whether I am right or not?

$\endgroup$
0
$\begingroup$

Pragmatically, computation cost is harder to measure, since the cost varies depending on the hardware. Should everyone test their method on a standard machine? Would you measure time or memory? Would you distinguish between full precision ops and half precision ops? Should you test on CPU, GPU, TPU, etc?

Of course when it is relevant (robotics, autonomous vehicle, or other "real time" applications), people do indeed report details about computational cost. And outside of these applications, when two algorithms are comparable in performance, if one runs substantially faster, that is often mentioned and accepted as a real advantage.

Theoretically, preferring a model with fewer parameters can be justified by appealing to Occam's razor, or Solomonoff 's universal prior, which dominates every other computable prior. For computation time, there is Juergen Schmidhuber's "Speed Prior", although I don't think the mathematical justification for that is nearly as strong. Also it's definitely not as well known.

$\endgroup$
  • $\begingroup$ I understand your point and that is very thoughtful. But people sometimes use a more abstracted way of measuring computational cost. Probably my term 'computational cost' is somewhat ill-posed. We may count the number of multiplication and addition of one epoch and the total number of those calculations can be regarded as 'computation cost' I believe. In this sense, the architecture of actual hardware does not matter. What do you think about it? $\endgroup$ – Minkyu Choi Nov 2 '19 at 21:48
  • $\begingroup$ @MinkyuChoi yes, I do believe some papers report "flops" of their algorithm. But as I mentioned, from a philosophical and theoretical standpoint, preferring the "simplest explanation" == smallest model is on more solid ground than finding the "fastest model". $\endgroup$ – shimao Nov 3 '19 at 5:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.