Reading many papers of deep-learning, I realized that the number of parameters and the computation costs are not exactly the same thing. When I read papers, people usually measure and compare model complexities based on the number of parameters. But I think computational cost should also be considered alongside with the number of parameters.
Taking a residual CNN network (ResNet) for example, this model reduces spatial dimension very fast and reduces computational cost very much by adopting stride=2 in the first conv layer and then max pool. Therefore, the spatial feature dimension becomes 1/4 of the input in the third layer.
Here, I did a simple experiment. Because the residual CNN is fully connected network, input size does not matter. I changed the residual network by removing maxpool in the second layer so that the spatial dimension is now reduced to 1/2 instead of 1/4. And I trained it with imagenet. And it looks like the performance is better in this case. It is straight forward because the computation cost for feature extraction becomes larger. But in this case, the number of parameters is the same. I think it is one example showing the model's capacity is also depending on computational cost not just the number of parameters.
Another example is, people sometimes use bilinear pooling at the end of the CNN for classification task. And this bilinear pooling is very expensive in computation cost but requires no additional parameters. When a bilinear pooling is attached to a conventional CNN, total number of parameter does not increase but computation cost does. (http://vis-www.cs.umass.edu/bcnn/docs/bcnn_iccv15.pdf)
In summary, my question is, "why many papers just compare models just based on the number of parameters? I think they also should consider computation cost". Could anyone give me any thoughts whether I am right or not?