Is the inference time directly proportional to both the number of operations in a network as well as the number of parameters? Or is it directly proportional to no. of operations and indirectly proportional to no. of parameters?
$\begingroup$
$\endgroup$
1
-
$\begingroup$ @shimao already answered it. Just commenting: the time would depend on the network architecture, but also on the software & hardware, e.g. in some cases they may utilize better the hardware parallelizing things, etc. The only reasonable way to learn about it is to run it and measure the time. $\endgroup$– TimCommented Jun 22, 2021 at 11:54
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
There's no reason time should be proportional to the number of parameters, for example, you could imagine a fully connected one layer network which computes $y = \sigma(w^Tx)$ and a simple RNN which computes $y_0 = 0, y_i = \sigma(y_{i-1}+wx_i)$. Both involve roughly $n$ multiplications / additions, but the RNN only has a single parameter, and the fully connected network has $n$.
As for whether time is proportional to number of operations, that depends on what you mean by "time". Sometimes when computer scientists talk about time, we're interested in some theoretical measure of how long it takes to compute something, and this measure is usually defined as "number of operations". So it's tautological that the run time is proportional to the number of operations.
On the other hand, if you care about runtime in real life, then there's no straightforward relationship between number of operations and time. The fully connected network described above can be very easily parallelized and the dot product effectively done in a single "cycle", whereas the RNN output needs to be computed sequentially, taking $n$ cycles.
