I think the learnable parameters are the set of all weights and biases in the neural network, but is it true in general?
$\begingroup$
$\endgroup$
6
-
$\begingroup$ The terms 'parameters' and 'weights' are used interchangeably and are learned during the training. Usually, the term 'parameters' is used predominantly in Statistics and 'weights' in the Machine Learning community; they are applicable to all types of models and not necessarily to only Neural Networks. $\endgroup$– NizamApr 1, 2020 at 6:45
-
$\begingroup$ @Nizam So, is it safe to refer to learnable parameters in deep learning as weights? $\endgroup$– AbdulkaderApr 1, 2020 at 6:57
-
$\begingroup$ Yes, works fine. $\endgroup$– NizamApr 1, 2020 at 6:58
-
$\begingroup$ Although, be aware that there may be more learnable parameters of the network than just weights. $\endgroup$– ajax2112Apr 1, 2020 at 7:34
-
$\begingroup$ @ajax2112 Do you mean biases? Though, I think it's not good to refer to weights and biases together as weights. $\endgroup$– AbdulkaderApr 1, 2020 at 7:42
|
Show 1 more comment
2 Answers
$\begingroup$
$\endgroup$
Learnable parameters usually means weights and biases, but there is more to it - the term encompasses anything that can be adjusted (i.e. learned) during training.
There are weights and biases in the bulk matrix computations; when thinking e.g. about a Conv2d operation with its number of filters and kernel size.
There are also parameters that are learned in layers such as BatchNorm, which are referred to as parameters (not just weights) perhaps because they are often discussed in terms of simple equations, where they look like coefficients in an equation. The parameters of this equation (the scale and shift) are learnt during training, so are included in the number of learnable parameters, but are a bit different from normal weights and biases. Have a look at the original paper (Ioffe, Szegedy, 2015) for more information/interpretation.
$\begingroup$
$\endgroup$
My previous answer was wrong. Right now my reputation is not enough to comment on this answer, so I'll write the addendum here.
The user n1k31t4 is right that there's more to learnable parameters than weights and biases. He proves his words with this paper. I read it and found the exact phrase that proves his words.
Page three, upper-right corner, below the formula. The paragraph was about the parameters of BatchNorm.
These parameters are learned along with the original model parameters
