How can I calculate the number of parameters in an artificial neural network in order to calculate its AIC?
$\begingroup$
$\endgroup$
2
2 Answers
$\begingroup$
$\endgroup$
3
Every connection that is learned in a feedforward network is a parameter. Here is an image of a generic network from Wikipedia:
This network is fully connected, although networks don't have to be (e.g., designing a network with receptive fields improves edge detection in images). With a fully connected ANN, the number of connections is simply the sum of the product of the numbers of nodes in connected layers. In the image above, that is $(3\times 4) + (4\times 2) = 20$. That image does not show any bias nodes, but many ANNs do have them; if so, include the bias node in the total for that layer. More generally (e.g., if your ANN isn't fully connected), you can simply count the connections.
answered Sep 28, 2015 at 17:59
-
$\begingroup$ Connections can be non-unique (see ieeexplore.ieee.org/document/714176). Hence, is it okay to simply count the connections? Maybe we should distinguish between parameter and hyperparameter? $\endgroup$ Commented Jul 24, 2019 at 13:50
-
2$\begingroup$ Total number of connections would be 26 if bias nodes were included. $\endgroup$– agcalaCommented Jan 9, 2020 at 13:24
-
$\begingroup$ To expand on the comment of @agcala including the bias, the computation for the number of parameters is $s_{j+1}\times{\left(s_{j}+1\right)}$ where $j$ is the index of a layer and $s$ the number of nodes in that layer. So, in this example, we have $\left(4\times{4}\right)+\left(2\times{5}\right)=26$. $\endgroup$ Commented Jan 29, 2022 at 20:55
$\begingroup$
$\endgroup$
I would argue that this is an ill posed problem. Same as for many other machine learning algorithms, in neural networks it is hard to say what exactly would we count as a "parameter" when penalizing AIC. The point of AIC is to penalize the log-likelihood by the complexity of the model. In case of simple models, like linear, or logistic regression this is simple, as the number of regression parameters determines the complexity of the model. For simple feed-forward neural network this would also be the case, but consider that you can increase complexity of a neural network without increasing the number of parameters: you can use skip-connections, max-pooling, masking, weight normalization, etc., they all have no parameters. Moreover, what would you say about dropout, it "turns-off" parameters that are available for the network, so maybe somehow we should discount the number of parameters when using it? In case of complicated machine learning algorithms, the number of parameters is much less useful as a measure of model complexity.
To complicate it even more, in neural networks it was observed that bias-variance trade-off seems not to apply. The rationale behind using AIC is that "simpler" model is better, because it is more explainable and less prone to overfit. If, as it appears, neural networks do not have to be more prone to overfitting with increasing number of parameters, than it is disputable if penalizing for it makes sense.
classifier.summary()fromsklearclass. $\endgroup$