16
$\begingroup$

How can I calculate the number of parameters in an artificial neural network in order to calculate its AIC?

$\endgroup$
2
  • $\begingroup$ This question seems perfectly clear to me. $\endgroup$ Sep 28 '15 at 17:16
  • $\begingroup$ You can use the command classifier.summary() from sklear class. $\endgroup$ Feb 23 '18 at 0:09
17
$\begingroup$

Every connection that is learned in a feedforward network is a parameter. Here is an image of a generic network from Wikipedia:

enter image description here

This network is fully connected, although networks don't have to be (e.g., designing a network with receptive fields improves edge detection in images). With a fully connected ANN, the number of connections is simply the sum of the product of the numbers of nodes in connected layers. In the image above, that is $(3\times 4) + (4\times 2) = 20$. That image does not show any bias nodes, but many ANNs do have them; if so, include the bias node in the total for that layer. More generally (e.g., if your ANN isn't fully connected), you can simply count the connections.

$\endgroup$
2
  • $\begingroup$ Connections can be non-unique (see ieeexplore.ieee.org/document/714176). Hence, is it okay to simply count the connections? Maybe we should distinguish between parameter and hyperparameter? $\endgroup$
    – Funkwecker
    Jul 24 '19 at 13:50
  • 1
    $\begingroup$ Total number of connections would be 26 if bias nodes were included. $\endgroup$
    – agcala
    Jan 9 '20 at 13:24
2
$\begingroup$

I would argue that this is an ill posed problem. Same as for many other machine learning algorithms, in neural networks it is hard to say what exactly would we count as a "parameter" when penalizing AIC. The point of AIC is to penalize the log-likelihood by the complexity of the model. In case of simple models, like linear, or logistic regression this is simple, as the number of regression parameters determines the complexity of the model. For simple feed-forward neural network this would also be the case, but consider that you can increase complexity of a neural network without increasing the number of parameters: you can use skip-connections, max-pooling, masking, weight normalization, etc., they all have no parameters. Moreover, what would you say about dropout, it "turns-off" parameters that are available for the network, so maybe somehow we should discount the number of parameters when using it? In case of complicated machine learning algorithms, the number of parameters is much less useful as a measure of model complexity.

To complicate it even more, in neural networks it was observed that bias-variance trade-off seems not to apply. The rationale behind using AIC is that "simpler" model is better, because it is more explainable and less prone to overfit. If, as it appears, neural networks do not have to be more prone to overfitting with increasing number of parameters, than it is disputable if penalizing for it makes sense.

$\endgroup$
1
$\begingroup$

For a MLP fully connected network you can use the following (Python) code:

def total_param(l=[]):
s=0
for i in range(len(l)-1):
    s=s+l[i]*l[i+1]+l[i+1]
return s

then if you have a network with the following layer configuration

input:  435
hidden: 166 
hidden: 103 
hidden:  64
output:  15

you just call the function with

total_param([435,166,103,64,15]) 
97208
$\endgroup$
0
$\begingroup$

Neural network is just a function of functions of functions ... (as dictated by the architecture of the model). If the resulting function can't be simplified then the total number of parameters (sum of all number of parameters from each nodes) in the model is the number you want for the AIC calculation.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.