# Interpreting neural networks in terms of generalized linear models?

in a 3-layer (input->hidden->output) feedforward network, the last layer can be interpreted as a generalized linear model (GLM) where the independent variables are the outputs of the hidden neurons.

Is it possible to interpret the coeffiecients of this GLM in some meaningful way? A major problem might be that the indpendent variables of the GLM are (transformed) linear combinations of the network input and thus are difficult to use for inference. However, I think that it should still be possible, isn't it?

Is there some existing literature which uses/demonstrates this approach?

• He is referring to the output layer of a neural network, which, on its own, is by all means a linear model. – Djib2011 Aug 3 '19 at 14:47

Yes you can interpret them as but, as you said, in most cases these variables have no real world interpretation because they are arbitrary linear combinations of the actual input variables.

However, in some networks, these variables aren't completely meaningless and can be interpreted. One such example are Convolutional Neural Networks (CNNs), which somewhat preserve the spatial information of their input due to their sparse connectivity.

An example where this idea is used is to produce the so called Class Activation Maps from a CNN. These essentially tell us "where a CNN is looking when it makes a prediction".

I will briefly explain the main concept, however it's best to read the actual paper which has

Imagine a simple CNN with four layers:

conv1 -> conv2 -> conv3 -> fc


The input of conv1 is an image that may have $$3$$ channels if it isn't grayscale. The output of the fully connected layer fc a the softmax output that classifies that image. Now, the output of the conv3 layer is another image with an arbitrary amount of channels (let's say $$100$$). Due to the fact that fc requires a 1D input, the output of conv3 is usually flattened before being fed to fc, so the actual CNN would look like this:

conv1 -> conv2 -> conv3 -> flatten -> fc


Let's take a look at the shapes of the tensors as they pass through the network. Let's say we have an input RGB image with a resolution of $$256 \times 256$$ and $$10$$ classes:

$$(256 \times 256 \times 3) \rightarrow (256 \times 256 \times C_1) \rightarrow (256 \times 256 \times C_2) \rightarrow (256 \times 256 \times 100) \rightarrow (6553600) \rightarrow 10$$

Note how the image preserved its spatial structure throughout the convolutional part of the network and lost this only to be classified by the fc layer. Now what does each convolution layer do? It pretty much extracts spatial features from the input image. The last layers of the network extract high-level features from the input image. The last convolutional layer produces $$100$$ feature maps for each input image.

The approach focuses replacing the flattening operation with a Global Average Pooling one, which averages each feature map to get a single number. The fc layer then learns to classify an image just from those $$100$$ numbers!

$$(256 \times 256 \times 3) \rightarrow \; ... \rightarrow (256 \times 256 \times 100) \rightarrow (100) \rightarrow 10$$

The each number however represents a feature map. By taking a specific class (out the $$10$$) and seeing its weights from the fc layer, we can tell which of the $$100$$ feature maps activates it the most. Furthermore, by applying the whole linear combination as designated by that class' weights we can produce what is referred to as the Class Activation Map (CAM) of that image. Through this we can tell what part of an image activates the CNN for a specific class. Furthermore, this is a very convenient way for a CNN to provide a reasoning for its decision, besides just the classification.