# How to check whether a unit in a neural network is responsible for "coding" a certain input dimension

The title question seems a bit too vague but actually I have in my mind a very precise problem.

Let's say I present to a neural network an image: an object in a canvas. I vary the image only across 2 dimensions: size and position of the object in the canvas (let's say I use only one object).

I want to check whether there are some neurons selectively "coding" for size/position. E.g. neuron x is not active for size=10px, quite active for size=50px and very active for size=100px. However, changing object's position does not affect neuron x. Instead, it affects neuron y.

I understand that not all neurons are going to be this selective. Most likely, they are going to have a distributed representation and coding a bit of both. I would like to have a measure of that. Let's say a measure of "selective" coding.

Assume that I can get many samples across both dimensions. I am talking about a pre-trained convolutional network with Swish activation function (this can be changed) and with N fully connected layers at the end, but I want to perform the analysis only on the units of the fully connected layers.

Any idea?

Thanks.

• Could you be more specific with the kind of neural network you are thinking of using? Whether a single unit will activate or not will highly depend on the activation function, among other things. As an alternative, you could set up a small experiment with a 2 layer network, with each layer consisting of only 2 units and then test your hypothesis. Oct 5 '21 at 10:46
• I have updated the question with more info Oct 5 '21 at 11:22

$$I(X;Y) = KL(P_{XY}||P_XP_Y) \approx \sum_{x\in X'}\sum_{y\in Y'}p(x,y) \log\frac{p(x,y)}{p(x)p(y)}$$
where $$X'$$ and $$Y'$$ denote some discretization of your choice for the generally continuous variables $$X$$ and $$Y$$. The higher the mutual information will be, the more information about the controlled variable does the unit capture.
I purposefully used the $$\approx$$ sign because the precision of this estimate will depend on the discretization (histogram) you use. Coarser discretization will yield less precise estimate, whereas finer discretization will need quadratically more samples.