The title question seems a bit too vague but actually I have in my mind a very precise problem.

Let's say I present to a neural network an image: an object in a canvas. I vary the image only across 2 dimensions: size and position of the object in the canvas (let's say I use only one object).

I want to check whether there are some neurons selectively "coding" for size/position. E.g. neuron x is not active for size=10px, quite active for size=50px and very active for size=100px. However, changing object's position does not affect neuron x. Instead, it affects neuron y.

I understand that not all neurons are going to be this selective. Most likely, they are going to have a distributed representation and coding a bit of both. I would like to have a measure of that. Let's say a measure of "selective" coding.

Assume that I can get many samples across both dimensions. I am talking about a pre-trained convolutional network with Swish activation function (this can be changed) and with N fully connected layers at the end, but I want to perform the analysis only on the units of the fully connected layers.

Any idea?


  • $\begingroup$ Could you be more specific with the kind of neural network you are thinking of using? Whether a single unit will activate or not will highly depend on the activation function, among other things. As an alternative, you could set up a small experiment with a 2 layer network, with each layer consisting of only 2 units and then test your hypothesis. $\endgroup$
    – mhdadk
    Oct 5, 2021 at 10:46
  • $\begingroup$ I have updated the question with more info $\endgroup$
    – Vaaal
    Oct 5, 2021 at 11:22

1 Answer 1


You could compute the mutual information between your controlled random variables, such as size and position, and the activation of the units:

$$ I(X;Y) = KL(P_{XY}||P_XP_Y) \approx \sum_{x\in X'}\sum_{y\in Y'}p(x,y) \log\frac{p(x,y)}{p(x)p(y)} $$

where $X'$ and $Y'$ denote some discretization of your choice for the generally continuous variables $X$ and $Y$. The higher the mutual information will be, the more information about the controlled variable does the unit capture.

I purposefully used the $\approx$ sign because the precision of this estimate will depend on the discretization (histogram) you use. Coarser discretization will yield less precise estimate, whereas finer discretization will need quadratically more samples.

  • $\begingroup$ Hi Jan, isn't this a measure of the independence of the response? Let's say that the mutual information is actually 0, so that p(x,y) = p(x)p(y), how can I say about whether the unit is coding for X or Y? $\endgroup$
    – Vaaal
    Oct 5, 2021 at 14:12
  • 1
    $\begingroup$ Sorry, I was not very clear. X would be a variable such as position and Y would be the unit response. Then Y encodes X if the mutual information is large. Conversely, if MI is zero, Y is independent of X and thus does not encode anything about it. $\endgroup$ Oct 5, 2021 at 14:15
  • $\begingroup$ Oh I see! Makes sense! Thanks! $\endgroup$
    – Vaaal
    Oct 5, 2021 at 14:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.