4
$\begingroup$

I have two neural networks. If I take only weights (the activation functions for both are the same), is there a way to tell the percent similarity of these two networks?

$\endgroup$
1
+50
$\begingroup$

You can try to estimate the similarity between two units, using their weights and the similarity matrix between units from which they receive inputs. This would lead to a process similar to back-propagation, but going from bottom to top. At first you estimate the similarity between each pair of 1st layer units, then you estimate the similarity between the second layer units, and so on.

At the end you will not have an exact answer. The best you can hope is to make an estimation which is not too far away from the truth.

$\endgroup$
1
$\begingroup$

You could consider the whole number of neurons as your population, and by checking the weights of the neurons, through this formulation say how similar these two are:

similarity = (number of neurons with same weight values) / (total number of neurons) * 100

As you mentioned, we are considering that the number of neurons, activation functions, ... are all the same. Of course, there is no straight-forward method to do this, and what I said here is only one possible way.

$\endgroup$
1
$\begingroup$

Assuming you are dealing with typical feed forward networks, which are fully connected between neighbouring layers: the order of neurons within layers could be changed without changing the networks behaviour. As different runs of training a network can easily end up with such configurations (different neurons in the same layer could learn the same information at different positions in different runs), such networks should be considered equal - from which I assume the similarity measure should consider such networks equal as well.

One hypothetical option would be to determine the optimal combination of "matching partners" for each neuron in one layer of the first network with the neurons of the same layer in the second network. Each comparison of two neurons could yield a scalar similarity value. By determining the optimal combination of matching partners per layer, the resulting vector of similarities for those neurons could be used to come up a scalar value for the similarity of the whole layer - across the two networks. Obtaining this similarity for all N+1 layers would leave you with N weights, which you could use to compute a scalar similarity. A downside of this is that this will likely become very complex with multiple layers, because the order of neurons in the previous/successive layers might be scrambled as well.

A probably better option would be to consider each path from each input neuron to each output neuron, and compare those paths between the two networks. Again, the optimal "matching partner" for each path could be determined (so that all paths have a "matching partner" in the second network in the end). This could yield a scalar similarity value per path, from which the overall similarity of the network could be calculated again. The advantage of this is that it would be completely unaffected by changes of orders of neurons, but the big downside would be that the number of paths grows exponentially with the amount of neurons and layers.

$\endgroup$
0
$\begingroup$

That's a good question. I have never done that, but I would apply a Principal Component Analysis on the weight space and then comparing the first N PCs. I'd use this approach to deal with the fact that two NNs can have identical "behaviour" (same input-output) with different weights.

$\endgroup$
2
  • $\begingroup$ How could be PCA performed on weights? I také them as a vector, but PCA is usually done on matrices to reduce dimensionality... $\endgroup$ – Martin Perry May 19 '16 at 14:10
  • $\begingroup$ With a feed forward network (fully connected within neighbouring layers), the order of neurons should be shuffled without changing the network's behaviour, hence such network should probably be considered equal (the networks could easily learn similar features = similar weights on different neurons in different runs). As PCA does not consider such changes in order of data it might not be the way to go. $\endgroup$ – geekoverdose May 19 '16 at 14:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.