The dropout proposed by Hinton is said to prevent co-adaptation. My question is how can I measure the co-adaptation that occurs in a multi-layer perceptron that does not use a drop out?
$\begingroup$
$\endgroup$
0
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
0
I would argue that non-identity covariance in the hidden layer activations is one form of "co-adaptation."
To compute the hidden layer covariance, just take your trained MLP and find a stack of data. Run the data through the MLP until you compute the activations for the hidden layer of interest: $$ H = \sigma(WX + B) $$ for an MLP with one hidden layer and activation function $\sigma$. Once you have these values, just compute their covariance: $$ S = \frac{1}{n}(H-\bar{H})(H-\bar{H})^\top. $$
To get a single numeric metric from the hidden feature covariance, you could use lots of different things, but I think it's convenient to compute the norm of the covariance less the identity matrix (i.e., the amount of "non-identity stuff" in the observed covariance matrix): $$ \ell = \|S-I\|_F. $$
Using this method you could train up different MLPs and test whether dropout has an impact on the hidden layer activations' tendencies to covary.
Of course, there are many different ways to measure "co-adaptation" (or "dependence"); the covariance only measures dependence up through the second statistical moment. You could also do things like measure the kurtosis of the hidden activations, which would get you a different metric, but also one that measures co-adaptation in some sense---much in the same way that ICA models pursue independent components through several different losses.
Independence is a really interesting topic in these types of models. There are lots of good papers out there, particularly in the ICA area, but for a great discussion of the different types of independence see Bell & Sejnowski (1997) "The 'Independent Components' of Natural Scenes are Edge Filters" (especially the discussion around Figure 2).
