I found a paper on that has a varient on normal auto-encoders (contractive) which for its gradient uses the following regularization penalty:

$$\left|\left|J_f(x)\right|\right|^2_F = \sum_{ij}{\left( \frac{\partial h_j(x)}{\partial x_i} \right)}^2$$

where $\left|\left|\cdot\right|\right|_F^2$ is the Frobenius norm, $h$ is the hidden units, and $x$ is the input. The paper also gives an alternative form (when a sigmoid is used for $f$) to the equation that looks like:

$$\left|\left|J_f(x)\right|\right|^2_F = \sum_{i=1}^{d_h}(h_i(1-h_i))^2\sum_{j=1}^{d_x}W^2_{ij}$$

Question 1

As per usual, no actual derivation is given to get the second form of the equation in the paper. I attempted to derive it myself, but would greatly appreciate it if someone could check my work and let me know what mistakes I might have made.

$$ a(x) = W^T x + b $$ $$ h(x) = f(a(x)) $$ $$ f(x) = \frac{1}{1+e^{-x}} $$

Then, using the chain rule:

$$ \frac{\partial h(x)}{\partial x} = \frac{\partial h(x)}{\partial a(x)} \frac{\partial a(x)}{\partial x} $$

Using the standard derivation of the sigmoid: $$ \frac{\partial h(x)}{\partial a(x)} = f(a(x))(1 - f(a(x))) = h(1- h) $$


$$ \frac{\partial a(x)}{\partial x} = W $$

Thus, finally:

$$ \frac{\partial h(x)}{\partial x} = h(1-h)W $$

Question 2

The problem I run into is that I'm not entirely sure of how to take the derivative of $\left(\frac{\partial h(x)}{\partial x}\right)^2$ with respect to $W$ in order to be able to get the gradient. If I square my previous result, I should get:

$$ \frac{\partial \left[h(1-h)\right]^2W^2}{\partial W} $$

which I would think would give:

$$ 2\left[h(1-h)\right]^2 W $$

And I'm having difficulty in interpreting this. I am a bit confused about whether h(1-h) is then an dot product. If so, does that just give me a scalar multiplied by W? If not, and it's an element-wise multiplication, then I think the dimensionality would be all wrong.

Or perhaps I did all of this incorrectly. Any help would be greatly appreciated!


when I interpret your equations correctly, the $W$ is supposed to be a matrix. This means that $a(x)$ is a vector and the then chain rule actually reads: $$\frac{\partial h_i}{\partial x_j} = \sum_k \frac{\partial h_i}{\partial a_k} \frac{\partial a_k}{\partial x_j}.$$

In matrix notation, the second term is $\frac{\partial a}{\partial x} = W^\top$. If I interpret your equations correctly, $f(x)$ is applied to each element of $a$ individually. Therefore, $\frac{\partial h_i}{\partial a_j} = \delta_{ij}h(a_j)(1-h(a_j))$ which means that it is a diagonal matrix with the term $h(a_j)(1-h(a_j))$ as the $j$th entry. Therefore $$\frac{\partial h_i}{\partial x_j} = \sum_k \delta_{ik} h(a_k)(1-h(a_k)) W_{kj} = h(a_i)(1-h(a_i)) W_{ij}.$$ Thus, $$\sum_{ij}\left(\frac{\partial h_i}{\partial x_j}\right)^2 =\sum_{ij} h(a_i)^2(1-h(a_i))^2 W_{ij}^2 = \sum_{i} h(a_i)^2(1-h(a_i))^2 \sum_{i} W_{ij}^2$$

Edit (second derivative): Since I am guessing that you want the derivative of the regularizer w.r.t to $W$, here is what I get (please check it numerically for correctness)

$$\frac{\partial}{\partial W_{kl}}\|J_{f}\|_{F}^{2}=\frac{\partial}{\partial W_{kl}}\sum_{i}h(a_{i})^{2}(1-h(a_{i}))^{2}\sum_{j}W_{ij}^{2}$$ $$=\sum_{i}\left(h(a_{i})^{2}(1-h(a_{i}))^{2}\sum_{j}\delta_{ik}\delta_{jl}2W_{ij}+\left(\sum_{j}W_{ij}^{2}\right)\frac{\partial}{\partial W_{kl}}h(a_{i})^{2}(1-h(a_{i}))^{2}\right)$$ $$=h(a_{k})^{2}(1-h(a_{k}))^{2}2W_{kl}+\sum_{i}\left(\sum_{j}W_{ij}^{2}\right)\left(2h(a_{i})h'(a_{i})\frac{\partial a_{i}}{\partial W_{kl}}\cdot(1-h(a_{i}))^{2}-h(a_{i})^{2}2(1-h(a_{i}))h'(a_{i})\frac{\partial a_{i}}{\partial W_{kl}}\right)$$ $$=2h(a_{k})^{2}(1-h(a_{k}))^{2}W_{kl}+\sum_{i}\left(\sum_{j}W_{ij}^{2}\right)2\left(h(a_{i})^{2}(1-h(a_{i}))^{3}\delta_{ik}x_{l}-h(a_{i})^{3}(1-h(a_{i}))^{2}\delta_{ik}x_{l}\right)$$ $$=2h(a_{k})^{2}(1-h(a_{k}))^{2}W_{kl}+\left(\sum_{j}W_{kj}^{2}\right)2h(a_{k})^{2}(1-h(a_{k}))^{3}x_{l}-\left(\sum_{j}W_{kj}^{2}\right)2h(a_{k})^{3}(1-h(a_{k}))^{2}x_{l}$$ $$=2h(a_{k})^{2}(1-h(a_{k}))^{2}\left(W_{kl}+x_{l}\left(1-2h(a_{k})\right)\left(\sum_{j}W_{kj}^{2}\right)\right)$$

Sory for the mess but I wanted you to be able to follow my calculations in case the numerical derivative turns out to be not correct.

  • $\begingroup$ I'm sure you're right, but I'm not sure I follow why h_i' should be the sum over h_i/a_k * a_k/x_j due to a(x) being a vector (Which it is) versus the non-summed version I have above. Also, any advice on what the final derivative looks like? Thanks! $\endgroup$
    – Ranon
    Aug 26 '11 at 16:56
  • $\begingroup$ Oh, okay, I'm guessing that's because the number is being taken so you have to sum the activation values. I was originally hoping to understand it without the summation because it's easier to see how to take its derivative.. $\endgroup$
    – Ranon
    Aug 26 '11 at 22:16
  • $\begingroup$ Hi Ranon, the summation is because of the chain rule: Say you have a function g from R to R^n and a function f from R^n to R, then the derivative of f(g(x)) w.r.t. x is sum_i df/dg_i dg_i/dx. I think the confusion might be that you look at your function h as from R to R but the way it is used it is really R^n to R where h is applied to each component of a. That's also the reason why the Jacobian is diagonal. $\endgroup$
    – fabee
    Aug 27 '11 at 15:20
  • $\begingroup$ What do you mean by final derivative? It should be all there, or did I forget anything? $\endgroup$
    – fabee
    Aug 27 '11 at 15:21
  • $\begingroup$ Thanks for the explanation, it's not something I've thought about before. By final derivative I meant the derivative of $\sum_{ij}(\frac{\partial h_i}{\partial x_j})^2$. $\endgroup$
    – Ranon
    Aug 27 '11 at 15:26

fabee's calculations are correct. You can also express the terms without assuming a sigmoid activation function.

To simplify the notation, let $h'_i = \frac{\partial h_i}{\partial a_i}$.

The term is: $$ \| J_f \|_F^2 = \sum_{i} \left( h'_i \right)^2 \sum_j W_{i,j}^2 $$

The gradient is: $$ \frac{\partial }{\partial W_{k,l}} \| J_f \|_F^2 = 2 (h'_k)^2 W_{kl} + \left(\sum_j W_{kj}^2\right) \frac{\partial (h'_k)^2}{\partial a_k} x_l $$

You can then use any activation functions. Let's see some examples.


$ h_k = \text{sig}(a_k) $

$ (h'_k)^2 = h_k^2 (1 - h_k)^2 $

$ \frac{\partial (h'_k)^2}{\partial a_k} = 2 h_k^2 (1 -h_k)^2 (1 - 2 h_k)$

You can verify that you get the same term as fabee's.


$ h_k = \ln (1 + \exp(a_k))$

$ (h'_k)^2 = \text{sig}(a_k)^2 $

$ \frac{\partial (h'_k)^2}{\partial a_k} = 2 \text{sig}(a_k)^2 - 2 \text{sig}(a_k)^3$


$ h_k = \tanh(a_k) $

$(h'_k)^2 = (1 - h_k^2)^2$

$ \frac{\partial (h'_k)^2}{\partial a_k} = -4 h_k (1 - h_k^2)^2 $


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.