Should the sparsity penalty term be divided by the number on samples when computing an error term?

Question

I am studying a tutorial on Sparse Autoencoder.

In it, cost function $J(W,b)$ is modified by adding sparsity penalty term $\beta \sum_{j=1}^{s_2} \text{KL}(\rho || \hat{\rho}_j)$ (lets name it $\beta t$) to it.

As a result, it increases the error term $\delta ^{(2)}_j$ by 'derivative of sparsity penalty term' $\beta \left ( -\frac{\rho }{\hat{\rho}_j} + \frac{1-\rho }{1-\hat{\rho}_j}\right ) f'(z^{(2)}_j)$.

I checked the result, and, in my computation, derivative of sparsity penalty term is $m$ times smaller. The tutorial does not features the derivation process.

Is the tutorial correct and my computation wrong?

My computation

Since $\delta_j ^{(l)} = \frac{\partial J(W,b;x,y)}{\partial z_j^{(l)}}$, then $\delta_j ^{(2)}$ must be increased by $\frac{\partial \beta t }{\partial z_j^{(2)}}$. That is, using the chain rule, we get:

$\delta ^{(2)}_j \text{ += } \frac{\partial \beta t }{\partial z_j^{(2)}} = \beta \frac{\partial t}{\partial \hat{\rho}_j}\frac{\partial \hat{\rho}_j}{\partial a^{(2)}_j}\frac{\partial a^{(2)}_j}{\partial z^{(2)}_j} = \beta \frac{\partial t}{\partial \hat{\rho}_j}\frac{\partial \hat{\rho}_j}{\partial a^{(2)}_j(x)}\frac{\partial a^{(2)}_j(x)}{\partial z^{(2)}_j(x)}$

Note that $a^{(2)}_j$ and $z^{(2)}_j$ refer to the specific $x, y$, not to the whole training set.

$\frac{\partial t}{\partial \hat{\rho}_j} = -\frac{\rho }{\hat{\rho}_j} + \frac{1-\rho }{1-\hat{\rho}_j}$ (Same as in the tutorial)

$\frac{\partial \hat{\rho}_j}{\partial a^{(2)}_j(x)} = \frac{\partial \frac{1}{m}\sum_{i=1}^m a^{(2)}_j(x^{(i)})}{\partial a^{(2)}_j(x)} = \frac{1}{m}\frac{\partial \sum_{i=1}^m a^{(2)}_j(x^{(i)})}{\partial a^{(2)}_j(x)} = \frac{1}{m}1 = \frac{1}{m}$ (Different. The formula in the tutorial does not have this normalizer. Multiplier "1" means that that only one of summands in nominator is the same as in denominator)

$\frac{\partial a^{(2)}_j(x)}{\partial z^{(2)}_j(x)}=f'(z^{(2)}_j)$ (Same)

Answer 1

I think it's because you have the partial derivative of the sum = 1, but the partial derivative of the sum is the sum of the partial derivatives = m since each partial derivative is 1, and you're summing m times. And then m/m = 1.

Answer 2

The formula in the tutorial is correct. My computation is wrong.

I made a mistake right at the beginning of my computation, where I presumed that $\delta_j{(l)}$ must be increased by $\frac{\partial \beta t }{\partial z_j^{(2)}}$. $\delta_j^{(l)}$ referes to a specific sample $(x,y)$, but $\beta t$ refers to the final (total) cost and uses all samples.

My mistake was that I assumed that $\delta_j^{(l)}$ is always defined as $\frac{\partial J(W,b;x,y)}{\partial z_j^{(l)}}$. In actuality, $\delta_j^{(l)}$ is just an abreviation for some derivative to make computation of $\frac{\partial J(W,b)}{\partial W_{kj}^{(l)}}$ and $\frac{\partial J(W,b)}{\partial b_j^{(l)}}$ easier.

Recall given in the tutorial (without regularization): $\frac{\partial J(W,b)}{\partial W_{kj}^{(l)}} = \frac{1}{m} \sum_{i=1}^{m} \delta^{(l+1)}_k a^{(l)}_j(x^{(i)})$

The correct way of finding out updated value for $\delta_j^{(2)}$ would be to compute $\frac{\partial \beta t }{\partial W_{kj}^{(1)}} = ... = \frac{1}{m} \sum_{i} \beta \left ( -\frac{\rho }{\hat{\rho}_k} + \frac{1-\rho }{1-\hat{\rho}_k} \right ) f'(z^{(2)}_k(x^{(i)})) a^{(1)}_j(x^{(i)})$

(If someone wants, I can describe computation in more details)

Hence, for convenience, $\delta ^{(2)}_j \text{ += } \beta \left ( -\frac{\rho }{\hat{\rho}_k} + \frac{1-\rho }{1-\hat{\rho}_k} \right ) f'(z^{(2)}_k(x^{(i)}))$