Should the sparsity penalty term be divided by the number on samples when computing an error term? I am studying a tutorial on Sparse Autoencoder.
In it, cost function $J(W,b)$ is modified by adding sparsity penalty term $\beta \sum_{j=1}^{s_2} \text{KL}(\rho || \hat{\rho}_j)$ (lets name it $\beta t$) to it.
As a result, it increases the error term $\delta ^{(2)}_j$ by 'derivative of sparsity penalty term' $\beta \left (  -\frac{\rho }{\hat{\rho}_j} + \frac{1-\rho }{1-\hat{\rho}_j}\right ) f'(z^{(2)}_j)$.
I checked the result, and, in my computation, derivative of sparsity penalty term is $m$ times smaller. The tutorial does not features the derivation process.
Is the tutorial correct and my computation wrong?
My computation
Since $\delta_j ^{(l)} = \frac{\partial J(W,b;x,y)}{\partial z_j^{(l)}}$, then $\delta_j ^{(2)}$ must be increased by $\frac{\partial \beta t }{\partial z_j^{(2)}}$. That is, using the chain rule, we get:
$\delta ^{(2)}_j \text{  +=  } \frac{\partial \beta t }{\partial z_j^{(2)}} = \beta \frac{\partial t}{\partial \hat{\rho}_j}\frac{\partial \hat{\rho}_j}{\partial a^{(2)}_j}\frac{\partial a^{(2)}_j}{\partial z^{(2)}_j} = \beta \frac{\partial t}{\partial \hat{\rho}_j}\frac{\partial \hat{\rho}_j}{\partial a^{(2)}_j(x)}\frac{\partial a^{(2)}_j(x)}{\partial z^{(2)}_j(x)}$
Note that $a^{(2)}_j$ and $z^{(2)}_j$ refer to the specific $x, y$, not to the whole training set.
$\frac{\partial t}{\partial \hat{\rho}_j} = -\frac{\rho }{\hat{\rho}_j} + \frac{1-\rho }{1-\hat{\rho}_j}$ (Same as in the tutorial)
$\frac{\partial \hat{\rho}_j}{\partial a^{(2)}_j(x)} = \frac{\partial \frac{1}{m}\sum_{i=1}^m a^{(2)}_j(x^{(i)})}{\partial a^{(2)}_j(x)} = \frac{1}{m}\frac{\partial \sum_{i=1}^m a^{(2)}_j(x^{(i)})}{\partial a^{(2)}_j(x)} = \frac{1}{m}1 = \frac{1}{m}$ (Different. The formula in the tutorial does not have this normalizer. Multiplier "1" means that that only one of summands in nominator is the same as in denominator)
$\frac{\partial a^{(2)}_j(x)}{\partial z^{(2)}_j(x)}=f'(z^{(2)}_j)$ (Same)
 A: I think it's because you have the partial derivative of the sum = 1, but the partial derivative of the sum is the sum of the partial derivatives = m since each partial derivative is 1, and you're summing m times. And then m/m = 1.
A: The formula in the tutorial is correct. My computation is wrong.
I made a mistake right at the beginning of my computation, where I presumed that $\delta_j{(l)}$ must be increased by $\frac{\partial \beta t }{\partial z_j^{(2)}}$. $\delta_j^{(l)}$ referes to a specific sample $(x,y)$, but $\beta t$ refers to the final (total) cost and uses all samples.
My mistake was that I assumed that $\delta_j^{(l)}$ is always defined as $\frac{\partial J(W,b;x,y)}{\partial z_j^{(l)}}$. In actuality, $\delta_j^{(l)}$ is just an abreviation for some derivative to make computation of $\frac{\partial J(W,b)}{\partial W_{kj}^{(l)}}$ and $\frac{\partial J(W,b)}{\partial b_j^{(l)}}$ easier.
Recall given in the tutorial (without regularization): $\frac{\partial J(W,b)}{\partial W_{kj}^{(l)}} = \frac{1}{m} \sum_{i=1}^{m} \delta^{(l+1)}_k a^{(l)}_j(x^{(i)})$
The correct way of finding out updated value for $\delta_j^{(2)}$ would be to compute $\frac{\partial \beta t }{\partial W_{kj}^{(1)}} = ... = \frac{1}{m} \sum_{i} \beta \left ( -\frac{\rho }{\hat{\rho}_k} + \frac{1-\rho }{1-\hat{\rho}_k} \right ) f'(z^{(2)}_k(x^{(i)})) a^{(1)}_j(x^{(i)})$
(If someone wants, I can describe computation in more details)
Hence, for convenience, $\delta ^{(2)}_j \text{  +=  } \beta \left ( -\frac{\rho }{\hat{\rho}_k} + \frac{1-\rho }{1-\hat{\rho}_k} \right ) f'(z^{(2)}_k(x^{(i)}))$
