we know that Sparse encoding is to minimize the objective function: $$\sum\limits_{n=1}^N\Big(||x_n - Az_n||^2 + \lambda\rho(z_n)\Big).$$ Here

  1. $A = [a_1,\cdots,a_M]$ is overcomplete Dictionary;

  2. $z = (z^1,\cdots,z^M):\ x = Az$ is Encoding;

  3. $\rho(z_n)$ is sparsity function.

And we need to train both $(A,z_n)$

Sparse autoencoding is to minimize the objective function:

$$\sum\limits_{n=1}^N\Big(||x_n - \phi\cdot\psi(x_n)||^2 + \lambda\rho(h_n)\Big).$$ here $\psi(x) =h = \sigma(Ax); \phi(h) = x = \sigma'(A'x')$ are the non-linear activation functions. We need to train $A,A'.$

I am not sure if we ignore the non-linear activation functions $\sigma/\sigma',$ is Sparse encoding a special case of Sparse autoencoding?

Since if we represent PCA as encoding-decoding error: $$\sum\limits_{n=1}^N\left(||x_n-AA^Tx_n||^2\right)$$ it is exactly the linear case of autoencoding.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.