0
$\begingroup$

we know that Sparse encoding is to minimize the objective function: $$\sum\limits_{n=1}^N\Big(||x_n - Az_n||^2 + \lambda\rho(z_n)\Big).$$ Here

  1. $A = [a_1,\cdots,a_M]$ is overcomplete Dictionary;

  2. $z = (z^1,\cdots,z^M):\ x = Az$ is Encoding;

  3. $\rho(z_n)$ is sparsity function.

And we need to train both $(A,z_n)$

Sparse autoencoding is to minimize the objective function:

$$\sum\limits_{n=1}^N\Big(||x_n - \phi\cdot\psi(x_n)||^2 + \lambda\rho(h_n)\Big).$$ here $\psi(x) =h = \sigma(Ax); \phi(h) = x = \sigma'(A'x')$ are the non-linear activation functions. We need to train $A,A'.$

I am not sure if we ignore the non-linear activation functions $\sigma/\sigma',$ is Sparse encoding a special case of Sparse autoencoding?

Since if we represent PCA as encoding-decoding error: $$\sum\limits_{n=1}^N\left(||x_n-AA^Tx_n||^2\right)$$ it is exactly the linear case of autoencoding.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.