Imagine the following scenario:
Given a signal
x
, you pass it through a bank of convolutional filters to get a feature mapz
(the actual operation is a cross-correlation).Then you apply a nonlinearity to (potentially sparsify) the features, and
deconvolve using the exact same filter bank, to reconstruct
x'
with the objective of finding a filter bank such that the difference betweenx'
andx
by your distance metric of choice is minimized.
This corresponds to a very basic convolutional (sparse) autoencoder with tied weights.
In this context, the paper Convolutional Sparse Autoencoders for Image Classification (Luo, Yang, Zang) says the following in the introduction:
The weights of the encoder and decoder can be tied to force features to diverge as much as possible.
Further in the paper, they build up on that idea. I can understand the intuition behind that (coding efficiency, model capacity, etc), but I am interested in a more solid mathematical explanation for that. Unfortunately I couldn't find any references in the paper itself. Could you provide such references, and optionally a proof for that? You may assume that the objective is pursued via any form of Gradient Descent, Basis Pursuit, or similar.
Thanks in advance!
Andres