Total variation regularization in deep learning For current deep learning models, we can find basically two kinds of regularization on:


*

*Activation 

*Weights


The common $L_1$ and $L_2$ on weights can lead to a MAP problem where the regularization terms are from priors; On activation, it can lead to a sparse representation. 
But my questions are, 


*

*why it is very rare to see Total Variation (TV) regularization in deep learning (or maybe there are some?)? It is very useful in image denoising and reconstruction.

*what are the meaning if we add TV to activation or weight?

 A: Total variation (TV) is a meaningful measure for signals, where the neighboring elements have a meaningful relation. In images, for instance, this means that there is a relationship between pixels which are next/close to each other. TV measures how much these pixels differ from each other and measuring this makes sense, because you believe there is a meaningful relationship, some sort of "spatial continuity" in the image. Using TV as a regularization on an image then forces the neighboring pixels to have similar values except for a small number of positions, where it might make sudden jumps.
The weights/activations of a neural network (NN) do not have such a meaningful relationship between their neighbors. The weights in the matrix or in the kernels of convolutions are values on their own and do not necessarily follow a notion of "spatial continuity". You can arbitrarily reorder the weights and activations and the result will not change at all.
The essential difference is that L1 and L2 work directly on the values, but TV works on the gradients, and by taking the gradient you go to a domain where the values themselves are not important but rather their relationship to their neighbors.
I think this answers both your questions: 2) it does not make much sense in the framework of weight/activation regularization and 1) as it does not make sense, people do not use it.
