I'm interested in applying several different types of regularization to neural nets and want to make sure I haven't learned the material incorrectly. I have successfully coded Weight Decay and Dropout, among others; in the case of Dropout it was a simple matter of setting a flag on a randomly selected set of connections; with Weight Decay it was a simple matter of multiplying the connection weights by a regularization term (typically slightly less than 1) after each learning step. I think applying L1, L2 and Tikhonov Regularization may turn out to be just as easy, once I clear up a few potential misunderstandings and lingering doubts I have after wading through some of the literature:
Can L1, L2 and Tikhonov calculations be appended as extra terms directly to a cost function, such as MSE? As I said above, it was easy for me to apply Weight Decay simply by applying a regularization multiplier to a whole set of weights; it would likewise be easy for me to code L1, L2 and Tikhonov by applying a new term to the existing cost functions I've already written. From the notation used in Martin Toma's reply to this post on the Data Science Forum, it appears that I can simply add a weighted sum of the absolute value of each weight (in the case of L1) or a weighted sum of each squared weight to the cost function. Nevertheless, the topic is actually logistic regression, which is only tangentially related to neural nets, plus the added terms are not explicitly depicted together with the MSE term.
If #1 is correct, is is possible to apply them in supervised learning situations where there are no cost functions?
I've gathered that L2 is a special case of Tikhonov Regularization, after reading a few research papers like Training with Noise is Equivalent to Tikhonov Regularization and Tikhonov Training of the CMAC Neural Network and of course skimming various Crossvalidated and Data Science Forum threads and the Wikipedia pages on Regularization and Tikhonov Regularization. On occasion though I've seen sources refer to them as if they were equivalent, so please correct me if L2 is a synonym for Tikhonov and not just a special case.
L2 is differentiable and therefore compatible with gradient descent, which is probably why it is the most popular type of Tikhonov Regularization. From the Wikipedia page, however, it appears that we can substitute other operations like difference and Fourier operators to derive other brands of Tikhonov Regularization that are not equivalent to L2. Are any of these alternate Tikhonov types applicable to neural nets? Are any of them actually used often in practice? I suspect that some of them would not be differentiable or suitable for gradient descent and therefore would be impractical.
If #4 is true, can someone post an example of a Tikhonov formula that is not equivalent to L2, yet would be useful in neural nets? I have a harder time following matrix notation than with the kind of standard summation operator notation used here or common neural notation used in sources like Patrick K. Simpson's old reference "Artificial Neural Systems: Foundations, Paradigms, Applications, and Implementations." Most of the explanations of Tikhonov I've seen to date, however, use matrix notation, which has made it more difficult for me to clear up my confusion.
There are arcane references in the literature to a "Tikhonov Matrix," which I can't seem to find any definitions of (although I've run across several unanswered questions scattered across the Internet about its meaning). An explanation of its meaning would be helpful.
I may be better off moving #6 to the Mathematics forum if it doesn't have a simple answer, but the other questions are all closely interrelated so I don't want to break them up into separate threads, if possible. Most of them may have simple yes-or-no answers, except for the request for a non-L2 version of Tikhonov Regularization. I don't necessarily need a top-to-bottom explanation of these three types of regularization or their use cases, since there are already some good threads and other resources for that; I just want to nip some possible misconceptions in the bud before I try to implement them in code. Thanks in advance.