Skip to main content

# Questions tagged [tikhonov-regularization]

Tikhonov regularization, named for Andrey Tikhonov, is the most commonly used method of regularization of ill-posed problems, and is a generalization of ridge regression.

30 questions
Filter by
Sorted by
Tagged with
3 votes
0 answers
41 views

### L1/L2 regularization in neural nets

For linear regression, after doing L1/L2 regularization one can compute a closed form solution for the weights in nice cases. From here, one gets the intuition where: L2 regularization shrinks ...
• 275
2 votes
1 answer
145 views

### What are a priori advantages of Lasso regularization for linear regression models?

What are a priori advantages of Lasso regularization for linear regression models, over many other heuristically-justifiable methods that both regularize the problem and perform variable selection? ...
• 31
3 votes
0 answers
54 views

### What's the unified definition of Tikhonov regularization

I met with "Tikhonov regularization" in two textbooks. The first is "Pattern Recognition and Machine Learning" by Christopher M. Bishop. In page 267 of his book, the regularized ...
• 333
2 votes
0 answers
141 views

### Effect of Regularization on bias and variance

I have already seen several related questions: 1, 2, 3, 4, 5. The answer to 1 states Regularization attemts to reduce the variance of the estimator by simplifying it, something that will increase the ...
• 145
1 vote
0 answers
27 views

### simultaneous parameter and variance estimation in statistical inverse problem

Background: I'm working on a geophysical inverse problem and interested in regularization parameter selection. Just by way of explanation, we're trying to estimate the friction underneath a flowing ...
1 vote
1 answer
136 views

• 31
0 votes
1 answer
1k views

### Derivation of ridge regression for multi-value-target vectors

At university, I learned with these slides about ridge regression and its derivation with the assumption that the target- and predicted values have the dimensions $1\times1$. However, now I need to ...
• 145
1 vote
0 answers
50 views

### non-linear tikhonov/ridge regularization?

For traditional ridge regression, the loss function is $loss\_function = ||A\mathbf{x}-\mathbf{b}||_2^2 + ||\Gamma\mathbf{x}||_2^2$ https://en.wikipedia.org/wiki/Tikhonov_regularization Is there a ...
2 votes
0 answers
344 views

### $L^2$ Regularization and Hessian Matrix [duplicate]

In the second paragraph it is mentioned that eigenvector of $H$ is rescaled by a factor of $\frac{\lambda_i} {\lambda_i +\alpha}$ What exactly meant by that ?
2 votes
0 answers
441 views

### Difference Between Two Tikhonov Regularization Schemes

For the solution of $Ax = b$, where $A$ is a square matrix, what is the difference between these two regularized solutions: $x = (A + \alpha I)^{-1}b$ -- coressponding to eq.3 below \$x = (A^TA + \...
• 4,634
30 votes
3 answers
3k views

### The origin of the term "regularization"

When I introduce concepts to my students, I often find it fun to tell them where the terminology originates ("regression", for example, is a term with an interesting origin). I haven't been able to ...
2 votes
0 answers
272 views

### Finite difference based regularization matrix

I've just started reading about Tikhonov Regularization. Would someone please help with a simple numerical example of a case where regularization matrix of the form of a second order finite difference ...
• 1,419
5 votes
1 answer
3k views

### Applying L1, L2 and Tikhonov Regularization to Neural Nets: Possible Misconceptions

I'm interested in applying several different types of regularization to neural nets and want to make sure I haven't learned the material incorrectly. I have successfully coded Weight Decay and Dropout,...
• 1,299
40 votes
2 answers
19k views

### Is Tikhonov regularization the same as Ridge Regression?

Tikhonov regularization and ridge regression are terms often used as if they were identical. Is it possible to specify exactly what the difference is?
• 13.2k
3 votes
1 answer
3k views

### Tikhonov regularization in the context of deconvolution

I came across "Tikhonov regularization" and I have bare knowledge on it. It seems that it is a type of regularization that is important for deconvolution. Are there any good resources and examples? ...
• 13.1k
1 vote
0 answers
91 views

### Sequential Least Squares for Tikhonov Regularization [duplicate]

Given a Weighted Linear Least Squares problem where the cost function is given by: $$J = { \left( x - H \Theta \right) }^{T} {C}^{-1} { \left( x - H \Theta \right) }$$ There is a Sequential ...
• 1,151