Questions tagged [tikhonov-regularization]
Tikhonov regularization, named for Andrey Tikhonov, is the most commonly used method of regularization of ill-posed problems, and is a generalization of ridge regression.
28
questions
2
votes
0
answers
45
views
What's the unified definition of Tikhonov regularization
I met with "Tikhonov regularization" in two textbooks. The first is "Pattern Recognition and Machine Learning" by Christopher M. Bishop. In page 267 of his book, the regularized ...
- 303
2
votes
0
answers
81
views
Effect of Regularization on bias and variance
I have already seen several related questions: 1, 2, 3, 4, 5.
The answer to 1 states
Regularization attemts to reduce the variance of the estimator by simplifying it, something that will increase the ...
- 143
1
vote
0
answers
23
views
simultaneous parameter and variance estimation in statistical inverse problem
Background: I'm working on a geophysical inverse problem and interested in regularization parameter selection.
Just by way of explanation, we're trying to estimate the friction underneath a flowing ...
- 143
1
vote
1
answer
115
views
Iterative generalized ridge regression
I am looking for some references. Assume I have a series of observable input/output pairs $(y_t, X_t)$ for which I assume the following relations to hold:
$$\beta_t\text{ are i.i.d. }\sim N(\bar{\beta}...
- 35
0
votes
0
answers
246
views
Regularization Terms in MLE
Can you add regularization terms to any likelihood function you're trying to maximize? (e.g. L2/Tikhonov, Lasso terms)
I'm used to seeing this done with simple quadratic loss functions (e.g. for ...
- 117
1
vote
0
answers
52
views
effect of multiplying by Tikhonov regularization factor after an inverse?
I came across a repository which uses Tikhonov regularization to compute an inverse, but then in the inference step they multiply by the Tikhonov factor again...
Compute $\Phi\Phi^T$
Compute the ...
- 730
3
votes
1
answer
393
views
Tikhonov regularization equivalence to adding random noise
In Pattern Recognition and Machine Learning Ch 5.5.5 Bishop derives a regulariser for neural networks that is equivalent to the tangent propagation regulariser (a regulariser that is invariant to ...
- 259
1
vote
0
answers
51
views
Explain L1 vs. L2 regularization difference using the scientific mindset [duplicate]
I need to have job interview soon, one of the questions may be L1. vs L2 regularization. Yann LeCun explained best to my knowledge the difference between L1 and L2 regularization.
L1 or Lasso: ...
- 31
0
votes
1
answer
457
views
Simple explanation of Takeuchi’s information criterion?
Takeuchi’s information criterion is said to be the generalization of AIC to misspecified models. That publication presents DEGREES OF FREEDOM FOR NONLINEAR LEAST SQUARES ESTIMATION. From that source:
...
- 12.5k
1
vote
0
answers
345
views
Understanding how regularization penalizes higher order features
I am trying to understand how exactly L2 or ridge regression penalize higher order polynomial features to make fitting curve smoother. I read this article about regularization. I contemplated and I ...
- 205
3
votes
1
answer
288
views
Ridge regularization - intuition behind $\lambda$
I have seen many similar questions and I understand that $\lambda$ is some kind of a tuning parameter that decides how much we want to penalize the flexibility of our model. In other words $\lambda$ ...
- 604
3
votes
0
answers
539
views
Entropy regularization versus L2 norm regularization?
In multiple regression problems, the decision variable, coefficients $\beta$, can be regularized by its L2 (Euclidean) norm, shown below (in the second term) for least squares regression. This type of ...
- 3,481
1
vote
1
answer
201
views
Is there any need for regularization in an overdetermined multiple regression problerm?
Supposed I have a small number of features, say 4 or 5, and I have hundreds of data points. That is, I am in an over-determined situation.
Is there any benefit to using regularization in this setting ...
- 883
0
votes
0
answers
285
views
How does regularization affect the offset term and weights for OLS?
I have a couple of questions regarding regularization that I am confused on, and searches on this forum doesn't seem to answer my specific questions.
When we regularize with $\lambda ||w||^2$ or $\...
- 281
1
vote
0
answers
104
views
Regularization and multicollinearity
Multicollinearity/collinearity, from what I understand, occurs when 2 or more predictor/independent variables (variates) are strongly linearly dependent. This leads to overfitting, and we could use ...
- 1,067
2
votes
1
answer
1k
views
Trace of the Hat Matrix in Ridge Regression
Generally, I know that the trace of the hat matrix ($H$) is equal to the rank of H since it is an orthogonal projection. If I wanted to show the trace of $H$ in ridge regression, would I be able to ...
- 504
2
votes
0
answers
34
views
Why we only use L1 and L2 in penalized regression [duplicate]
I understand we do not want to go lower than 1 because the problem is no longer convex. But why not use L3 for example? Is there any reason why L2 is so popular but not higher norm?
- 141
3
votes
3
answers
2k
views
Why does $l_2$ norm regularization not have a square root?
Specifically talking about Ridge Regression's cost function since Ridge Regression is based off of the $l_2$ norm. We should expect the cost function to be:
$$J(\theta)=MSE(\theta) + \alpha\sqrt{\...
- 31
0
votes
1
answer
927
views
Derivation of ridge regression for multi-value-target vectors
At university, I learned with these slides about ridge regression and its derivation with the assumption that the target- and predicted values have the dimensions $1\times1$.
However, now I need to ...
- 145
1
vote
0
answers
50
views
non-linear tikhonov/ridge regularization?
For traditional ridge regression, the loss function is
$loss\_function = ||A\mathbf{x}-\mathbf{b}||_2^2 + ||\Gamma\mathbf{x}||_2^2$
https://en.wikipedia.org/wiki/Tikhonov_regularization
Is there a ...
2
votes
0
answers
342
views
$L^2$ Regularization and Hessian Matrix [duplicate]
In the second paragraph it is mentioned that eigenvector of $H$ is rescaled by a factor of $\frac{\lambda_i} {\lambda_i +\alpha}$ What exactly meant by that ?
- 133
2
votes
0
answers
419
views
Difference Between Two Tikhonov Regularization Schemes
For the solution of $Ax = b$, where $A$ is a square matrix, what is the difference between these two regularized solutions:
$x = (A + \alpha I)^{-1}b$ -- coressponding to eq.3 below
$x = (A^TA + \...
- 4,106
30
votes
3
answers
2k
views
The origin of the term "regularization"
When I introduce concepts to my students, I often find it fun to tell them where the terminology originates ("regression", for example, is a term with an interesting origin). I haven't been able to ...
- 35k
2
votes
0
answers
254
views
Finite difference based regularization matrix
I've just started reading about Tikhonov Regularization. Would someone please help with a simple numerical example of a case where regularization matrix of the form of a second order finite difference ...
- 1,379
5
votes
1
answer
2k
views
Applying L1, L2 and Tikhonov Regularization to Neural Nets: Possible Misconceptions
I'm interested in applying several different types of regularization to neural nets and want to make sure I haven't learned the material incorrectly. I have successfully coded Weight Decay and Dropout,...
- 1,201
39
votes
2
answers
18k
views
Is Tikhonov regularization the same as Ridge Regression?
Tikhonov regularization and ridge regression are terms often used as if they were identical. Is it possible to specify exactly what the difference is?
- 12.5k
3
votes
1
answer
2k
views
Tikhonov regularization in the context of deconvolution
I came across "Tikhonov regularization" and I have bare knowledge on it. It seems that it is a type of regularization that is important for deconvolution. Are there any good resources and examples? ...
- 12.5k
1
vote
0
answers
88
views
Sequential Least Squares for Tikhonov Regularization [duplicate]
Given a Weighted Linear Least Squares problem where the cost function is given by:
$$ J = { \left( x - H \Theta \right) }^{T} {C}^{-1} { \left( x - H \Theta \right) } $$
There is a Sequential ...
- 1,091