Skip to main content

Questions tagged [tikhonov-regularization]

Tikhonov regularization, named for Andrey Tikhonov, is the most commonly used method of regularization of ill-posed problems, and is a generalization of ridge regression.

Filter by
Sorted by
Tagged with
3 votes
0 answers
41 views

L1/L2 regularization in neural nets

For linear regression, after doing L1/L2 regularization one can compute a closed form solution for the weights in nice cases. From here, one gets the intuition where: L2 regularization shrinks ...
dummy's user avatar
  • 275
2 votes
1 answer
145 views

What are a priori advantages of Lasso regularization for linear regression models?

What are a priori advantages of Lasso regularization for linear regression models, over many other heuristically-justifiable methods that both regularize the problem and perform variable selection? ...
AL1117's user avatar
  • 31
3 votes
0 answers
54 views

What's the unified definition of Tikhonov regularization

I met with "Tikhonov regularization" in two textbooks. The first is "Pattern Recognition and Machine Learning" by Christopher M. Bishop. In page 267 of his book, the regularized ...
zzzhhh's user avatar
  • 333
2 votes
0 answers
141 views

Effect of Regularization on bias and variance

I have already seen several related questions: 1, 2, 3, 4, 5. The answer to 1 states Regularization attemts to reduce the variance of the estimator by simplifying it, something that will increase the ...
muser's user avatar
  • 145
1 vote
0 answers
27 views

simultaneous parameter and variance estimation in statistical inverse problem

Background: I'm working on a geophysical inverse problem and interested in regularization parameter selection. Just by way of explanation, we're trying to estimate the friction underneath a flowing ...
Daniel Shapero's user avatar
1 vote
1 answer
136 views

Iterative generalized ridge regression

I am looking for some references. Assume I have a series of observable input/output pairs $(y_t, X_t)$ for which I assume the following relations to hold: $$\beta_t\text{ are i.i.d. }\sim N(\bar{\beta}...
MthQ's user avatar
  • 35
0 votes
0 answers
373 views

Regularization Terms in MLE

Can you add regularization terms to any likelihood function you're trying to maximize? (e.g. L2/Tikhonov, Lasso terms) I'm used to seeing this done with simple quadratic loss functions (e.g. for ...
Mich55's user avatar
  • 117
1 vote
0 answers
52 views

effect of multiplying by Tikhonov regularization factor after an inverse?

I came across a repository which uses Tikhonov regularization to compute an inverse, but then in the inference step they multiply by the Tikhonov factor again... Compute $\Phi\Phi^T$ Compute the ...
Joff's user avatar
  • 942
3 votes
1 answer
509 views

Tikhonov regularization equivalence to adding random noise

In Pattern Recognition and Machine Learning Ch 5.5.5 Bishop derives a regulariser for neural networks that is equivalent to the tangent propagation regulariser (a regulariser that is invariant to ...
tail_recursion's user avatar
1 vote
0 answers
61 views

Explain L1 vs. L2 regularization difference using the scientific mindset [duplicate]

I need to have job interview soon, one of the questions may be L1. vs L2 regularization. Yann LeCun explained best to my knowledge the difference between L1 and L2 regularization. L1 or Lasso: ...
Nabuko's user avatar
  • 31
0 votes
1 answer
624 views

Simple explanation of Takeuchi’s information criterion?

Takeuchi’s information criterion is said to be the generalization of AIC to misspecified models. That publication presents DEGREES OF FREEDOM FOR NONLINEAR LEAST SQUARES ESTIMATION. From that source: ...
Carl's user avatar
  • 13.2k
1 vote
0 answers
411 views

Understanding how regularization penalizes higher order features

I am trying to understand how exactly L2 or ridge regression penalize higher order polynomial features to make fitting curve smoother. I read this article about regularization. I contemplated and I ...
Rnj's user avatar
  • 205
3 votes
1 answer
406 views

Ridge regularization - intuition behind $\lambda$

I have seen many similar questions and I understand that $\lambda$ is some kind of a tuning parameter that decides how much we want to penalize the flexibility of our model. In other words $\lambda$ ...
bajun65537's user avatar
3 votes
0 answers
658 views

Entropy regularization versus L2 norm regularization?

In multiple regression problems, the decision variable, coefficients $\beta$, can be regularized by its L2 (Euclidean) norm, shown below (in the second term) for least squares regression. This type of ...
develarist's user avatar
  • 4,025
1 vote
1 answer
238 views

Is there any need for regularization in an overdetermined multiple regression problerm?

Supposed I have a small number of features, say 4 or 5, and I have hundreds of data points. That is, I am in an over-determined situation. Is there any benefit to using regularization in this setting ...
ManUtdBloke's user avatar
0 votes
0 answers
333 views

How does regularization affect the offset term and weights for OLS?

I have a couple of questions regarding regularization that I am confused on, and searches on this forum doesn't seem to answer my specific questions. When we regularize with $\lambda ||w||^2$ or $\...
roulette01's user avatar
1 vote
0 answers
124 views

Regularization and multicollinearity

Multicollinearity/collinearity, from what I understand, occurs when 2 or more predictor/independent variables (variates) are strongly linearly dependent. This leads to overfitting, and we could use ...
24n8's user avatar
  • 1,157
2 votes
1 answer
2k views

Trace of the Hat Matrix in Ridge Regression

Generally, I know that the trace of the hat matrix ($H$) is equal to the rank of H since it is an orthogonal projection. If I wanted to show the trace of $H$ in ridge regression, would I be able to ...
324's user avatar
  • 504
2 votes
0 answers
40 views

Why we only use L1 and L2 in penalized regression [duplicate]

I understand we do not want to go lower than 1 because the problem is no longer convex. But why not use L3 for example? Is there any reason why L2 is so popular but not higher norm?
user3661376's user avatar
3 votes
3 answers
3k views

Why does $l_2$ norm regularization not have a square root?

Specifically talking about Ridge Regression's cost function since Ridge Regression is based off of the $l_2$ norm. We should expect the cost function to be: $$J(\theta)=MSE(\theta) + \alpha\sqrt{\...
Nick L's user avatar
  • 31
0 votes
1 answer
1k views

Derivation of ridge regression for multi-value-target vectors

At university, I learned with these slides about ridge regression and its derivation with the assumption that the target- and predicted values have the dimensions $1\times1$. However, now I need to ...
Daniel's user avatar
  • 145
1 vote
0 answers
50 views

non-linear tikhonov/ridge regularization?

For traditional ridge regression, the loss function is $loss\_function = ||A\mathbf{x}-\mathbf{b}||_2^2 + ||\Gamma\mathbf{x}||_2^2$ https://en.wikipedia.org/wiki/Tikhonov_regularization Is there a ...
mathew gunther's user avatar
2 votes
0 answers
344 views

$L^2$ Regularization and Hessian Matrix [duplicate]

In the second paragraph it is mentioned that eigenvector of $H$ is rescaled by a factor of $\frac{\lambda_i} {\lambda_i +\alpha}$ What exactly meant by that ?
Daki Withanage's user avatar
2 votes
0 answers
441 views

Difference Between Two Tikhonov Regularization Schemes

For the solution of $Ax = b$, where $A$ is a square matrix, what is the difference between these two regularized solutions: $x = (A + \alpha I)^{-1}b$ -- coressponding to eq.3 below $x = (A^TA + \...
Cagdas Ozgenc's user avatar
30 votes
3 answers
3k views

The origin of the term "regularization"

When I introduce concepts to my students, I often find it fun to tell them where the terminology originates ("regression", for example, is a term with an interesting origin). I haven't been able to ...
Matthew Drury's user avatar
2 votes
0 answers
272 views

Finite difference based regularization matrix

I've just started reading about Tikhonov Regularization. Would someone please help with a simple numerical example of a case where regularization matrix of the form of a second order finite difference ...
user2696565's user avatar
  • 1,419
5 votes
1 answer
3k views

Applying L1, L2 and Tikhonov Regularization to Neural Nets: Possible Misconceptions

I'm interested in applying several different types of regularization to neural nets and want to make sure I haven't learned the material incorrectly. I have successfully coded Weight Decay and Dropout,...
SQLServerSteve's user avatar
40 votes
2 answers
19k views

Is Tikhonov regularization the same as Ridge Regression?

Tikhonov regularization and ridge regression are terms often used as if they were identical. Is it possible to specify exactly what the difference is?
Carl's user avatar
  • 13.2k
3 votes
1 answer
3k views

Tikhonov regularization in the context of deconvolution

I came across "Tikhonov regularization" and I have bare knowledge on it. It seems that it is a type of regularization that is important for deconvolution. Are there any good resources and examples? ...
RockTheStar's user avatar
  • 13.1k
1 vote
0 answers
91 views

Sequential Least Squares for Tikhonov Regularization [duplicate]

Given a Weighted Linear Least Squares problem where the cost function is given by: $$ J = { \left( x - H \Theta \right) }^{T} {C}^{-1} { \left( x - H \Theta \right) } $$ There is a Sequential ...
Royi's user avatar
  • 1,151