Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange
Join us in building a kind, collaborative learning community via our updated Code of Conduct.

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

1
vote
0answers
16 views

Bayesian linear regression on complex : how to use the prior laws and more

My model is as follows : With $y\in\mathbb{C}^{40},A\in\mathbb{C}^{40\times10},x\in\mathbb{C}^{10},b\in\mathbb{C}^{40}$ : $$y=Ax+b$$ $y$ and $A$ are known and I have a normal prior law on the module ...
1
vote
1answer
60 views

How does lasso regularization select the “less important” features?

I'm just starting in machine learning and I can't figure out how does lasso method find which features are redundant to shrink their coefficients to zero?
0
votes
0answers
21 views

How to determine the parameter in Ridge and Lasso [duplicate]

According to HERE, in Ridge Regression, we decide the parameter with cross-validation, But I could not understand what it said. Moreover, I have no idea to choose that of Lasso. Why does Lasso use ...
0
votes
0answers
6 views

ridge method for weibull PH model in survival data [on hold]

I have the survival data includes 252 patients, 25 independent variables and 35 events. I want to use ridge method for weibull model to these data.could anyone tell me which R package use for fit ...
5
votes
2answers
95 views

Why is multicollinearity so bad for machine learning models and what can we do about it?

Why is multicollinearity so bad for machine learning models? Is there ever a time when we can ignore multicollinearity? How does regularization ($L_1$, $L_2$) help us deal with multicollinearity?
0
votes
0answers
18 views

Why should each layer's child network output be close to parent network's output for variance regularizer?

I am reading up on PEA (Pseudo ensemble agreement) regularizer. specificaly in the neural networks domain. It introduces the concept of perturbing the model a little and forcing the model to make ...
0
votes
0answers
9 views

What is “Entropic Capacity”?

I found this term on the Keras blog website, quoted below Your main focus for fighting overfitting should be the entropic capacity of your model --how much information your model is allowed to ...
0
votes
1answer
55 views

How do Lasso coefficients change as lambda approaches infinity [closed]

I have encountered such a problem. I think 2, 3 and 4 pictures are true, but no. Can anyone help?
6
votes
1answer
646 views

How Ridge or Lasso regression really work?

Very basic question here, but I would like to understand (not mathematically) how the fact to add a "penalty" (sum of squared coeff. times a scalar) to the residual sum of square can reduce big ...
0
votes
0answers
27 views

Is the regularization term necessary when classifying one feature?

I'm using the Stochastic Gradient Descent linear classifier (implemented in Scikit-learn) to classify an image pixel by pixel. So my dataset has only one feature, ...
0
votes
0answers
23 views

Soft-thresholding for the LASSO with complex valued data

I'm currently implementing coordinate descent for the LASSO with complex-valued data. For this, one needs a complex version of the soft-thresholding operator, which seems hardly available on the net. ...
1
vote
0answers
25 views

Relationship between regularization parameter in Ridge/Lasso with budget constraint

The equation for lasso and ridge regression are given as follows in the ISLR textbook: The dual form of the above equations are given in terms of budget as below: I am wondering if there is a ...
0
votes
1answer
21 views

Stochastic gradient descent update

Equation 93 of Chapter 3 of Michael Nielsen's neural networks book describes the stochastic gradient descent update rule as the following: $w \leftarrow (1-\frac{\eta\lambda}{n})w - \frac{\eta}{m}\...
2
votes
1answer
53 views

Why weights are not negative in Lasso regression?

I can understand that lasso could make some weights to zero and prevent over-fitting. But for all the figure I see about lasso regression, weight will stay at zero once it reach zero and increasing ...
1
vote
0answers
17 views

Preventing logloss from over-optimizing on some points (or groups of points)

I've got a neural network classification model that does ok in terms of binary logloss. However I watched the training quite closely, outputting the logloss of each batch (batches are 100 samples) and ...
0
votes
1answer
32 views

Should we average weight decay loss in neural network?

In a typical neural network, which way is the common way to add regularization? Assuming regression task, regression error loss is Mean-squared-error Then we can have two choice of regularization ...
0
votes
0answers
14 views

gLASSO - regularization makes upper and lower triangles of estimated matrix equal

The overall goal is to get an estimate of the precision matrix (inverse of a covariance matrix given some nxp data), which can be translated into a graph showing the partial correlation relationships ...
4
votes
1answer
137 views

Question about Xgboost paper weights and decision-rules

Can someone please explain what the weight $w$ is doing and how it works here? I also didn't understand how $q$ transforms an $m$-dimensional vector to $T$. Edit Answer by usεr11852 is pretty good. ...
0
votes
0answers
26 views

Why do we not take the average of the regularization hyperparameters chosen by hold-out/cross validation?

I will talk restrict to hold-out for simplicity but my question applies to cross validation too. Say we have a regularization hyper-parameters we are looking for $\lambda$. We choose it by training ...
1
vote
1answer
27 views

How to conduct confirmatory factor analysis with small sample size?

I plan to conduct a confirmatory factor analysis, wherein there are 12 observed variables and 3 latent variables. My sample size is 30. However, I read that to conduct a factor analysis, the sample ...
1
vote
1answer
40 views

why small weights are preferred in neural networks

I'm viewing the CS231n lectures and trying to understand some of the regularization concepts. I think I understand the rationale behind "spread out" weights that are preferred by L2 regularization, e....
2
votes
1answer
85 views

What are the implications of scaling the features to xgboost?

Doing research about the xgboost algorithm I went through the documentation. I have heard that xgboost does not care much about the scale of the input features In this approach trees are ...
0
votes
0answers
26 views

What are the Advantages of Using Both $L_1$ and $L_2$ for Regularization? [duplicate]

This is what I found to compare the two: But I could not find the advantages of using both, for example for a linear regression model?
0
votes
0answers
62 views

Why are the coefficients in a multinomial logistic regression a matrix?

I am conducting an analysis in which I have 3 different groups and a set of 80 continuous variables that I think can discriminate between the 3 groups. I want to: see if indeed I can discriminate the ...
2
votes
1answer
66 views

What modeling problem does ridge regression solve?

If your modeling problem is that you have too many features, a solution to this problem is LASSO regularization. By forcing some feature coefficients to be zero, you remove them, thus reducing the ...
1
vote
0answers
93 views

Understanding regularization in xgboost

A general loss function is: \begin{split}\text{obj} = \sum_{i=1}^n l(y_i, \hat{y}_i^{(t)}) + \sum_{i=1}^t\Omega(f_i) \\ \end{split} which is prediction cost + regularization cost A decision tree is ...
0
votes
1answer
34 views

Is the solution at tangent point an optimal solution?

From what I understood from this article, the blue circles are the level curves and the blue dot is the optimal solution that minimizes the cost function. The yellow circle is the L2-norm constraint. ...
1
vote
1answer
50 views

Why limiting the weights to not grow to big numbers help to avoid overfitting? [duplicate]

I understand that in order to avoid overfitting we need to reduce the complexity of the network. Or, in other words we can reduce the degree of polynomial. L1-norm does exactly this - reduces the ...
5
votes
2answers
175 views

Regularized parameter overfitting the data (example)

Possible duplicate of (Why) do overfitted models tend to have large coefficients? How does regularization reduce overfitting? In the Coursera's machine learning course by Andrew Ng, I came across ...
4
votes
2answers
202 views

What is the geometric explanation for high variance = overfitting = bad generalization?

I am reading about the L2 regularization. According to Python Machine Learning - Second Edition, "by increasing the regularization strength via the regularization parameter λ , we shrink the weights ...
4
votes
0answers
40 views

Using covariates from penalized regression model in unpenalized model

The good news where I am is that researchers are doing less stepwise covariate selection now that I've introduced penalized regression. The bad news is that researchers want to use elastic-net ...
0
votes
1answer
34 views

Optimization problems and strict vs non-strict inqualities?

If there are strict inequalities can we always replace them with non-strict ones? I'm inclined to say yes, but I'm struggling to think of an example where we could do this.
8
votes
2answers
542 views

The graphical intuiton of the LASSO in case p=2 [duplicate]

Since this question was already ask here, I hope that it isn't a duplicate because it isn't answered. In my question I'll use another graph to make it more clear: The left plot is describing the ...
1
vote
0answers
214 views

Early stopping by epoch-limit

Is limiting the maximum number of training epochs during optimization a standard regularization process? I have seen it in many source codes of matrix factorization implementations, but I was not ...
1
vote
2answers
63 views

Weight decay VS “model-capacity-reduction” regularization

In artificial neural networks, is weight decay regularization the same sort of regularization that would reducing the capacity of the model be? I've learned that applying weight decay shrinks the ...
2
votes
0answers
120 views

What does L2-regularization in LightGBM do?

I use LightGBM for regression task and I'm planning to use L2-regularization to avoid overfitting. It's quite clear for me what L2-regularization does in linear regression but I couldn't find any ...
10
votes
4answers
352 views

The proof of equivalent formulas of ridge regression

I have read the most popular books in statistical learning 1- The elements of statistical learning. 2- An introduction to statistical learning. Both mention that ridge regression has two formulas ...
5
votes
2answers
120 views

Graphical interpretation of LASSO

I've a question regarding to the graphical intuition of the LASSO. I'm understanding why the lasso produce zero coefficient in case of intersecting a corner of the diamond. But I don't understand the ...
2
votes
0answers
15 views

How to derive a recursive version of a regularised cost function

I am to derive a recursive version of the following cost function and examine for which choice of D can we have a estimator windup $V(\theta) = \frac{1}{2}\sum_{t=1}^n(y(t)-\phi(t)^T\theta)^2 + \...
1
vote
0answers
36 views

Confidence Interval for Multinomial Elastic Net Predicted Probabilities

I am building an application which involves multinomial logistic regression models with the elastic net penalty using the glmnet-library on automatically collected data in R. My interest in particular ...
0
votes
0answers
29 views

Equivalent of using a Poisson prior in terms of a penalized regression?

I know that most penalized regressions have also a Bayesian interpretation, e.g. ridge least squares regression corresponds to the MAP estimate obtained under a Gaussian prior in a Bayesian regression,...
1
vote
1answer
27 views

(1995) Bishop's cite on weight decay regularization

On the book "Neural networks for pattern recognition" [Bishop, 1995], in chapter 9 about regularization there is a paragraph that says: Some heuristic justification for the weight-decay regularizer ...
1
vote
1answer
58 views

Coordinate descent for lasso

I was reading abut the coordinate descent procedure for lasso. I have to write about it. I understood how it works but what I don't get are the formulas. Everywhere I see some additions or something ...
4
votes
1answer
309 views

Why Ridge regression increase and not decrease the model's error?

I am trying to optimize the linear regression using the regularisation of Ridge using glmnet function. But the problem is, instead of decreasing the error after using Ridge method, the error increase. ...
46
votes
7answers
4k views

Why is the regularization term *added* to the cost function (instead of multiplied etc.)?

Whenever regularization is used, it is often added onto the cost function such as in the following cost function. $$ J(\theta)=\frac 1 2(y-\theta X^T)(y-\theta X^T)^T+\alpha\|\theta\|_2^2 $$ This ...
0
votes
0answers
27 views

Where should I start to understand Lasso regularization formula?

I'm learning Google machine learning crash course and in the section about L1 regularization. And I'm very interested about the mathematical way to explain the "selection" in such regularization ...
2
votes
0answers
54 views

Interpretation of Elastic Net Regression Coefficients

I would like to interprete the coefficients of a elastic net regression (i'm using function glmnet()$beta in R). The coefficients of the elastic net regularized ...
33
votes
7answers
5k views

Why doesn't regularization solve Deep Neural Nets hunger for data?

An issue I've seen frequently brought up in the context of Neural Networks in general, and Deep Neural Networks in particular, is that they're "data hungry" - that is they don't perform well unless we ...
1
vote
0answers
14 views

Lasso as parameter norm minimization

In James, Radchenko & Lv (2009), the authors state in section 3.1 what they say is an equivalent formulation of Lasso, without proof of equivalence: I have never seen this Dantzig-like version ...
-2
votes
1answer
33 views

How to get the Mean squared error of LassoLarsIC?

lassoLarsIC give a predict methods but it does not give a MSE method, can someone specify the reason why it is not considered by SKlearn ? Using both the AIC and BIC method I was able to get the ...