Stack Exchange Network

Stack Exchange network consists of 175 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

0
votes
0answers
17 views

Looking for a Regression Method with way to enfornce a reflection/ flip consistency for input [on hold]

I have a set of $N$ dimensional 1D features using which I build a linear a regression model to predict a single scalar value. Say, $\hat{y}(w, x) = w_0 + w_1 x_1 + ... + w_p x_p$ with the regression ...
1
vote
0answers
5 views

Variation in accuracy of data splitting before and after data augmentation

How much accuracy of the system varied/changes between two cases Data augmentation before splitting Data augmentation after splitting, only on training data Is there any literature published?
0
votes
1answer
26 views

Data augmentation on entire dataset before splitting

If I apply rotation of 5 different angles and randomly cropp 10 different images from each rotated image and than divided into training testing and validation. Will it be totally incorrect evaluation ...
1
vote
2answers
39 views

In “A Topology Layer for Machine Learning,” are the topological priors learned by the network or imposed by humans?

In this paper by Gabrielsson, Nelson, et al. the authors "present a differentiable topology layer that can, among other things, construct a loss on the output of a deep generative network to ...
1
vote
0answers
12 views

Linear regression: How to demand similar MSE across different subgroups?

In typical least square regression, we want to minimize $||y-\hat{y}||$ where $\hat{y}=B*x$ I am now working on a car fleet management problem, $y$ can be split into several groups (in my case, ...
0
votes
0answers
42 views

Should we penalize dummy variables?

Using glmnet we run the following regression cvfit = cv.glmnet(x,y, alpha = 0, intercept = FALSE) where $y$ is the response variable and $x$ is the input matrix....
2
votes
0answers
9 views

Is there a formal relation between weight regularization and compression?

In my understanding, compression, strictly speaking, means that we diminish the amount of data required to describe something, such as a model. E.g. compressing an image file means to create a file ...
2
votes
1answer
32 views

How does the penalized form of RSS (residual sum of squares) work?

In another word, how to reverse engineering the equation (5.9) by explain all the assumption and reasoning after the plus sign of (5.9) in Elements of Statistical Learning. Note: I had used the ...
0
votes
1answer
33 views

Inconsistency between poisson and negativebinomial in glm

I am working with the negativebinomial distribution for GLM. I have done one test which is finding the poisson distribution results. Here is the first test: ...
0
votes
0answers
22 views

Terminology: “L1 regularization” even if I'm using mean instead of sum? [duplicate]

In my loss function I'm using the mean of the log-cosh error between the predictions and targets, as well as an additional regularization term that scales as the mean of the absolute value of another ...
1
vote
0answers
28 views

How does regularized regression overcome the p > n problem?

So, I understand why simple linear or logistic regression will have infinite solutions in this case (good answers here and here). But while LASSO will only select n features, Elastic net does not have ...
1
vote
1answer
29 views

Enforcing Dirac delta-like Activations on a Neural Network

I am working on a custom neural network model including convolutional and dense layers. I intend to enforce outputs a certain dense layer to approximate a Dirac delta function (or any localized pulse)....
15
votes
2answers
341 views

Why does shrinkage really work, what's so special about 0?

There is already a post on this site talking about the same issue: Why does shrinkage work? But, even though the answers are popular, I don't believe the gist of the question is really addressed. It ...
2
votes
1answer
16 views

Reference Request: Proof of Early Stopping Criterion

I am looking for a proof that "Validation-based early stopping" methods work but I have no idea where to start, as I am new to this field. Any recomendations of some rigerous papers that focus on ...
0
votes
0answers
22 views

Notation problem with sparse regularized correlation matrix

I am trying to apply a specific method to obtain a sparse correlation matrix $R$ from a regularized correlation matrix $\Sigma^{\delta}$, which was computed from $N$ observations of a multivariate ...
0
votes
0answers
22 views

Elastic Net - number of non-zero variables

I have a question regarding the interpretation of the trace of coefficients when running Elastic net with the package glmnet in R. This is the plot I obtain with alpha = 0.5 My understanding is that ...
0
votes
0answers
9 views

Supervised learning vs Regularity based learning

I have some confusion about regularity based learning and supervised learning. Are they in essence, not the same thing? We have some labelled data, and our algorithms are structured based on learning ...
0
votes
1answer
107 views

Why do we need regularized logistic regression?

We use regularized Linear Regression to prevent the model from overfitting (reduce model complexity). Does the same idea hold with regularized Logistic Regression? Is regularized Logistic ...
0
votes
1answer
26 views

Question about the location of regularization constant C in SVM

I've encountered very similiar but different functions in SVM optimization problem, the diffrence is in the location of regularization constant C. $\sum_{i=1}^n(1-(y_i(w^tx))_+ +\frac{1}{2C} \left\...
8
votes
1answer
298 views

In linear regression why does regularisation penalise the parameter values as well?

Currently learning ridge regression and I was a little confused about the penalisation of more complex models (or the definition of a more complex model). From what I understand, model complexity ...
1
vote
1answer
59 views

James-Stein regularizing covariance like a mean

In James-Stein's estimator we have a $p$-dimensional random vector $X\sim N_{p}(\mu ,I)$ where $\mu \neq 0$ and the goal is to estimate the mean vector using the single ($n=1$) data vector $X$. The ...
2
votes
1answer
101 views

Why is lasso more robust to outliers compared to ridge?

In my attempt to reason about it intuitively I am concluding that ridge might be more robust to outliers. Following is my intuitive/lose reasoning : If there is an outlier then to match my ...
0
votes
1answer
19 views

How can I store information in a custom regularizer? [closed]

I'm trying to create a custom keras regularizer that uses the distance of the layer's weights from it's original weights, but what I used doesn't seem to work. I get a zero difference at all times. ...
3
votes
1answer
34 views

Should MLE estimation always be using penalizers?

I am referring to the family of estimation techniques like MLEs, least-squares, etc., that an l2 penalizer/regularizer can be added to. I'm not interested in NHST, but just estimation (say, of some ...
1
vote
0answers
10 views

Does the expected cross validated error/NLL from regularisation have one minimum?

When trying to choose the value of the regularisation parameter(s) in lasso, ridge regression, or elasticnet, one generally computes the cross validated error or negative log-likelihood as explained ...
2
votes
1answer
36 views

How does L2 penalize large weights

The L2 regularization term is useful because it penalizes large weights over smaller weights which is good to prevent overfitting. I'm having a hard time understanding how exactly it does this. This ...
0
votes
0answers
19 views

Link between norm of weights/coefficients and smoothness

We often avoid overfitting by penalizing the norm of the weights/coefficients (in a classic Ridge or Lasso regression). I understand that we want smooth functions as they will be more likely to ...
0
votes
0answers
29 views

How is the L2 regularization derived? [duplicate]

I just proved to myself why the regularization is added rather than multiplied to loss function. I did so by taking the MLE formula... $$\operatorname{argmax}\sum \log(P(x_i\mid\Theta ))$$ and ...
0
votes
0answers
32 views

Lasso Logistic Regression in the presence of Class Imbalance

Since class imbalance only affects the estimate of the intercept in vanilla logistic regression, the orientation of the optimal separating hyperplane remains unchanged. However in $L_1$-regularized ...
3
votes
1answer
235 views

Does regularization penalize models that are simpler than needed?

Yes, regularization penalizes models that are more complex than needed. But does it also penalize models that are simpler than needed?
1
vote
1answer
21 views

Consisten Regularizer for Neural Network

In the book 'Pattern Recognition and Machine Learning' by Bishop (p.257 ff.) he considers a weight decay regularizer of the error function $$\hat E(w)=E(w)+\frac{\lambda}{2}w^tw$$ where $w$ is a ...
0
votes
0answers
13 views

Package to perform 2nd degree polynomial regression with L1 penalty for use of the 2nd degree

I'm trying to fit either a straight line or 2nd degree polynomial through many sets of points (2-dimensional data). I would much prefer a straight line over a polynomial, so am trying to penalize the ...
0
votes
0answers
13 views

Finding significant explanatory values with unregularized logistic regression?

So i'm working currently with a colleague on a classification problem where both of us have different approaches. So i try to find the most important features through standard L1 regularization and he ...
1
vote
0answers
34 views

Is there a theoretical basis for using partial least squares with categorical responses

I am using what is called PLS-DA in JMP to find a model for predicting a categorical (Positive/Negative) response. The documentation says that the responses are simply coded as 0/1, thereby ...
2
votes
1answer
288 views

Ridge Regression with Gradient Descent Converges to OLS estimates

I'm implementing a homespun version of Ridge Regression with gradient descent, and to my surprise it always converges to the same answers as OLS, not the closed form of Ridge Regression. This is ...
0
votes
0answers
36 views

Using Quadratic Programming to solve Lasso and Ridge regression models?

I'm trying to build linear, ridge and lasso regression models for at set of data (40 obs., 4 features, 1 response). I'm building the models using the sklearn package for Python and I can easily find ...
0
votes
1answer
33 views

Interpreting ridge coefficients as a function of regularization

Data consists of 40 observations with 4 dimensions and a response-variable. When doing a ridge regression on my data and plotting the coefficients and coefficient errors (MSE of the ridge ...
0
votes
0answers
58 views

How to calculate One Standard Error rule tuning parameter for prediction error during k-fold CV

I'm trying to wrap my head around exactly how this rule goes into place, so I can use it by hand in other model selection setups. So here's some R code to get it started: ...
0
votes
0answers
9 views

choice of the lambda parameter in the logit multinomial ridge model

I can not find clear literature on how to choose the penalty parameter in the logit multinomial ridge model. As read in the linear models and trying to adapt it to the model I need it would be through ...
0
votes
0answers
45 views

Random partition techniques for lasso and elastic net

I think this is the correct place for this question. I have implemented lasso, elastic net and a different estimator on a real data set. I used 10-fold cross-validation (CV) one time without ...
0
votes
0answers
57 views

Why does Group Lasso use L2 norm for individual group penalties?

In group lasso $$\min_{\beta}\left\{\frac{1}{2} \left\lVert{y}-\sum_{l=1}^mX^{(l)}{\beta^{(l)}}\right\rVert_2^2 +\lambda\sum_{l=1}^m\sqrt{p_l}\left\lVert{\beta^{(l)}}\right\rVert_q\right\}$$ the ...
2
votes
0answers
48 views

Probability that feature selection in elastic net regularisation is meaningful - evaluating the statistical significance of chosen features

I have a research question - can I use baseline clinical features to predict my binary clinical outcome in individual patients? I am interested if the performance of my model is greater than chance. I ...
1
vote
0answers
37 views

Which metric should be used for learning rate reduction on plateau?

I am testing three different neural network architectures on a dataset to see which architecture performs the best. My methodology for every architecture is to Split the data into train/test Take ...
1
vote
1answer
90 views

Convex optimization: Is gradient descent faster if a regularizer is added?

I am not sure if this is a true statement or not but there appears to be an intuition around this among experts in the field that I do not quite understand. The idea is: Given a convex optimization ...
3
votes
1answer
359 views

what does regularization mean in xgboost (tree)

In xgboost (xgbtree), gamma is the tunning parameter to control the regularization. I understand what regularization means in <...
3
votes
0answers
78 views

Why does cv.glmnet not use the same lambda sequence across different folds to find the hypertuning parameter lambda?

I assumed that cv.glmnet works as follows: Generate multiple glmnet fits for the entire data, presumably for automated lambda sequence using coordinate descent Use the lambdas gotten in step 1, and ...
0
votes
0answers
40 views

How dose dropout affect Weights and Bias?

I applied dropout in my network , and it worked , but i can't interpret dropout effects on weight and bias, to be more specific , i can't interpret why appling droput and not applying dropout have a ...
6
votes
1answer
150 views

Variance of average of $n$ correlated random variables

Reading about deep leaning, I came across the following formula. $$ \mbox{var} \left( \frac{1}{n} \sum_{i=1}^{n} X_i \right) = \rho \sigma^2 + \frac{1-\rho}{n} \sigma^2 $$ where $X_1, \dots, X_n$ ...
2
votes
0answers
31 views

$L^2$ Regularization and Hessian Matrix [duplicate]

In the second paragraph it is mentioned that eigenvector of $H$ is rescaled by a factor of $\frac{\lambda_i} {\lambda_i +\alpha}$ What exactly meant by that ?
5
votes
1answer
81 views

Literature on $\ell_q$ LASSO, $q < 1$

I am not sure how is $\ell_q$-LASSO called, but here I am talking about LASSO regression, with $\| \beta \|_{\ell_q}$ regularization, $q< 1$. In popular literature, such as Elements of Statistical ...