Skip to main content

Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

Filter by
Sorted by
Tagged with
0 votes
0 answers
13 views

difference between l2 penalty and l2 loss in SAE

I was reading this paper from Anthropic https://transformer-circuits.pub/2024/scaling-monosemanticity/index.html and in the paper loss is defined like this :$$ L = \mathbb{E}_x \left[ \| x - \hat{x} \|...
Mrnobody's user avatar
1 vote
0 answers
21 views

Multi-task learning-Loss function

0 I am training a convolutional autoencoder with two velocity fields (2D array) as inputs and outputs. These fields represent wind velocities in both the x and y directions within a square domain. My ...
Sarah's user avatar
  • 11
1 vote
0 answers
31 views

Ask a coding problem for the equivalence of unconstrained Optimization with L1 Regularization

I recently read a statistics paper: DAGs with NO TEARS: Continuous Optimization for Structure Learning It has an unconstrained problem: $$\min_\theta F(\theta)+\lambda || \theta||_1$$, where $$F(\...
PiVoyager's user avatar
0 votes
1 answer
45 views

why does LASSO regression return unstandardized coefficients [closed]

I have more general questions that does not refer to a coding issue. Why does LASSO regression require standardization of the predictors but return unstandardized coefficients (glmnet function - https:...
Simon's user avatar
  • 1
1 vote
0 answers
14 views

penalized package [closed]

Has anyone used penalized package? I was using it for lasso in Cox regression, with time-varying coefficients. The problem is when I made a plot with ...
Danny's user avatar
  • 11
-1 votes
1 answer
23 views

How does a neural network differentiate between a neuron that outputs 0 and a dropped-out one?

How does a network differentiate between a neuron with output 0 and a dropped-out neuron (this neuron might output a non-zero value but due to dropout it outputs 0)?
ado sar's user avatar
  • 477
3 votes
1 answer
35 views

What is the boundary curve for $λ_1$ and $λ_2$ that give at least a 0 component in elastic net?

Define the elastic net estimate: $ \hat{\beta}^{\lambda_1, \lambda_2} = \arg \min_{\beta \in \mathbb{R}^p} \left( \frac{1}{2n} \| y - X\beta \|_2^2 + \lambda_1 \ \frac{1}{2} \|\beta \|_2^2 + \lambda_2 ...
george1994's user avatar
1 vote
0 answers
37 views

how to approximate the eigendecomposition of a correlation matrix when the data have been standardized?

Context I am working to develop a penalized regression framework that will scale up to analyzing high dimensional data with a certain correlation structure. Let $X$ represent an $n \times p$ matrix of ...
Tabitha Peter's user avatar
0 votes
0 answers
34 views

Is there any test I can apply to the data to tell whether the adaptive LASSO or the LASSO is likely to perform better in prediction?

Is there a. test I can perform on a sample that will tell me if coefficients estimated using the LASSO, the adaptive LASSO, or the relaxed adaptive LASSO are likely to give better (in the mean squared ...
andrewH's user avatar
  • 3,157
0 votes
0 answers
22 views

Confidence intervals ODE ridge regression

i want to find confidence intervals for a least squares loss which is L2 regularized. I have only found something for linear problems, but in my case i want to estimate ordinary differential equation ...
LH44Stat's user avatar
6 votes
1 answer
112 views

Valid confidence intervals in GAM’s using shrinkage estimation

In this blog article: https://www.fharrell.com/post/improve-research/ it states: “The frequentist paradigm does not provide confidence intervals or p-values when parameters are penalized”. I was ...
user167591's user avatar
0 votes
0 answers
21 views

Testing difference between two models using WAIC and degrees of freedom of WAIC

I am conducting Bayesian penalised regression, horseshoe specifically, in R using the bayesreg package see here. One model is nested within the other, i.e. to the second model I have simply used all ...
llewmills's user avatar
  • 2,043
3 votes
0 answers
29 views

Is there a likelihood penalization or (im)proper prior to remove estimation bias for gamma parameters?

So I am learning that maximum likelihood estimation of the parameters for a gamma distribution are biased. As far as I understand there is no guarantee in general that there exists a prior (or base ...
Galen's user avatar
  • 9,381
4 votes
1 answer
92 views

Shrinkage of covariates in the Cox model

In a regression model (e.g Cox model) when there are too few events to support modeling all desired covariates / confounders, a possible solution is to apply shrinkage / penalise all but the exposure(...
user167591's user avatar
1 vote
0 answers
13 views

Positive distance weighting

I have an overdetermined linear system of equations that's solved with least squares. I'd like to weight the equations to penalize a bunch of inputs clumped up together. Ideally if two (or more) ...
Marsupilami's user avatar
3 votes
0 answers
41 views

L1/L2 regularization in neural nets

For linear regression, after doing L1/L2 regularization one can compute a closed form solution for the weights in nice cases. From here, one gets the intuition where: L2 regularization shrinks ...
dummy's user avatar
  • 275
1 vote
1 answer
78 views

Can I utilize Ridge Regression to update coefficients of a Linear Regression model for a new dataset?

I have fitted a Linear Regression Model using one dataset. Now, I have another smaller dataset that I want to refine the model with. Can I use Ridge regression to update the estimated coefficients for ...
Adham Enaya's user avatar
1 vote
0 answers
21 views

Optimal lamda in ridge regressioin

Original Ridge paper (Hoerl and Kennard, 1970) proposed the iterative estimation of optimal lambdas (which is k in their notation) in the generalized form of ridge, a form that would yield multiple k'...
Kaiwen Wang's user avatar
1 vote
0 answers
21 views

Maximum a Posteriori (MAP) in practice for machine learning

I'm a beginner in machine learning and I had a a few questions regarding MAP. From my limited understanding it seems to me a bayesian approach, specifically an MLE approach is incredibly useful when ...
Sloth Man's user avatar
  • 111
3 votes
1 answer
173 views

Explanation of the proof that SCAD penalty has the oracle property

I am trying to understand the proof that the SCAD has the oracle property. Could you help me with an explanation and a full break down of the steps, so that I can understand it? I'm unclear on how ...
mike's user avatar
  • 31
0 votes
0 answers
21 views

Lasso Regression Problem [duplicate]

$\operatorname*{argmin}_\beta\{\|y-X\beta\|^2 + \lambda\|\beta\|_1$, where $X$ is orthonormal. $\beta \in \mathbb R^d$. $X = [x_1,\ldots,x_n]^T$ and $y=(y_1,\ldots,y_n)^T \in \mathbb R^n$. $X^TX=I_{d\...
Harry Lofi's user avatar
0 votes
0 answers
19 views

How to sample with the 1-norm?

I am currently working on ridge regression, which can be interpreted using Bayesian statistics (DOI: 10.1016/j.electacta.2015.03.123). In particular, I know that the maximum-a-posteriori (MAP) ...
user avatar
5 votes
1 answer
134 views

Deep learning book, noise to weights - How do we get $\eta\operatorname{\mathbb E}_{p(\mathbf{x},y)}E[\|\nabla_{\mathbf{W}}\hat{y}(x)\|^2]$

I am reading Deep Learning by Goodfellow, Bengio, and Courville. On Page 238, they introduce noises as one way to do regularization. Quote: Noise applied to the weights can also be interpreted as ...
Kaira's user avatar
  • 299
3 votes
2 answers
129 views

How many zeroes from lasso linear regression?

Given a dataset $X$ with $d$-dimensional features $x \in R^d$, and a response variable $y$ you can perform a lasso regression, ie linear regression with L1 regularization, as $$ \min_{\beta} (X\beta - ...
alexmolas's user avatar
  • 288
3 votes
1 answer
109 views

Why is the regularization term multiplied by the error term in the cost function of SVM?

The cost function of the Optimal Margin Classifier(non-kernelized SVM) is given as : $$ J(\mathbf{\vec w}, b) = \frac{1}{2}\|\mathbf{\vec w}\|_{2}^{2} + C \sum_{i=1}^{n}\max(0, 1-y ^{(i)}(\mathbf{\vec ...
Sagnik Taraphdar's user avatar
0 votes
0 answers
16 views

The relationship between ridge regularization and CNN Data Augmentation

In Chapter 10.3.4 of Introduction to Statistical Learning with Applications in Python by James et al. there is a sentence on data augmentation for CNNs (adding natural transformations of images into ...
pvelayudhan's user avatar
0 votes
0 answers
15 views

Increasing the clarity in the tasks of image generation using CNN

What methods exist to improve the quality of generated images and the clarity of contours in the tasks of image denoising/debluring (using CNN), style transfer etc? I am interested in approaches that ...
Alimagadov K.'s user avatar
0 votes
0 answers
44 views

How to penalize disagreement between two classification loses?

I am working with a multi-head, multi-loss neural network. Each of the two heads is associated with a multi-class classification loss. The losses are combined additively. Assume loss 1 is trained to ...
Gertrude Porter's user avatar
5 votes
1 answer
118 views

Order of derivative for MGCV splines and low EDF

I'm following up on the informative discussion here concerning choice of m (order of derivative) for MGCV splines. Using the default options in MGCV (thin plate spline, REML for optimization, k=10, ...
dean's user avatar
  • 455
0 votes
0 answers
18 views

time series cross validation and avoidance of overfitting

So. I am doing Time Series Classification on various datasets using different types of classifiers (deep learning, dictionary-based, distance-based, interval-based, feature-based, convolution). As far ...
Sophia Vei's user avatar
0 votes
2 answers
43 views

Regularization using complexity penalty

I am currently reading murphy and i came across regularization The issue i am facing is building intuitive sense behind the complexity penalty they have used how can having log on my prior belief ...
Satej Raste's user avatar
2 votes
1 answer
144 views

What are a priori advantages of Lasso regularization for linear regression models?

What are a priori advantages of Lasso regularization for linear regression models, over many other heuristically-justifiable methods that both regularize the problem and perform variable selection? ...
AL1117's user avatar
  • 31
1 vote
0 answers
68 views

Why do Mixed Effects Regression models Shrink Parameter Estimates? [duplicate]

I have always been interested in understanding the following point: During the parameter estimation process, why are Mixed Effects Models able to "implicitly" perform shrinkage/...
Uk rain troll's user avatar
1 vote
1 answer
66 views

Sklearn's LogisticRegression C hyperparameter issue

In sklearn user's guide for LogisticRegression it is said that: where C is so, shouldn't C hyperparameter be in front of the regularization term r(w) rather than in front of the sum?
Ivan Mitriakhin's user avatar
1 vote
0 answers
33 views

Group Lasso optimization

I read that, for the group lasso, to solve the zero subgradient equations, one approach involves keeping all block vectors fixed, denoted as $\{\hat\theta_k, k \ne j\}$, and then solving for $ \hat \...
Jenny's user avatar
  • 261
4 votes
2 answers
250 views

Generalized additive model: Variable & model selection

I know this type of question has been asked many times before, so I apologize for re-posting about it. I bring it up again because it's been taught in one of my courses of study and I want to make ...
Nate's user avatar
  • 1,641
0 votes
0 answers
102 views

Exponentially Weighted Covariance Matrix with Ledoit Wolf Shrinkage

The Ledoit Wolf paper "Honey, I Shrunk the Sample Covariance Matrix" presents the formulation for the shrinkage intensity parameter estimate in Appendix B. The formula for a weighted ...
nka5we's user avatar
  • 49
1 vote
0 answers
44 views

Fitting a Gaussian function to Poisson noisy data

Let $A$, $\mu$, $\sigma$ be some positive, a priori unknown parameters. Define a Gaussian function $f$ as $$f(x) = A \mathrm{exp}\left(-\frac{1}{2} \left( \frac{x-\mu}{\sigma}\right)^2\right).$$ One ...
mathslover's user avatar
1 vote
0 answers
35 views

Obtain the R^2 of just the top predictors identified in a LASSO/ L1 penalized regression model

I've developed a model using L1 penalized regression using tidymodels and 10-fold cross-validation, and determined the predictors that explain approximately 87% of the variance in test data. I need ...
Fredrik Nylén's user avatar
1 vote
2 answers
104 views

Multicollinearity and large OLS estimates vs ridge regression

The point of regularization methods (for example ridge regression) is to penalize large ordinary least squares estimates. We know that variance-covariance matrix for OLS estimates can be decomposed ...
Adam Bogdański's user avatar
0 votes
0 answers
49 views

How can you constrain the intercept of a glmnet model to be positive?

If I use the lower.limits = 0 argument, it doesn't apply to the intercept for some reason. I can't find any documentation as to why or how to do it. Any ideas? ...
PaulG's user avatar
  • 1,297
3 votes
0 answers
29 views

For variable selection, would a viable alternative to using lasso be to use ridge with a threshold, or is switching to elastic net preferred?

A similar question was asked here Why can't ridge regression provide better interpretability than LASSO?, and the answer suggested that a main difference between lasso and ridge is that a zero ...
another_student's user avatar
0 votes
1 answer
34 views

Intuition for how individual coefficients change with increasing regularization penalties [duplicate]

I'm trying to build intuition around how individual coefficients change as a regularization penalty is increased (for both ridge and lasso). This is what I understand the curves of the l1 and l2 ...
another_student's user avatar
3 votes
2 answers
108 views

Positive Semidefinite Kernel in RKHS

The following shows part of the page 170 of The Element of Statistical Learning that I want to make clear. The solution can be characterized in two equivalent ways $$\min_{c_j}\sum_{i=1}^N(y_i - \...
jason 1's user avatar
  • 311
3 votes
1 answer
72 views

Is my regularized logistic regression model overfit?

I have a dataset with the following characteristics: moderate sample size (~300 samples) moderate class imbalance (~20% positives) high-dimensional (the number of independent variables, again ~300, ...
ladislaw94's user avatar
2 votes
1 answer
85 views

Regularization Problem and Reproducing Kernel Hilbert Space

The following shows part of the page 169 of The Element of Statistical Learning that I want to make clear. We have $$\min_{f \in \mathcal H_K}[\sum_{i = 1}^NL(y_i, f(x_i)) + \lambda\Vert f\Vert_{\...
jason 1's user avatar
  • 311
1 vote
1 answer
85 views

What is the objective function for weighted lasso & ridge?

For weighted OLS, the objective function can be written as $$ \arg \min_{\beta} ||W^{0.5}(y - X\beta)||^2 $$ This is quite similar to the objective function for plain OLS, except without the $W$ term: ...
24n8's user avatar
  • 1,157
1 vote
1 answer
104 views

Robust way to add predictors to existing linear model

I'm looking for a robust way to gradually build up a regression model -- namely I have a linear base-model with a robust set of predictors for which I'm fairly certain I have near optimal weights for, ...
ron burgundy's user avatar
1 vote
0 answers
8 views

Relationship between the t-statistic of a coefficient in an OLS multivariate regression and Ridge shrinkage?

If I'm running a multivariate OLS regression and look at the t-stats of coefficients, is it the case that the coefficients with smaller t-stats are shrunk relatively more if I were to run the same ...
user3089215's user avatar
1 vote
0 answers
21 views

Understanding application Lasso and Ridge Regression

Currently reading up on Ridge and Lasso regression, have some questions to clarify. Suppose Model 1 has all predictors (i.e., 8) and Model 2 only has a specific subset chosen after EDA (i.e., 5) ...
scoosch's user avatar
  • 11

1
2 3 4 5
29