Questions tagged [regularization]

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

Filter by
Sorted by
Tagged with
1
vote
0answers
16 views

penalised logistic regression with brglm - warnings

I've built a logistic model to predict a binary response. I've got four categorical predictors. One of them (posicion) has 6 levels, 3 of which occur not too frequently and are ALWAYS (by definition) ...
0
votes
0answers
15 views

What is the essence of Linear Discriminant Analysis while considering the correlated features for Inter Class problem?

Suppose, I have samples from APPLE, MANGO, and ORANGE --- these 3 classes. The goal is to do multiclass classification. Now, let's say, I have calculated 4 features from all of the 3 classes. By rules ...
0
votes
1answer
39 views

Is it possible to penalize the $k$NN classification algorithm?

I read through this paper on building an approach to Binary Classification in high-dimensional data. I wondered if there is a way to penalize the regular KNN classification algorithm?
0
votes
1answer
20 views

Need help understanding how C hyperparameter influences $w $ in regularized SVM

I am following Andrew Ng's lecture notes on SVM from CS229. What I am having trouble understanding is the new objective function. $min_{γ,w,b} \frac{1}{2} ||w||^2 + C\sum_{i = 1}^{m}ξ_i$ From what I ...
0
votes
0answers
10 views

Weighting observations according to their leverage glmnet

I wish to estimate the association between several biological features and a binary health outcome. These biological features, however, have occasional extreme (but valid) outlying observations. Given ...
1
vote
0answers
14 views

Is it meaningful to regularise a GEV log-likelihood?

Situation/Data: I'd like to start with an example from climate science. Suppose you have a univariate time series $\vec{z} = (z_1, z_2, ..., z_n)^T$, where $z_t$ are block maxima of time step $t\in1,.....
1
vote
0answers
15 views

Effect of weight decay on loss [duplicate]

I am new to deep learning and the terminology "L2 regularization" and "weight decay" seems to be used almost interchangeably... L2 regularization however modifies the loss function ...
0
votes
0answers
12 views

Questions on cross entropy, regularisation and linear softmax

I am working on obtaining skills and bridge gaps in the theory in ML. I pick up course that covers topics I am interested in and for self verification\where I stubmle upon I take a look on how others ...
2
votes
1answer
33 views

How to fit a smooth polynomial?

We have $N$ samples of the unknown function $f(x)$ on the finite interval $[a, b]$. The samples are subject to white noise of known variance. We want to approximate the function $f(x)$ by a polynomial ...
3
votes
1answer
39 views

Eigenvalues in Ridge regression [duplicate]

The ridge regression estimate is given by $$\beta^{*}=(X'X+kI)^{-1}X'y, k≥0,$$ where $X$ is the feature matrix. The original paper, Hoerl and Kennard's Ridge Regression: Biased Estimation for ...
0
votes
1answer
21 views

Multiplying a predictor by a constant in Lasso/Ridge regression

If we multiply one of predictors by a constant $c$ in the regression set-up for all data points. What happens to the weights (or specifically weight corresponding to that predictor) if we are doing ...
4
votes
1answer
50 views

When is cross validation necessary to estimate a parameter?

In 2013, @Donbeo asked whether there were any theoretical results supporting use of Cross Validation to choose the lasso penalty, and was scolded in the comments for asking "a pretty generic ...
1
vote
0answers
19 views

Does L1 regularization (Lasso) always leads to feature reduction? [duplicate]

This is a basic question about regularization term but I have searched for a while and cannot find the answer. My question is: does Lasso regularization always make some coefficients zero? A famous ...
0
votes
0answers
14 views

How is the smoothing spline penalty computed in practice?

I'm digging into smoothing splines and finding good resources, but no one talks about how to actually calculate the penalty $\int \hat{f}^{"}(x)^2 dx$ in the standard smoothing spline loss: Since <...
1
vote
1answer
44 views

Why don't I see a minimum in out-of-sample mean squared error along my lasso path?

I'm hoping to reproduce the following figure from Matt Taddy's book Business Data Science using the Happiness data set from Kaggle. Running linear regression using lasso regularization, he observes a ...
2
votes
1answer
35 views

How is prior distribution of weights selected in MAP estimates?

I just read MAP estimate of linear regression , and got to know that the regularization term is the result of considering prior distribution of weights . So , my question is how is this prior ...
1
vote
0answers
15 views

Is cross-validation (test) error below chance an indicator of overfitting?

I am training a binary classifier (e.g. logistic regression) on some multidimensional problem. I have tried leave-one-out and k-fold cross-validation. I have tried L1 and L2 regularization, and I have ...
0
votes
0answers
9 views

Enforcing sparsity constraints that make use of spatial contiguity

I have a deep learning network that outputs grayscale image reconstructions. In addition to good reconstruction performance (measured through mean squared error or some other measure like psnr), I ...
4
votes
2answers
38 views

What is the ideal approach to determine relationship between candidate predictors and a dependent variable in a data driven way?

I have asked several related questions (1, 2, 3), but now I would like to ask the most basic questions and hope to get a very solid answer. I have 40 treatment variables, and I am interested to find ...
1
vote
1answer
51 views

Model that shrinks a set of coefficients towards their common mean

I am interested in estimating the odds of a certain disease based on a medium sized group of correlated biological markers (roughly 20 markers). The model will also include several confounding ...
0
votes
0answers
32 views

I am “forcing” some control variables into a LASSO. Is it a problem that they are significantly correlated with some of the other predictors?

I am forcing two control variables into an adaptive LASSO model. I am doing this by constraining lambda to 0 for these predictors during the second step. I have ~40 other predictors that I would like ...
1
vote
1answer
86 views

Is this the correct way to run an adaptive LASSO?

I have been using the code here to run an adaptive LASSO in R using glmnet. Essentially it first runs ridge regression to get coefficients for each predictor. It ...
2
votes
1answer
45 views

Do we assume graphical LASSO explanatory variables to be normally distributed? And what if this assumption fails?

I am working on a graphical LASSO (GLASSO) shrinkage of the variance-covariance matrix of financial log-returns data for 10 years. I tested for normality and the Jarque-Bera test (but also other tests)...
2
votes
1answer
36 views

Penalization in multi class neural networks backpropagation

the cost function for my neural network. In neural networks back propagation we are trying to minimise our cost function W.R.T our parameters ( theta) . We penalize our neural network for every wrong ...
2
votes
0answers
16 views

Strong rules for the elastic net

In their paper (here), Tibshirani et al defined the lasso as the solution to $$ \text{argmin}_{\boldsymbol{\beta}}\frac{1}{2}\left\Vert \mathbf{y}-\mathbf{X}\boldsymbol{\beta}\right\Vert ^{2}+\lambda\...
0
votes
1answer
31 views

How to apply the uniform shrinkage factor to the logistic regression to get the updated coefficients and intecept in R?

Hope to ask a bit about uniform shrinkage factor in updating the coefficients and intercept of prediction model: I have built up a prediction model with "rms" and got the uniform (global) ...
0
votes
1answer
24 views

Not sure if Cox or Gaussian glmnet regression is appropriate

I am working with data from an outpatient physical rehab clinic and am trying to figure out what variables can predict the number of hours of service provided (DV). We have two years of data with a ...
2
votes
1answer
127 views

When in Adaptive LASSO process does it make sense to constrain control variable lambdas to 0?

Lets take this example for how to conduct an adaptive LASSO. Essentially an initial model is fit using ridge regression. Then, a LASSO is fit, in which the value of lambda is tuned individually for ...
2
votes
2answers
104 views

L1 Regularization vs Constraint

It is my understanding that the these: \begin{equation} min_{x}f(x)+\lambda\vert\vert x\vert\vert _{L_{1}} \end{equation} \begin{equation} min_{x}f(x) \text{,}\hspace{5pt}\vert\vert x\vert\vert _{L_{1}...
1
vote
0answers
18 views

How does glmnet put constraints on coefficient upper and lower bounds?

Based on glmnet documentation at https://web.stanford.edu/~hastie/glmnet/glmnet_alpha.html Coefficient upper and lower bounds These are recently added features that enhance the scope of the models. ...
0
votes
0answers
12 views

Forcing control variables into an Adaptive LASSO model?

I am using code modified from here to perform an Adaptive LASSO analysis. My question is: if I wanted to "force" some control variables into the model. Is there a way to do this? Using the ...
1
vote
0answers
21 views

Entropy regularization versus L2 norm regularization?

In multiple regression problems, the decision variable, coefficients $\beta$, can be regularized by its L2 (Euclidean) norm, shown below (in the second term) for least squares regression. This type of ...
0
votes
0answers
25 views

Shrunken covariance matrix in the Sparse inverse covariance selection

The original version of the L1 regularization method uses sample covariance matrix ${\mathbf{S}}$ as follows: \begin{equation} \hat{\mathbf{\Omega}}= argmin_{\mathbf{\Theta}\succ 0} \bigg(tr(\...
1
vote
1answer
29 views

what's the purpose of trying to have small model weights via regularization?

In machine learning, it is often advised to use weight regularization so that the model parameters don't grow big while training. I am not convinced that having small weights improves the model's ...
0
votes
1answer
40 views

Is there any need for regularization in an overdetermined multiple regression problerm?

Supposed I have a small number of features, say 4 or 5, and I have hundreds of data points. That is, I am in an over-determined situation. Is there any benefit to using regularization in this setting ...
1
vote
0answers
15 views

Fast solvers for LASSO-type non-convex optimization problems

Given $y \in \mathbb{R}^{n \times 1}, X \in \mathbb{R}^{n \times p}$, $p > n$, assume a LASSO-type optimization problem in the form of $$ \frac{1}{2}||y - X \beta ||_{2}^{2} + \sum_{i}p(|\beta_i|; \...
0
votes
1answer
37 views

What should you do if you have too many features in your dataset, dimensionality reduction or regularization?

I have just started machine learning and was asked this concept-based question, "Suppose you are working on a stock market prediction model and the data you collected have millions of features, ...
2
votes
0answers
20 views

Covariance matrix of regularized likelihood

My question is how to estimate the covariance matrix of parameters in a regularized likelihood maximization. Lets assume we have constructed some negative log-likelihood with a set of parameters and a ...
0
votes
0answers
2 views

Should transformed models be back-transfored before penalization? And should ensemble forecasts from these also be penalized? From the same data?

I have generated a number of regression models in an effort to predict a particular outcome. All the models are based on the same data over the same period. They include, e.g., a vector error ...
0
votes
0answers
25 views

Does regularized online machine learning exist?

For tikhonov regularization, we add a regularization term to the least squares objective function for numerical stability. In online machine learning you minimize the regret which is just a difference ...
0
votes
0answers
14 views

Fitting shifted $\ell 1 $ regularized logistic regression: Which parts stay in the final fitted model?

I want to fit a logistic regression model with an additional "shift" term $\boldsymbol{S(\beta)}$ linear in $\beta$. (based on Wang, J., Kolar, M., Srebro, N., & Zhang, T. (2017). ...
1
vote
0answers
31 views

Can we make PCA solutions unique by regularization?

PCA solutions are not unique, and are not continuous, i.e. we don't have that if we decompose $X^TX=U\Sigma V$ and then change $X^TX$ by a small amount, the updated $U,\Sigma,V$ will be close to the ...
1
vote
0answers
42 views

Loss function vs regularizer

Probably this is an easy thing. But I am struggling with the concept of regularization and loss function. Let's assume that we want a robust sparse solution of a linear regression problem. The data is ...
2
votes
0answers
26 views

Regression prediction with combination of predictors

I have one linear regression model: y ~ x1 + x2 (1) and now let $x_3 = x_1+x_2$, $x_4=x_1-x_2$, to form a new regression y ~ x3 + x4 (2), would the prediction of (1) and (2) be the same? If I add L1 ...
0
votes
1answer
24 views

Penalized Canonical Correlation in R with PMA Module

I am trying to use sparse canonical correlation analysis as implemented in the R PMA package. I'm finding that the correlations output by the package seem slightly inconsistent with the ones you would ...
0
votes
0answers
23 views

Optimal GCV Ridge Regression in Closed Form

I’ve searched all over but I can’t find anywhere a closed form solution to the optimal penalty term in ridge regression using generalized cross validation as the objective function. I’m starting to ...
1
vote
0answers
12 views

How to re-scale the inputs for forward-propagation and backpropagation in the drop out?

Assume $p$ is the keep probability for drop out, for the forward-propagation, we do the scaling for the inputs as $A_r = A/p$. In the backpropagation, as many other people said (dropout: forward prop ...
1
vote
1answer
27 views

Is it appropriate to use regularised regression for low-dimensional N>>p variable selection problems?

I am currently examining which of sixteen variables are the most important in predicting a binary outcome. There are 907 observations, so obviously $N$ is much larger than $p$ In the last six months ...
0
votes
0answers
28 views

Simple way to know how many models your dataset needs

I have a dataset and was wondering if there is a simple way to know how to break it up so that different models can be used for different subsets based upon the underlying mechanisms. Say I have a ...
2
votes
0answers
58 views

k fold cross validation with regularisation R code - unsure whether it is correct

I'm working through a book called A First Course In Machine Learning. The first chapter introduces K-fold Cross Validation as well as other validation methods, and then has a few paragraphs on ...

1
2 3 4 5
24