Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

0
votes
0answers
53 views

Nonparametric Bayesian estimation of several black-box functions of different variables from their noisy sums

In order to introduce my problem, let’s start with the nonparametric estimation of a single unknown/black-box function $f:{\Omega _f} \to \mathbb{R}$ of a discrete variable $x$ in a finite domain ${\...
1
vote
1answer
48 views

Ridge/Lasso regression negative Lambda

I am here to ask something that I think it is interesting, first I just read about the shrinkage using the Ridge or Lasso regression by using the lambda as the penalty to introduce a little bias that ...
1
vote
0answers
31 views

Lasso vs. Linear Regression with features selected by lasso: what to expect

I cross validate a lasso regression with multiple values of lambda (the multiplier for the penalty) e.g. from 0.00001 to 100 I get the best solution is with a certain lambda, e.g. 0.7 Given some of ...
0
votes
0answers
24 views

How to penalize change of states in Hidden Markov model?

I'm trying to fit a HMM on a sequence of observations and I would like to introduce some constraints that would penalize an excessive number of changes of state in the complete sequence (where "change"...
0
votes
0answers
8 views

What causes a high testing deviance vs. training deviance in a gradient boosting classifier?

My main goal is to classify multi-class data using supervised learning. Currently, I am looking into GradientBoostingClassifier as the estimator. I want to make sure I am selecting the model ...
0
votes
1answer
28 views

Adding more samples to ordinary regression is equall to ridge regression [duplicate]

I am a beginner in machine learning. I have a question why adding more samples to a data set is equal to adding regularization term to the ordinary least squares loss function? (In other words why can ...
1
vote
0answers
19 views

Could a mismatch between loss functions used for fitting vs. tuning parameter selection be justified?

Could it make sense (and if so, under what circumstances) to define a penalized estimator based on one loss function but then select its tuning parameter (say, via cross validation) based on another ...
1
vote
0answers
10 views

How is the location of this point derived?

Given a vector $\mathbf{w_{\theta}}$ which is normal to the orange line: And two point $\mathbf{x}$ how is $\mathbf{x_p}$ derived? I understand that the vector $\mathbf{w_{\theta}}$ is the vector ...
4
votes
0answers
77 views

Is there a theoretical reason why simple models perform better than complex models on time series forecasting tasks?

Empirically, simple forecasting methods such as damped trend exponential smoothing, STL, or even random walks typically outperform more complex models such as higher order ARIMA models or ML based ...
1
vote
0answers
34 views

Intuitive definition of manifold regularization for neural networks [closed]

I am studying the deep neural networks and I have been assigned a project on the manifold regularized neural networks, in particular the definition is as follows: "enabling semi-supervised learning ...
0
votes
1answer
61 views

stochastic gradient descent of ridge regression when regularization parameter is very big

As we know, the gradient of ridge regression is: $$ g = \frac{\partial L}{\partial \theta} = -X_i^T(y_i-X_i\theta)+2\lambda\theta $$ where $X_i$ is the $i$th training sample. The update of $\theta$ is ...
4
votes
1answer
60 views

MAP estimation as regularisation of MLE

Going through the Wikipedia article on Maximum a posteriori estimation, it got confusing after reading this: It is closely related to the method of maximum likelihood (ML) estimation, but employs ...
1
vote
0answers
16 views

Training an ANN further once it reaches 100 % accuracy on training set

I have a very simply question: Does it make sense to further train an ANN once it reaches an accuracy of 100 % on the training data? I'm facing a binary classification problem and read this article ...
1
vote
0answers
15 views

What can be implied from loss function that its regularizer needs large coefficient

I run loss function with l1-norm as regularizer for source separation. $min\sum_{i=1}^{n} V(f(x_{i}), y_{i}) + \lambda R(f)$ I varied the coefficient ($\lambda$) from 0 to 1e14. The results ($\frac{\...
0
votes
0answers
10 views

What is difference in feature selection by using l0 and l1 regularization?

Both l0 and l1 can be used for feature selection, so what is the difference between them?
1
vote
2answers
35 views

Variables reduction required for Random Forest, Boosting, L1, L2 regularization

I have close to 10,000 variables. I know how random forest/XGB picks number of variables randomly for building the tree. Also regularization takes care of significance of variable by its coefficient. ...
11
votes
4answers
417 views

What causes lasso to be unstable for feature selection?

In compressed sensing, there is a theorem guarantee that $$\text{argmin} \Vert c \Vert_1\\ \text{subject to } y = Xc $$ has a unique sparse solution $c$ (See appendix for more details). Is there a ...
0
votes
1answer
36 views

Does regularization leads to stucking in local minima?

I frequently hear some very conflicting claims regarding deep learning algorithms. Currently, I am a bit confused on the role of regularization. I have listed my queries below regarding regularization ...
0
votes
1answer
23 views

Regression regularization penalty center at w0 instead of 0 [duplicate]

How do I regress with regularization penalty term lambda * (w - w0)^2 instead of lambda * w^2? Is there any package to do it? I ...
1
vote
1answer
55 views

$\lambda \Vert k \Vert_0$ or $\Vert k \Vert_0 \leqslant n$

Say $Y \in \Bbb R^n$ is a response, $X = (x_1, x_2, \cdots, x_m)^T \in \Bbb R^{n \times m}$ are predictors. In a linear regression problem, we want to add an $l_0$ regularization for feature selection....
6
votes
0answers
116 views

Sparse linear regression 0-norm and 1-norm

We have a response $Y \in \Bbb R^n$ and predictors $X = (x_1, x_2, \cdots, x_m)^T \in \Bbb R^{n \times m}$ The problem we want to solve is $$\text{argmin}_{k \in \Bbb R^{m}} (\Vert Y - Xk \Vert_2^2 +...
0
votes
0answers
15 views

How to deal with numeric instability in stochastic gradient descent?

Imagine that we try to perform sgd using a gradient that takes very small or very large values (e.g. it is a product of many terms that are larger than 1). Is there a standard approach to deal with ...
4
votes
1answer
78 views

Linear regression with $l_0$ regularization

In a linear regression problem with sparsity constraint, $P = (P_1, \cdots, P_N)^{T}$ is the column vector of the outputs, and $D = (d_{j, k})$ is the $(N \times M)$- dimensional matrix of inputs. The ...
0
votes
1answer
26 views

Why aren't there there two regularization terms in SVC?

I'm looking at the formulation for SVC as stated on sklearn's website (http://scikit-learn.org/stable/modules/svm.html#svc). The loss function here minimizes a "flatness term" and a (regularized) "...
9
votes
2answers
845 views

Why does the L2 norm loss have a unique solution and the L1 norm loss have possibly multiple solutions?

http://www.chioka.in/differences-between-l1-and-l2-as-loss-function-and-regularization/ If you look at the top of this post, the writer mentions that L2 norm has a unique solution and L1 norm has ...
1
vote
0answers
55 views

Difference between random effect and fixed effect with regularization/prior

Let's say I have a random effect intercept. For example: lme4::lmer(yield ~ 1 + (1|Batch)) How is that different than just ordinary regression using ...
0
votes
0answers
41 views

Weighting matrix in regularization term

While learning about regularization I came across this MATLAB article on regularization. It says The penalty term is made more effective by using a positive definite matrix R, which allows ...
1
vote
0answers
17 views

At a loss regarding feature selection vs coefficient estimation. Can you ever re-do the latter after the former?

I'm looking at a binary classification problem where p>>>n (9,000 gene expression variables for 290 patients who either have or don't have disease). I hypothesized that it would be easy to find "...
0
votes
0answers
33 views

Bayesian linear regression on complex : how to use the prior laws and more

My model is as follows : With $y\in\mathbb{C}^{40},A\in\mathbb{C}^{40\times10},x\in\mathbb{C}^{10},b\in\mathbb{C}^{40}$ : $$y=Ax+b$$ $y$ and $A$ are known and I have a normal prior law on the module ...
2
votes
2answers
97 views

How does lasso regularization select the “less important” features?

I'm just starting in machine learning and I can't figure out how does lasso method find which features are redundant to shrink their coefficients to zero?
5
votes
2answers
141 views

Why is multicollinearity so bad for machine learning models and what can we do about it?

Why is multicollinearity so bad for machine learning models? Is there ever a time when we can ignore multicollinearity? How does regularization ($L_1$, $L_2$) help us deal with multicollinearity?
0
votes
0answers
22 views

Why should each layer's child network output be close to parent network's output for variance regularizer?

I am reading up on PEA (Pseudo ensemble agreement) regularizer. specificaly in the neural networks domain. It introduces the concept of perturbing the model a little and forcing the model to make ...
0
votes
0answers
36 views

What is “Entropic Capacity”?

I found this term on the Keras blog website, quoted below Your main focus for fighting overfitting should be the entropic capacity of your model --how much information your model is allowed to ...
0
votes
1answer
72 views

How do Lasso coefficients change as lambda approaches infinity [closed]

I have encountered such a problem. I think 2, 3 and 4 pictures are true, but no. Can anyone help?
6
votes
1answer
703 views

How Ridge or Lasso regression really work?

Very basic question here, but I would like to understand (not mathematically) how the fact to add a "penalty" (sum of squared coeff. times a scalar) to the residual sum of square can reduce big ...
0
votes
0answers
33 views

Is the regularization term necessary when classifying one feature?

I'm using the Stochastic Gradient Descent linear classifier (implemented in Scikit-learn) to classify an image pixel by pixel. So my dataset has only one feature, ...
0
votes
0answers
82 views

Soft-thresholding for the LASSO with complex valued data

I'm currently implementing coordinate descent for the LASSO with complex-valued data. For this, one needs a complex version of the soft-thresholding operator, which seems hardly available on the net. ...
1
vote
0answers
39 views

Relationship between regularization parameter in Ridge/Lasso with budget constraint

The equation for lasso and ridge regression are given as follows in the ISLR textbook: The dual form of the above equations are given in terms of budget as below: I am wondering if there is a ...
0
votes
1answer
25 views

Stochastic gradient descent update

Equation 93 of Chapter 3 of Michael Nielsen's neural networks book describes the stochastic gradient descent update rule as the following: $w \leftarrow (1-\frac{\eta\lambda}{n})w - \frac{\eta}{m}\...
2
votes
1answer
107 views

Why weights are not negative in Lasso regression?

I can understand that lasso could make some weights to zero and prevent over-fitting. But for all the figure I see about lasso regression, weight will stay at zero once it reach zero and increasing ...
1
vote
0answers
19 views

Preventing logloss from over-optimizing on some points (or groups of points)

I've got a neural network classification model that does ok in terms of binary logloss. However I watched the training quite closely, outputting the logloss of each batch (batches are 100 samples) and ...
0
votes
1answer
91 views

Should we average weight decay loss in neural network?

In a typical neural network, which way is the common way to add regularization? Assuming regression task, regression error loss is Mean-squared-error Then we can have two choice of regularization ...
0
votes
0answers
22 views

gLASSO - regularization makes upper and lower triangles of estimated matrix equal

The overall goal is to get an estimate of the precision matrix (inverse of a covariance matrix given some nxp data), which can be translated into a graph showing the partial correlation relationships ...
4
votes
1answer
185 views

Question about Xgboost paper weights and decision-rules

Can someone please explain what the weight $w$ is doing and how it works here? I also didn't understand how $q$ transforms an $m$-dimensional vector to $T$. Edit Answer by usεr11852 is pretty good. ...
0
votes
0answers
27 views

Why do we not take the average of the regularization hyperparameters chosen by hold-out/cross validation?

I will talk restrict to hold-out for simplicity but my question applies to cross validation too. Say we have a regularization hyper-parameters we are looking for $\lambda$. We choose it by training ...
1
vote
1answer
37 views

How to conduct confirmatory factor analysis with small sample size?

I plan to conduct a confirmatory factor analysis, wherein there are 12 observed variables and 3 latent variables. My sample size is 30. However, I read that to conduct a factor analysis, the sample ...
2
votes
1answer
49 views

why small weights are preferred in neural networks

I'm viewing the CS231n lectures and trying to understand some of the regularization concepts. I think I understand the rationale behind "spread out" weights that are preferred by L2 regularization, e....
2
votes
1answer
163 views

What are the implications of scaling the features to xgboost?

Doing research about the xgboost algorithm I went through the documentation. I have heard that xgboost does not care much about the scale of the input features In this approach trees are ...
0
votes
0answers
32 views

What are the Advantages of Using Both $L_1$ and $L_2$ for Regularization? [duplicate]

This is what I found to compare the two: But I could not find the advantages of using both, for example for a linear regression model?
0
votes
0answers
176 views

Why are the coefficients in a multinomial logistic regression a matrix?

I am conducting an analysis in which I have 3 different groups and a set of 80 continuous variables that I think can discriminate between the 3 groups. I want to: see if indeed I can discriminate the ...