Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Inclusion of additional constraints (typically a penalty for complexity) in the model fitting process. Used to prevent overfitting / enhance predictive accuracy.

7
votes
0answers
979 views

Post processing random forests using regularised regression: what about bias?

I have been playing around with post processing the results of the random forest for regression machine learning algorithm in order to try and do better than the default mean of all trees prediction. ...
3
votes
1answer
650 views

Standard error of parameter estimates in regularized regression

In a regularized linear regression model (e.g., ridge regression, lasso, etc.), what is the best way to obtain standard errors for parameter estimates? If cross-validation is used, is it ...
12
votes
1answer
5k views

Regularized bayesian logistic regression in JAGS

There are several math-heavy papers that describe the Bayesian Lasso, but I want tested, correct JAGS code that I can use. Could someone post sample BUGS / JAGS code that implements regularized ...
3
votes
0answers
78 views

What's a good range of weights to evaluate for $L_2$ regularized logistic regression?

I want to find a weight that minimizes an averaged cross validated misclassification score from an $L_2$ logistic regression classifier. Obviously, the search space for the weights should be bounded ...
9
votes
2answers
362 views

Regularization $L_1$ norm and $L_2$ norm empirical study

There are many methods to perform regularization -- $L_0$, $L_1$, and $L_2$ norm based regularization for example. According to Friedman Hastie & Tibsharani, the best regularizer depends on the ...
1
vote
1answer
103 views

What does the index variable k define in the Lasso regularization function

In the Lasso L1 regularization, from where comes the value of the variable $k$ in the second part of the function? Why isn't it $n$, too? $$L(\beta) = \sum_{i=1}^n (y_i - \phi(x_i)^T \cdot \beta)^2 + ...
2
votes
1answer
219 views

Model function for discovering irrelevant dimensions with L1 regularization

For homework I have been given a 20-dimensional input $x \in \mathbb{R}^{20}$, many of which are suspected to be irrelevant. I tried using L1-norm Lasso regularization to uncover which dimensions ...
4
votes
1answer
3k views

Using glmnet to solve the LASSO problem

I have recently been made aware of the Lasso algorithm and found that the package glmnet can be used to solve it. (I have the glmnet package on R). If I have a matrix $A$ and a vector $y$ how do I ...
2
votes
1answer
453 views

Non-linear regularized SVM implementation

Just a general question. Are there any good non-linear SVM (kernelized) implementations that include a regularization component (e.g. $L_1$, SCAD etc)? I've been looking around but man there are a lot ...
5
votes
4answers
1k views

Bayesian prior corresponding to penalized regression coefficients

I'm working on a Bayesian Regression problem where I would like to estimate the beta coefficients subject to this constraint (penalty): $\sum|\beta_i|<C$ or similarly $\sum \beta_i^2<C$ Which ...
3
votes
1answer
109 views

Problem specific regularization

I've been reading a lot recently about the concept of joint regularization in computer vision. Joint regularization builds on the observation that when learning multiple related concepts, for example "...
1
vote
1answer
257 views

High dimensional time series

I'm not sure what words I should look for. I have an under determined dataset of 8000 correlated variables (sales) over 12 months (ie 12 observations for each variable). And I basically want to ...
15
votes
1answer
9k views

Need for centering and standardizing data in regression

Consider linear regression with some regularization: E.g. Find $x$ that minimizes $||Ax - b||^2+\lambda||x||_1$ Usually, columns of A are standardized to have zero mean and unit norm, while $b$ is ...
5
votes
1answer
305 views

When is there a representer theorem?

The case of regularization in a hilbert space is considered---an optimization problem with an error term and a Tikhonov-regularizer. In the article "When is there a representer theorem" it is stated ...
4
votes
1answer
590 views

Feature selection with k-fold cross-validated least angle regression

I am using the least angle regression (LARS) to extract the most important predictors ($x_1, x_2,...,x_p$) for my response variable ($y$). I have seven predictors ($x_1,x_2,...,x_7$) for each ...
6
votes
3answers
7k views

Cross validation with two parameters: elastic net case

I want to know the cross validation procedure to find the two parameters of elastic net presented by Zou and Hastie on the prostate dataset as example. I can't improve the error rate lasso with k-fold ...
75
votes
2answers
73k views

What is the lasso in regression analysis?

I'm looking for a non-technical definition of the lasso and what it is used for.
3
votes
2answers
2k views

How to calculate derivative of the contractive auto-encoder regularization term?

Setup I found a paper on that has a varient on normal auto-encoders (contractive) which for its gradient uses the following regularization penalty: $$\left|\left|J_f(x)\right|\right|^2_F = \sum_{ij}...
8
votes
2answers
3k views

How is the intercept computed in GLMnet?

I've been implementing the GLMNET version of elastic net for linear regression with another software than R. I compared my results with the R function glmnet in lasso mode on diabetes data. The ...
2
votes
1answer
1k views

Cross-validating for model parameters with time series

This question's context is time series forecasting using regression, with multivariate training data. With a regularization method like LARS w/ LASSO, elastic net, or ridge, we need to decide on the ...
2
votes
1answer
425 views

Number of segments to divide a time-series

Suppose we have time-series $ X_t $ and it has the following decomposition $$X_t=\mu + \varepsilon_t,$$ where $\mu$ is a mean and $\varepsilon_t$ - the error term. The model complexity will ...
27
votes
2answers
4k views

Fitting an ARIMAX model with regularization or penalization (e.g. with the lasso, elastic net, or ridge regression)

I use the auto.arima() function in the forecast package to fit ARMAX models with a variety of covariates. However, I often have a large number of variables to select from and usually end up with a ...
7
votes
1answer
533 views

Regularized fit from summarized data: choosing the parameter

Following on from my earlier question, the solution to the normal equations for ridge regression is given by: $$\hat{\beta}_\lambda = (X^TX+\lambda I)^{-1}X^Ty$$ Could you offer any guidance for ...
5
votes
1answer
216 views

Regularized fit from summarized data

I have a multiple linear regression problem $y=X\beta+\epsilon$. The number of observations $m$ is large, so by the time the data gets to me it's been summarized into: $m$ $X^TX$ $X^Ty$ $y^Ty$ $\sum_{...
3
votes
1answer
314 views

Stopping condition for least-angle regression

Suppose I have data with p explanatory variables, and I want to use a LARS algorithm to build a model. Do I Run LARS until all p variables have been added to my model, and the correlation between ...
11
votes
3answers
1k views

GLM after model selection or regularization

I would like to pose this question in two parts. Both deal with a generalized linear model, but the first deals with model selection and the other deals with regularization. Background: I utilize ...
13
votes
3answers
8k views

GLMNET or LARS for computing LASSO solutions?

I would like to get the coefficients for the LASSO problem $$||Y-X\beta||+\lambda ||\beta||_1.$$ The problem is that glmnet and lars functions give different answers. For the glmnet function I ask ...
5
votes
2answers
492 views

Stochastic coordinate descent for $\ell_1$ regularization

I recently came across the following paper: "Stochastic Methods for $\ell_1$ Regularized Loss Minimization" by Shai Shalev-Shwartz and Ambuj Tewari, ICML 2009. In the paper, the authors propose a ...
10
votes
4answers
5k views

Lasso fitting by coordinate descent: open-source implementations?

What open-source implementations -- in any language -- exist out there that can compute lasso regularisation paths for linear regression by coordinate descent? So far I am aware of: glmnet scikits....
68
votes
5answers
38k views

What is regularization in plain english?

Unlike other articles, I found the wikipedia entry for this subject unreadable for a non-math person (like me). I understood the basic idea, that you favor models with fewer rules. What I don't get ...
12
votes
1answer
3k views

LARS vs coordinate descent for the lasso

What are the pros and cons of using LARS [1] versus using coordinate descent for fitting L1-regularized linear regression? I am mainly interested in performance aspects (my problems tend to have <...
9
votes
2answers
5k views

Coordinate descent for the lasso or elastic net

Are there any good papers or books dealing with the use of coordinate descent for L1 (lasso) and/or elastic net regularization for linear regression problems?