# Tagged Questions

The tag has no wiki summary.

0answers
11 views

### Calculating Second-Order Tikhonov Regularization Parameter in Mathematica

I am trying to map slowness of underwater sound velocity in a river using some tomographic device. The location of each acoustic receiver/transmitter is shown in the picture below To find the ...
3answers
63 views

### Different regularization parameter per parameter

I have never seen a regularization parameter (usually lambda or alpha) be different for each parameter. People consider different regularization parameters, but I believe they penalize all the ...
1answer
59 views

### How to interpret glmnet?

I am trying to fit a multivariate linear regression model with approximately 60 predictor variables and 30 observations, so I am using the glmnet package for regularized regression because p>n. I ...
0answers
10 views

### Why the maxStages argument in biglars.fit does not work

Why doesn't the biglars.fit function work when maxStages is specified? I've tried multiple values and multiple ways of casting $y$ but it doesn't work. ...
0answers
18 views

### Standardizing response variable in shrinkage/regularization

I know that I should standardize my predictors before estimating something like Lasso. But what about the response variable? Do I standardise this as well? Only ...
2answers
253 views

### Why does the Lasso provide Variable Selection?

I've been reading Elements of Statistical Learning, and I would like to know why the Lasso provides variable selection and ridge regression doesn't. Both methods minimize the residual sum of squares ...
0answers
29 views

### Comparing different types of losses as functions of lambda?

The usual pictures we see when dealing with different loss functions look similar to this: Here we see y*f(x) on the x-axis with an error associated with it. Suppose I have a logistic regression ...
0answers
73 views

### When would I choose Lasso over Elastic Net

What are the scenarios where Lasso is likely to perform better than Elastic Net (out of sample prediction)?
0answers
34 views

### $\lambda$ in $\mathcal{L}_1$ penalized logistic regression and the likelihood on the training dataset

Define $J(w) = -l({w},\mathcal{D})+\lambda||{w}||_2^2$, then is it true that $l(\hat{{w}},\mathcal{D}_{train})$ always increase as we increase $\lambda$?
0answers
25 views

### Kernel/Basis function design with regularizer

I am solving this problem: $$\sum_i \parallel f(x_i)- y_i\parallel_2^2 + \lambda <\psi f, \psi f>_{L_2}^2$$ where the second part $<\psi f, \psi f>_2^2$ is regularizer using the linear ...
1answer
70 views

### Running regularized logistic regressions on very large datasets

I want to run a regularized logistic regression on a dataset with 25 million observations and about a 1000 mostly non-sparse columns with non-ignorable weights. My first choice would be BayesGLM, ...
1answer
263 views

### Feature selection & model with glmnet on Methylation data (p>>N)

I would like to use GLM and Elastic Net to select those relevant features + build a linear regression model (i.e., both prediction and understanding, so it would be better to be left with relatively ...
1answer
105 views

### Neural Networks: weight change momentum and weight decay

Momentum $\alpha$ is used to diminish the fluctuations in weight changes over consecutive iterations: \Delta\omega_i(t+1) = \omega_i - \eta\frac{\partial E}{\partial w_i} + \alpha \Delta ...
1answer
101 views

### How big are regularization parameters values?

I wanted to know how big are the regularization parameter values for ridge or lasso. I have seen most of the places generally using values like 0.1 or 0.01 but in some of my experiments the cross ...
0answers
44 views

### LASSO method: prediction for multi-dimentional reponses

I have a feature matrix, that is 'X' 2000 (observation) x 200 (variable). I also have a response matrix, that is 'Y' 2000 (response) x 2 (variable). I would like to apply LASSO method to the data ...
1answer
154 views

### What does it mean if all the coefficient estimates in a lasso regression converge to zero?

I attempted to run lasso on a 12 X 52 matrix (11 predictors) using this MATLAB function http://www.mathworks.com.au/help/stats/lasso.html. I found that the results converged to zero. How should I ...
1answer
39 views

### Is it bad to leave in poor predictors in regularized multiple regression?

There are some variables that I measured but strongly suspect are useless because (for example) almost all my data points scored the same on that (binary) variable. It's been put to me that I may ...
0answers
64 views

### Needle-in-a-haystack Regularized Regression

I'm in a setting where I am trying to model a continuous output variable given ~100 variables and ~100k datapoints. The signal-to-noise ratio is extremely low, and colinearity is very high. Among the ...
0answers
27 views

### Maximum risk and sparse estimation

On Larry Wasserman's blog, he talks about the "Steep price of sparsity" here: http://normaldeviate.wordpress.com/2013/07/27/the-steep-price-of-sparsity/ In it, he points out that a sparse estimation ...
2answers
175 views

### Regularization and feature scaling in online learning?

Let's say I have a logistic regression classifier. In normal batch learning, I'd have a regularizer term to prevent overfitting and keep my weights small. I'd also normalize and scale my features. In ...
3answers
270 views

### (Why) do overfitted models tend to have large coefficients?

I imagine that the larger a coefficient on a variable is, the more ability the model has to "swing" in that dimension, providing an increased opportunity to fit noise. Although I think I've got a ...
3answers
128 views

### Why does regularization of coefficient magnitude improve the generalization of linear regression?

What is the basic argument upon which ridge and lasso regression are based on? I went through Tikhonov regularization wiki where it was mentioned that In many cases, tikhonov matrix is chosen as ...
1answer
39 views

### Is regularization required with overdetermined data

I'm doing least squares estimation on large set of data and I started to wonder whether I should regularize my OLS estimator. My professor told me that this isn't necessary, because the data is ...
0answers
36 views

### Sparsity regularization for eigenvectors

One way to think about finding the eigenvectors of a matrix $A$ is that they are the critical points of the functional $\vec x^\top A \vec x$ subject to $\|\vec x\|_2=1$. To regularize this problem, ...
1answer
91 views

### Effect of features that are highly correlated with each other on a decision tree

I have a dataset of roughly 500 features and am training a binary classifier using GBM - gradient boosted machines, an ensemble of decision trees. Of these 500 variables, I am sure some are highly ...
0answers
26 views

### Use GLM weights to regularize noise

I do a GLM containing 8 predictors on a multivariate data set. Six of these predictors encode effects that have actually been manipulated in my experiment (effects of interest), the other two ...
1answer
357 views

### Coefficients paths – comparison of ridge, lasso and elastic net regression

I would like to compare models selected with ridge, lasso and elastic net. Fig. below shows coefficients paths using all 3 methods: ridge (Fig A, alpha=0), lasso (Fig B; alpha=1) and elastic net (Fig ...
0answers
20 views

### Feature generation with Chebychev polyomials?

A common approach for more complex models is to use additional polynomial features $x_i^2$, $x_i^3$,... If I do a logistic regression or an SVM, would it have any advantage to use Chebychev ...
0answers
55 views

### GBDT and model building: How am I overfitting?

Here's my situation: Binary classification and I've got a training set of roughly 250k samples and 10 features, and a validation set of roughly 100k with the same number of features. I'm fitting GBDT ...
0answers
59 views

### Penalized methods for categorical data: combining levels in a factor

Penalized models can be used to estimate models where the number of parameters is equal to or even greater than the sample size. This situation can arise in log-linear models of large sparse tables of ...
1answer
31 views

### Confusion related to regularization parameter selection by cross validation

I can see lots of paper mentioning they selected some parameters like regularization parameter $\lambda$ by cross validation. What do they mean by that?
0answers
9 views

### Do I use regularizer when I measure validation error?

I have a cost function with a regularizer term, and I'd like to find to optimal hyperparameters for the regularizer term. So I train with different parameter, but when I measure the validation error, ...
0answers
61 views

### Questions regarding predict.glmpath()

I'm trying to do LASSO in R with the package glmpath. However, I'm not sure if I am using the accompanying prediction function predict.glmpath() correctly. Suppose I fit some regularized binomial ...
4answers
432 views

### What does “degree of freedom” mean in neural networks?

In Bishop's book "Pattern Classification and Machine Learning", it describes a technique for regularization in the context of neural networks. However, I don't understand a paragraph describing that ...
0answers
77 views

### Forward Stepwise selection

I am assuming the following model: $Y = \beta X + \epsilon$ Here both $X$ and $Y$ are matrices. I fit the least squares model without any regularization and get the matrix $\beta$. I would like to ...
2answers
284 views

### Alternatives to glmnet for feature selection on data with lots of NAs

I have a surgical database in which there are approximately 100,000 observations and 200 features. Each observation corresponds to a separate patient while the features correspond to either ...
0answers
99 views

### Ridge regression on subset of variables using SVD

I am trying to figure out an algorithm using singular value decomposition to run a modification of ridge regression in which only some of the variables are penalized. I want the output to match the ...
1answer
61 views

### Question on the usage of regularization in applied statistics/science

I was reading the paper A significance test for the lasso'' by Lockhart, Tibshirani et al and was considering the issue of applying regularization in the applied sciences (for example, behavioral ...
1answer
179 views

### Classification with 3 groups, repeated measurements, missing values, more predictors than subjects

I am working on a classification problem with the following characteristics: Individuals belong to one of three groups. The groups are "somewhat ordinal": controls, subclinical and clinical group. ...
1answer
101 views

3answers
202 views

### Sparsity-inducing regularization for stochastic matrices

It is well-known (e.g. in the field of compressive sensing) that the $L_1$ norm is "sparsity-inducing," in the sense that if we minimize the functional (for fixed matrix $A$ and vector $\vec{b}$) ...