Stack Exchange Network

Stack Exchange network consists of 174 Q&A communities including Stack Overflow, the largest, most trusted online community for developers to learn, share their knowledge, and build their careers.

Visit Stack Exchange

Questions tagged [penalized]

Methods of modifying objective functions to control the solutions of optimization problems.

0
votes
0answers
10 views

CV.GLMNET warnings convergence

I get the warnings when I try to perform a cross validation with cv.glment for a Logistic penalized model ...
0
votes
0answers
36 views

Finding the “optimal” non linear relationship between two variables

I am looking for finding associations between a binary outcome regarding women fertility and several potential risk factors. Since this study is quite exploratory, I was planning to include all my ...
1
vote
0answers
14 views

Removing the intercept term for penalized logistic regression

I am working on lasso logistic regression and am trying to remove the intercept term from the penalty function. I have tried to use the mean centering theory but for logistic regression it can not be ...
2
votes
1answer
59 views

Large value of $X\beta$ in logistic regression?

In logistic regression, the probability is obtained from $$ Pr = \frac{\exp(X\beta)}{1 + \exp(X\beta)} ~~~~ (1) $$ From the plot below, it is obvious that if $X\beta$ > 10, the probability approaches ...
0
votes
1answer
49 views
+50

Confidence limits for constrained penalized log likelihood model

I am estimating parameter $\beta$ as: \begin{align} \hat \beta &= \mathop{\mathrm{arg\,max}}_\beta \;\; l(\beta;X,y) - \frac{\lambda}{2}\left(\tilde y-g(\beta,\tilde X)\right)^\prime C^\prime C\...
0
votes
0answers
39 views

Adding additional constrains to OpenAi Gym

I'm currently working trough some examples which should finally end in a DQN Reinforcement Learning for the CartPole example in the openAI-Gym. Copied some code from GitHub which isn't deep yet: <...
3
votes
0answers
60 views

Calibration of penalized (LASSO or ELasticNet) logistic regression models

I would be very grateful for any help me with the following general query regarding calibration of penalized models with a binary outcome. I would like my prediction model to be calibrated (mean ...
2
votes
0answers
46 views

Elastic Net and collinearity

I am performing elastic net for variable selection on a dataset of 95 records and 41 variables. The response is a continuous numerical. I choose the alpha and lambda parameters through 10 fold cross ...
0
votes
1answer
68 views

Elastic net visualization [closed]

Sorry for the naive question, but is there a way to display in a graph the elastic net (or penalized regression in general) results? Specifically, how can I render the coefficients of the variables?
6
votes
1answer
312 views

Does Regularized Logistic Regression Produce Calibrated Results?

It has been asked and addressed here that logistic regression modelling is calibrated already and there is no need for calibration of it. To me it seems the argument provided there does not follow ...
1
vote
1answer
59 views

Sample size calculation for elastic net regression

I am using elastic net regression to investigate the effect of preditors on the response variable while accounting for multicollinearity among the predictors. But I wish to perform a sample size ...
3
votes
0answers
16 views

How to prove oracle properties in Fan and Li (2001) paper

I am studying Fan and Li's 2001 paper "Variable selection via nonconcave penalized Likelihood andits oracle properties" but I am having troubles understanding Theorem 1 proof (page 1359). I follow the ...
3
votes
1answer
125 views

Interpreting glmnet Lasso coefficients on dummy variables (multiple levels) [duplicate]

I am trying to apply glmnet's lasso to a set of features in which there are multiple categorical variables with multiple levels. My intention is to let lasso reduce ...
4
votes
0answers
47 views

Using covariates from penalized regression model in unpenalized model

The good news where I am is that researchers are doing less stepwise covariate selection now that I've introduced penalized regression. The bad news is that researchers want to use elastic-net ...
1
vote
0answers
48 views

In LASSO, does it make sense to choose lambda based on the mean error associated with different lambda values, over multiple cross-validations?

I am running a LASSO regression, but am put off by the different values of lambda each time I run the cross-validation. Does it make sense to run a cross validation multiple times, take the mean error ...
9
votes
1answer
653 views

Why is R Squared not a good measure for regressions fit using LASSO?

I have read in several places that R Squared is not an ideal measure when a model is fit using LASSO. However, I'm not clear on exactly why that is. In addition, could you recommend the best ...
0
votes
0answers
19 views

Combining multiple imputation with penalization

I have used Frank Harrell's excellent rms package both for penalization (pentrace and setting ...
0
votes
0answers
48 views

Equivalent of using a Poisson prior in terms of a penalized regression?

I know that most penalized regressions have also a Bayesian interpretation, e.g. ridge least squares regression corresponds to the MAP estimate obtained under a Gaussian prior in a Bayesian regression,...
0
votes
0answers
82 views

How do you explain many optimal models in penalized Logistic regression?

I am building penalized logistic regression using Lasso and Ridge methods. I know that the best model chosen by the program is which has alpha = 1 and lambda = 0.06....
0
votes
0answers
35 views

“beta_given” column in the h2o.glm beta_constraints

What does the "beta_given" column do in the h2o.glm beta_constraints parameter? h2o is an open source library for machine learning algorithms. There are several online examples on how to install the ...
0
votes
0answers
137 views

Minimum number of observations for Lasso, LassoCV, LassoLarsCV

I have monthly fund returns for 3 years i.e. 36 observations (in some cases I may have 2 years i.e. 24 observations as well), and there are ~50 predictor variables. Both the dependent and independent ...
1
vote
0answers
20 views

Comparing shrunk model vs null model?

I have a null model $M_0$ with parameter $\theta$ and a more general model $M_1$ with parameters $(\theta,\alpha)$. I know that $M_0=M_1$ for $\alpha=0$. I am estimating $M_1$ using a penalized ...
6
votes
1answer
185 views

How is the minimum $\lambda$ computed in group LASSO?

The LASSO problem works by minimizing $$\min_\beta (\frac{1}{2}\left\rVert y-X\beta\right\rVert^2_2+\lambda\left\rVert\beta\right\rVert_1)$$ Here in this webpage I found that the minimal value of ...
1
vote
0answers
10 views

Elastic net with increased penalty for lower quality features

I’m building multinomial classification models using features characterized with high false-positive rate. Meaning, as the signal rate of the feature is lower (say gene expression abundance) the more ...
0
votes
0answers
25 views

Is the smallest $\lambda$ such all lasso coefficients are set to 0 the same regardless the model?

In the case of the linear regression model I have found that the smallest $\lambda$ for which all the $\beta$ coefficients are set to 0 while performing a LASSO penalization is $\lambda = \left\lVert ...
4
votes
1answer
93 views

How to find smallest $\lambda$ such all lasso coefficients are set to 0, depending on the intercept?

While bouilding a LASSO-penalized model it is well known that $\lambda =\left\lVert X^ty\right\lVert_\infty$ is the minimum value for which all the $\beta$ coefficients of the model are 0. Consider ...
1
vote
0answers
8 views

Bayesian Information Criteria variable penalisation for increasing N

I am trying to understand the Bayesian Information Criteria for model performance, given by the formula below: BIC=−2ln(L)+kln(n) I understand the aim is to minimize this function. I can see that ...
0
votes
0answers
146 views

GLMNET, Mpath Package- Lasso penalty on Poisson and Negative Binomial

I have been trying to analyze a high dimensional data (p exceeds n) with limited observation (n=50). I want to use Lasso method for variable selection and parameter estimation to develop a prediction ...
1
vote
0answers
106 views

Why is the penalty in the logistic regression likelihood ratio test different from the penalty I specified when fitting the model?

I'm fitting a penalized logistic regression model using the rms package in R. When I print the result, the penalty in the model likelihood ratio test is different from the penalty I used to fit the ...
0
votes
0answers
120 views

Penalized Negative Binomial- Subset selection and Coefficient estimation

I have a dataset with count variable of 50 observation and 260 independent variables. As the variance exceeds mean, I want to use Negative Binomial distribution. My objective is to build a model that ...
0
votes
0answers
71 views

Relation between cvm of glmnet and cvl of penalized?

I am tring to find a relation between the partial likelihood deviance = cvm (mean cross-validation error) and cvl (cross-validated likelihood) obtained by optL1 of penalized with Cox model. I checked ...
0
votes
0answers
24 views

Develop a (penalized?) SVM kernel for polynomials containing only even exponents

I want to fit an SVM classifier to some data; however, we have a priori reasons to want to limit the choice of decision boundary to equations of the form: $$ ax^{2\alpha} + by^{2\beta} = 1 \;\; a,b \...
0
votes
0answers
35 views

Ho do I generate penalty matrix for LOESS smoothing?

I am trying to optimize the Span value of LOESS fit based on GCV score. While calculating Hat matrix, I want to incorporate the penalty matrix in it. How to calculate the smoothing constraints matrix ...
3
votes
3answers
545 views

Why does L2 regularization smooth the loss surface?

Fitting neural nets with L2 penalization, I've noticed that I often attain lower in-sample mean-squared errors with higher rates of L2 "weight decay", then I do with lower rates of L2 weight decay. ...
0
votes
0answers
16 views

penalty makes k-sparse vector's non-zero elements come together

In the (multi-task) Group LASSO problem, the coefficients (coefficient matrix) are given a $\ell_{2,1}$ norm to make groups of coefficients selected or not selected together. In the theory of Group ...
1
vote
0answers
17 views

Efficient Penalized MLE to Maintain Stationarity

I am trying to penalize a likelihood function, $-\log f(Y|\theta)$, to maintain stationarity of the autoregressive process. An vector autoregressive process is said to be stationary if the spectral ...
1
vote
1answer
48 views

$\ell_0$ penalised and $K$ sparse problems

Consider the $\ell_0$ penalized problem: $$\min_{x\in \mathbb{R}^n} \frac{1}{2}\|Ax-b\|_2^2+\lambda\|x\|_0 \qquad \qquad \qquad \qquad \qquad \qquad (1)$$ and $K$-sparse problem $$\min\frac{1}{2}\|Ax-...
1
vote
1answer
131 views

Maximizing (and derivating) log-likelihood of penalized logistic regression

I'm trying to solve Exercise 18.3 of "Elements of Statistical learning" by Hastie et al. and I'd be really grateful for any hints. Show that the fitted coefficients for the regularized multiclass ...
9
votes
2answers
1k views

B-Splines VS high order polynomials in regression

I do not have a specific example or task in mind. I'm just new on using b-splines and I wanted to get a better understanding of this function in the regression context. Let's assume that we want to ...
2
votes
1answer
143 views

Confidence intervals with penalized likelihood

I am trying to perform parameter estimation using something like a maximum likelihood ratio method, however I need to add a penalty term to constrain nuisance parameters which describe certain ...
0
votes
0answers
145 views

Does the SCAD penalty solve the issue of multicollinearity?

Is the SCAD penalty by Fan and Li (2001) able to solve the issue of multicollinearity and why? I read a paper published on the Journal of Financial Econometrics, “Asset pricing with a general ...
0
votes
1answer
51 views

Group lasso penalization

In group lasso we try to minimize the following $$\frac{1}{2}\|Y - \sum_{i=1}^JX_j\beta_j\|^2 + \lambda\sum_{i=1}^J\|\beta_j\| $$ Often they change this to $$\frac{1}{2}\|Y - \sum_{i=1}^JX_j\beta_j\|^...
1
vote
1answer
57 views

imputation with penalized (Cox) regression

I'm doing penalized (elastic net) Cox regression. But I also have missing data. Now, as I understand it, the reason we do multiple imputation is that once we do single imputation, our data points are ...
4
votes
2answers
999 views

Rare Events Logistic Regression

Suppose the event of interest occurs in approximately $10 \%$ of the cases where the number of cases is around $5,000$. Should you use a penalized logistic regression for this or is regular logistic ...
1
vote
1answer
40 views

LASSO for two types of variables: optimal tuning?

I have a location and scale linear regression model: $$y_j = x_j^T\beta + e_j,$$ where $e_j\sim N(0,\sigma_j)$, and $\sigma_j = \exp(x_j^T\theta)$, $j=1,...,n$, $x_j \in{\mathbb R}_p$, $\beta \in{\...
3
votes
0answers
230 views

Standardization in penalized regression using glmnet

I want to run a penalized multinomial logit and logit regression using the glmnet package in R. I understand, that before fitting the penalized model, one should ...
5
votes
2answers
130 views

LASSO for unit norm parameters?

I have the model $y = X\theta +e$, where $\vert\vert \theta\vert\vert =1$ and $e$ are iid normal errors and $\vert\vert\cdot\vert\vert$ represents the Euclidean norm. Can I use a LASSO penalty in this ...
4
votes
0answers
59 views

Only minimizers of quadratic penalized least squares can be linear

Question Define the matrix $A \in \mathbb{R}^{m \times n}$, the vector $b \in \mathbb{R}^m$, and the function $\mathrm{pen}: \mathbb{R}^n \to \mathbb{R}$. Then, put the minimizer $$\hat{x} = \arg\...
1
vote
0answers
80 views

Calculating standard error from minimax concave penalty (MCP)

I am working on a high dimensional mediation analysis using the method in this paper. (Zhang H. 2016) The idea is to Step 1: Reduce the pool of potential mediators from a very large to a moderate ...
0
votes
0answers
144 views

Random effects in penalised regression in R

I have a dataset with a binary dependent variable, and a small number of independent variables (at minimum a categorical variable with 6 levels, a categorical variable with 3 levels and two binary ...