0
$\begingroup$

I am working on my MSc. Statistic which is on the Penalized Logistic Regression with the LASSO penalty.

I am trying to understand the difference in two objective functions:

argmin {$\frac{1}{n}$ $\sum_{i=1}^n (Y_i-{\beta_0}-\sum_{j=1}^p {\beta_j}X_ij)^2+{\lambda}||{\beta}||_1$}

argmin { $\sum_{i=1}^n (Y_i-{\beta_0}-\sum_{j=1}^p {\beta_j}X_ij)^2+{\lambda}||{\beta}||_1$}

as you can see in the first formula we have $\frac{1}{n}$ while in the 2nd we don't when $n$ represent the sample size. Does anyone know why is this?

$\endgroup$

1 Answer 1

1
$\begingroup$

Both formulas are equivalent, since the $argmin$ will be the same regardless of whether you divide by n or not. You could also add an arbitrary constant the the expression would remain equivalent. For example, 3 is less than 5, so for every $a>0$ and any number $b$, $\frac{3}{a} + b < \frac{5}{a} + b$ In short, stay with the one you sounds more intuitive to you

For a similar reason, we sometimes use "sum of squares" instead of mean-squared error when comparing regression models, as the only difference is in scale

$\endgroup$
1
  • 1
    $\begingroup$ One thing to note is that the scale of $\lambda$ will change based on which expression is used, but as long as an appropriately scaled grid is used to choose $\lambda$, the coefficient values at the optimum will be the same. $\endgroup$
    – Noah
    Commented Jun 20, 2019 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.