2
$\begingroup$

Ridge Regression objective$$\underset{\beta}{\text{min}} \sum_{i=1}^n (y_i - \beta \cdot x_i)^2 + \lambda \|\beta\|_2^2$$

SVM primal problem:

$$\begin{align} \max_{\mathbf{\alpha}} \quad &\min_{\mathbf{w},b} \frac{\|\mathbf{w}\|}{2}+ C \sum_{i=1}^{N} \alpha^{(i)} \left(1-\mathbf{w^T}\phi \left(\mathbf{x}^{(i)}\right)+b)\right), \\ s.t. \quad&0 \leq \alpha^{(i)} \leq C, &\forall i \in \{1,\dots,N\} \end{align}$$

Why does SVM have a hyperparameter parameter C on the hinge-loss function while in Ridge Regression There's No C parameter infront of the quadratic loss function?

Likewise, why is there no $\lambda$ parameter behind $||w||$ in SVM so $\lambda$ is assumed 1/2 in SVM?

Why isn't ridge: $$\underset{\beta}{\text{min}} \ C\sum_{i=1}^n (y_i - \beta \cdot x_i)^2 + \lambda \|\beta\|_2^2$$

Why isn't the SVM:

$$\begin{align} \max_{\mathbf{\alpha}} \quad &\min_{\mathbf{w},b} \lambda\|\mathbf{w}\|+ C \sum_{i=1}^{N} \alpha^{(i)} \left(1-\mathbf{w^T}\phi \left(\mathbf{x}^{(i)}\right)+b)\right), \\ s.t. \quad&0 \leq \alpha^{(i)} \leq C, &\forall i \in \{1,\dots,N\} \end{align}$$?

$\endgroup$

1 Answer 1

5
$\begingroup$

To introduce both hyperparameters in one equation would be redundant.

If you have both $C$ and $\lambda$, then there are several equivalent optimization problems with the same ratio of $C$ to $\lambda$. For instance, in your proposed ridge regression equation, the solution for $C=1, \lambda=1$ is necessarily the same as for $C=1000, \lambda=1000$. (This is because you can just factor out the common factor.)

\begin{align} &\min_{x,y} 1x + 1 y \\ =&\min_{x,y} 1000x + 1000 y \end{align}

In other words, it looks like you have two hyperparameters ($C$ and $\lambda$), but you truly have only one: the ratio between $C$ and $\lambda$.

By clamping one of the two values, you avoid a bit of wasteful overparameterization. If $C$ is always $1$, then adjusting $\lambda$ is the only way to adjust the ratio. Same for keeping $\lambda$ always at $1$.

$\endgroup$
4
  • $\begingroup$ How is Elastic Net/Generalizations of the Lasso with two+ hyperparameters in one equation non-redundant? $\endgroup$
    – user318514
    Commented May 14, 2021 at 4:13
  • 1
    $\begingroup$ Lasso has two summands and one hyperparameter. ElasticNet has three summands and two hyperparameters. In general, you need $n-1$ hyperparameters for $n$ summands. The remaining one is clamped to 1. $\endgroup$ Commented May 14, 2021 at 4:20
  • $\begingroup$ There's no redundancy: it's perfectly possible to introduce multiple parameters into Ridge Regression. See my remarks in the last paragraph at stats.stackexchange.com/a/164546/919. $\endgroup$
    – whuber
    Commented May 14, 2021 at 12:08
  • 2
    $\begingroup$ @whuber I think we’re talking past each other. There are perfectly valid opportunities to introduce multiple parameters. In this particular introduction, you gain no expressive power in your model. $\endgroup$ Commented May 14, 2021 at 12:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.