Return to Answer

Minimizing the total cost $L$ can be split up into two distinct parts, namely minimizing the squared norm of the separating hyperplane $\|\mathbf{w}\|^2$ and the misclassification penalties $\lambda \sum_{i=1}^T \xi_i$. Note that minimizing $\|\mathbf{w}\|^2$ is equivalent to maximizing the margin, which is $1/\|\mathbf{w}\|$.

If $\lambda$ is large, minimizing the latter term becomes increasingly important to minimize the overall cost $L$ and vice versa. If this is still unclear, imagine the extreme cases $\lambda = 0$ and $\lim_{\lambda\rightarrow\infty}$. In the former, we only care about minimizing $\|\mathbf{w}\|^2$ whereas in the latter we only care about minimizing $\sum_{i=1}^R \xi_i$.

Training an SVM really involves finding the coefficients $\alpha$ and $b$ in the model. The model is a separating hyperplane in feature space, e.g. $$f(\mathbf{z}) = \sum_{i=1}^{n_{SV}} y_i \alpha_i \mathbf{x}_i^T \mathbf{z} + b,$$ where $\mathbf{y}$ is the vector of labels and $\mathbf{x}_i$ are support vectors. For notational simplicity I assume we work with the linear kernel.

A certain solution, e.g. vector of $\alpha$ values and $b$ induces a certain margin (inversely related to $\|\mathbf{w}\|^2$) and a certain number of misclassifications/points within the margin (quantified by $\sum_{i=1}^N \xi_i$). The goal now is to find a sweet spot between having a large margin on one hand (e.g. minimize $\|\mathbf{w}\|^2$, simple model) and having little training misclassifications (e.g. minimize $\sum_{i=1}^N \xi_i$, complex model). We need to make a tradeoff. That is what $\lambda$ is for.

Consider the following contrived example (this is a simplified version of how it really works): suppose that we have two possible solutions, with the following properties:

1. $(\alpha^{(1)}, b^{(1)})$ which yields $\|\mathbf{w}^{(1)}\|^2$ = 1 and $\sum_{i=1}^N \xi_i^{(1)}$ = 2.
2. $(\alpha^{(2)}, b^{(2)})$ which yields $\|\mathbf{w}^{(2)}\|^2$ = 2 and $\sum_{i=1}^N \xi_i^{(2)}$ = 1.

For $\lambda < 1$, solution 1 is preferable since it yields lower cost ($L^{(1)} < 3$ vs $L^{(2)} > 3$). On the other hand, if $\lambda > 1$, solution 2 is preferable ($L^{(1)} = 1 + 2\lambda$ vs $L^{(2)} = 2 + \lambda$).

Minimizing the total cost $L$ can be split up into two distinct parts, namely minimizing the squared norm of the separating hyperplane $\|\mathbf{w}\|^2$ and the misclassification penalties $\lambda \sum_{i=1}^T \xi_i$. Note that minimizing $\|\mathbf{w}\|^2$ is equivalent to maximizing the margin, which is $1/\|\mathbf{w}\|$.

If $\lambda$ is large, minimizing the latter term becomes increasingly important to minimize the overall cost $L$ and vice versa. If this is still unclear, imagine the extreme cases $\lambda = 0$ and $\lim_{\lambda\rightarrow\infty}$. In the former, we only care about minimizing $\|\mathbf{w}\|^2$ whereas in the latter we only care about minimizing $\sum_{i=1}^R \xi_i$.

Training an SVM really involves finding the coefficients $\alpha$ and $b$ in the model. The model is a separating hyperplane in feature space, e.g., $$f(\mathbf{z}) = \sum_{i=1}^{n_{SV}} y_i \alpha_i \mathbf{x}_i^T \mathbf{z} + b,$$ where $\mathbf{y}$ is the vector of labels and $\mathbf{x}_i$ are support vectors. For notational simplicity I assume we work with the linear kernel.

A certain solution, e.g., a vector of $\alpha$ values and $b$, induces a certain margin (inversely related to $\|\mathbf{w}\|^2$) and a certain number of misclassifications  / points within the margin (quantified by $\sum_{i=1}^N \xi_i$). The goal now is to find a sweet spot between having a larger margin on one hand (e.g. minimize $\|\mathbf{w}\|^2$, simple model) and having fewer training misclassifications (e.g., minimize $\sum_{i=1}^N \xi_i$, complex model). We need to make a tradeoff. That is what $\lambda$ is for.

Consider the following contrived example (this is a simplified version of how it really works): suppose that we have two possible solutions, with the following properties:

1. $(\alpha^{(1)}, b^{(1)})$ which yields $\|\mathbf{w}^{(1)}\|^2$ = 1 and $\sum_{i=1}^N \xi_i^{(1)}$ = 2.
2. $(\alpha^{(2)}, b^{(2)})$ which yields $\|\mathbf{w}^{(2)}\|^2$ = 2 and $\sum_{i=1}^N \xi_i^{(2)}$ = 1.

For $\lambda < 1$, solution 1 is preferable since it yields lower cost ($L^{(1)} < 3$ vs $L^{(2)} > 3$). On the other hand, if $\lambda > 1$, solution 2 is preferable ($L^{(1)} = 1 + 2\lambda$ vs $L^{(2)} = 2 + \lambda$).

Minimizing the total cost $L$ can be split up into two distinct parts, namely minimizing the squared norm of the separating hyperplane $\|\mathbf{w}\|^2$ and the misclassification penalties $\lambda \sum_{i=1}^T \xi_i$. Note that minimizing $\|\mathbf{w}\|^2$ is equivalent to maximizing the margin, which is $1/\|\mathbf{w}\|$.

If $\lambda$ is large, minimizing the latter term becomes increasingly important to minimize the overall cost $L$ and vice versa. If this is still unclear, imagine the extreme cases $\lambda = 0$ and $\lim_{\lambda\rightarrow\infty}$. In the former, we only care about minimizing $\|\mathbf{w}\|^2$ whereas in the latter we only care about minimizing $\sum_{i=1}^R \xi_i$.

Training an SVM really involves finding the coefficients $\alpha$ and $b$ in the model. The model is a separating hyperplane in feature space, e.g. $$f(\mathbf{z}) = \sum_{i=1}^{n_{SV}} y_i \alpha_i \mathbf{x}_i^T \mathbf{z} + b,$$ where $\mathbf{y}$ is the vector of labels and $\mathbf{x}_i$ are support vectors. For notational simplicity I assume we work with the linear kernel.

A certain solution, e.g. vector of $\alpha$ values and $b$ induces a certain margin (inversely related to $\|\mathbf{w}\|^2$) and a certain number of misclassifications/points within the margin (quantified by $\sum_{i=1}^N \xi_i$). The goal now is to find a sweet spot between having a large margin on one hand (e.g. minimize $\|\mathbf{w}\|^2$) and having little training misclassifications (e.g. minimize $\sum_{i=1}^N \xi_i$). We need to make a tradeoff. That is what $\lambda$ is for.

Consider the following contrived example (this is a simplified version of how it really works): suppose that we have two possible solutions, with the following properties:

1. $(\alpha^{(1)}, b^{(1)})$ which yields $\|\mathbf{w}^{(1)}\|^2$ = 1 and $\sum_{i=1}^N \xi_i^{(1)}$ = 2.
2. $(\alpha^{(2)}, b^{(2)})$ which yields $\|\mathbf{w}^{(2)}\|^2$ = 2 and $\sum_{i=1}^N \xi_i^{(2)}$ = 1.

For $\lambda < 1$, solution 1 is preferable since it yields lower cost ($L^{(1)} < 3$ vs $L^{(2)} > 3$). On the other hand, if $\lambda > 1$, solution 2 is preferable ($L^{(1)} = 1 + 2\lambda$ vs $L^{(2)} = 2 + \lambda$).

Minimizing the total cost $L$ can be split up into two distinct parts, namely minimizing the squared norm of the separating hyperplane $\|\mathbf{w}\|^2$ and the misclassification penalties $\lambda \sum_{i=1}^T \xi_i$. Note that minimizing $\|\mathbf{w}\|^2$ is equivalent to maximizing the margin, which is $1/\|\mathbf{w}\|$.

If $\lambda$ is large, minimizing the latter term becomes increasingly important to minimize the overall cost $L$ and vice versa. If this is still unclear, imagine the extreme cases $\lambda = 0$ and $\lim_{\lambda\rightarrow\infty}$. In the former, we only care about minimizing $\|\mathbf{w}\|^2$ whereas in the latter we only care about minimizing $\sum_{i=1}^R \xi_i$.

Training an SVM really involves finding the coefficients $\alpha$ and $b$ in the model. The model is a separating hyperplane in feature space, e.g. $$f(\mathbf{z}) = \sum_{i=1}^{n_{SV}} y_i \alpha_i \mathbf{x}_i^T \mathbf{z} + b,$$ where $\mathbf{y}$ is the vector of labels and $\mathbf{x}_i$ are support vectors. For notational simplicity I assume we work with the linear kernel.

A certain solution, e.g. vector of $\alpha$ values and $b$ induces a certain margin (inversely related to $\|\mathbf{w}\|^2$) and a certain number of misclassifications/points within the margin (quantified by $\sum_{i=1}^N \xi_i$). The goal now is to find a sweet spot between having a large margin on one hand (e.g. minimize $\|\mathbf{w}\|^2$, simple model) and having little training misclassifications (e.g. minimize $\sum_{i=1}^N \xi_i$, complex model). We need to make a tradeoff. That is what $\lambda$ is for.

Consider the following contrived example (this is a simplified version of how it really works): suppose that we have two possible solutions, with the following properties:

1. $(\alpha^{(1)}, b^{(1)})$ which yields $\|\mathbf{w}^{(1)}\|^2$ = 1 and $\sum_{i=1}^N \xi_i^{(1)}$ = 2.
2. $(\alpha^{(2)}, b^{(2)})$ which yields $\|\mathbf{w}^{(2)}\|^2$ = 2 and $\sum_{i=1}^N \xi_i^{(2)}$ = 1.

For $\lambda < 1$, solution 1 is preferable since it yields lower cost ($L^{(1)} < 3$ vs $L^{(2)} > 3$). On the other hand, if $\lambda > 1$, solution 2 is preferable ($L^{(1)} = 1 + 2\lambda$ vs $L^{(2)} = 2 + \lambda$).

Minimizing the total cost $L$ can be split up into two distinct parts, namely minimizing the squared norm of the separating hyperplane $\|\mathbf{w}\|^2$ and the misclassification penalties $\lambda \sum_{i=1}^T \xi_i$. Note that minimizing $\|\mathbf{w}\|^2$ is equivalent to maximizing the margin, which is $1/\|\mathbf{w}\|$.

If $\lambda$ is large, minimizing the latter term becomes increasingly important to minimize the overall cost $L$ and vice versa. If this is still unclear, imagine the extreme cases $\lambda = 0$ and $\lim_{\lambda\rightarrow\infty}$. In the former, we only care about minimizing $\|\mathbf{w}\|^2$ whereas in the latter we only care about minimizing $\sum_{i=1}^R \xi_i$.

Training an SVM really involves finding the coefficients $\alpha$ and $b$ in the model. The model is a separating hyperplane in feature space, e.g. $$f(\mathbf{z}) = \sum_{i=1}^{n_{SV}} y_i \alpha_i \mathbf{x}_i^T \mathbf{z} + b,$$ where $\mathbf{y}$ is the vector of labels and $\mathbf{x}_i$ are support vectors. For notational simplicity I assume we work with the linear kernel.

A certain solution, e.g. vector of $\alpha$ values and $b$ induces a certain margin (inversely related to $\|\mathbf{w}\|^2$) and a certain number of misclassifications/points within the margin (quantified by $\sum_{i=1}^N \xi_i$). The goal now is to find a sweet spot between having a large margin on one hand (e.g. minimize $\|\mathbf{w}\|^2$) and having little training misclassifications (e.g. minimize $\sum_{i=1}^N \xi_i$). We need to make a tradeoff. That is what $\lambda$ is for.

Consider the following contrived example (this is a simplified version of how it really works): suppose that we have two possible solutions, with the following properties:

1. $(\alpha^{(1)}, b^{(1)})$ which yields $\|\mathbf{w}\|^2$ = 1 and $\sum_{i=1}^N \xi_i$ = 2.
2. $(\alpha^{(2)}, b^{(2)})$ which yields $\|\mathbf{w}\|^2$ = 2 and $\sum_{i=1}^N \xi_i$ = 1.

For $\lambda < 0.5$, solution 1 is preferable since it yields lower cost ($L < 2$). On the other hand, if $\lambda > 0.5$, solution 2 is preferable.

Minimizing the total cost $L$ can be split up into two distinct parts, namely minimizing the squared norm of the separating hyperplane $\|\mathbf{w}\|^2$ and the misclassification penalties $\lambda \sum_{i=1}^T \xi_i$. Note that minimizing $\|\mathbf{w}\|^2$ is equivalent to maximizing the margin, which is $1/\|\mathbf{w}\|$.

If $\lambda$ is large, minimizing the latter term becomes increasingly important to minimize the overall cost $L$ and vice versa. If this is still unclear, imagine the extreme cases $\lambda = 0$ and $\lim_{\lambda\rightarrow\infty}$. In the former, we only care about minimizing $\|\mathbf{w}\|^2$ whereas in the latter we only care about minimizing $\sum_{i=1}^R \xi_i$.

Training an SVM really involves finding the coefficients $\alpha$ and $b$ in the model. The model is a separating hyperplane in feature space, e.g. $$f(\mathbf{z}) = \sum_{i=1}^{n_{SV}} y_i \alpha_i \mathbf{x}_i^T \mathbf{z} + b,$$ where $\mathbf{y}$ is the vector of labels and $\mathbf{x}_i$ are support vectors. For notational simplicity I assume we work with the linear kernel.

A certain solution, e.g. vector of $\alpha$ values and $b$ induces a certain margin (inversely related to $\|\mathbf{w}\|^2$) and a certain number of misclassifications/points within the margin (quantified by $\sum_{i=1}^N \xi_i$). The goal now is to find a sweet spot between having a large margin on one hand (e.g. minimize $\|\mathbf{w}\|^2$) and having little training misclassifications (e.g. minimize $\sum_{i=1}^N \xi_i$). We need to make a tradeoff. That is what $\lambda$ is for.

Consider the following contrived example (this is a simplified version of how it really works): suppose that we have two possible solutions, with the following properties:

1. $(\alpha^{(1)}, b^{(1)})$ which yields $\|\mathbf{w}^{(1)}\|^2$ = 1 and $\sum_{i=1}^N \xi_i^{(1)}$ = 2.
2. $(\alpha^{(2)}, b^{(2)})$ which yields $\|\mathbf{w}^{(2)}\|^2$ = 2 and $\sum_{i=1}^N \xi_i^{(2)}$ = 1.

For $\lambda < 1$, solution 1 is preferable since it yields lower cost ($L^{(1)} < 3$ vs $L^{(2)} > 3$). On the other hand, if $\lambda > 1$, solution 2 is preferable ($L^{(1)} = 1 + 2\lambda$ vs $L^{(2)} = 2 + \lambda$).