2 added 700 characters in body
source | link

SVM: $$\min \|w\|_2 + C\sum_{i = 1}^{n}(1 - y_i(wx_i + w_0))_+ $$ Perceptron $$\min \sum_{i = 1}^{n}(- y_i(wx_i + w_0))_+ $$

We can see that SVM has almost the same goal as L2-regularized perceptron.

Since the objective is different, we also have different optimization schemes for these two algorithms, from the $\|w\|_2$, we see that it is the key reason for using quadratic programming for optimizing SVM.

Why does perceptron allow online update? If you see the gradient descent update rule for the hinge loss (hinge loss is used by both SVM and perceptron),

$$w^t = w^{t-1} + \eta\frac{1}{N}\sum_{i = 1}^{N}y^ix^i\mathbb{I}(y^iw^tx^i \leq 0)$$

Since all machine learning algorithms can be seen as the combination of loss function and optimization algorithm.

Perceptron is no more than hinge loss (loss function) + stochastic gradient descent (optimization)

$$w^t = w^{t-1} + y^{y+1}x^{t+1}\mathbb{I}(y^{t+1}w^{t}x^{t+1} \leq 0)$$

And SVM can be seen as hinge loss + l2 regularization (loss + regularization) + quadratic programming or other fancier optimization algorithms like SMO (optimization).

SVM: $$\min \|w\|_2 + C\sum_{i = 1}^{n}(1 - y_i(wx_i + w_0))_+ $$ Perceptron $$\min \sum_{i = 1}^{n}(- y_i(wx_i + w_0))_+ $$

We can see that SVM has almost the same goal as L2-regularized perceptron.

Since the objective is different, we also have different optimization schemes for these two algorithms, from the $\|w\|_2$, we see that it is the key reason for using quadratic programming for optimizing SVM.

SVM: $$\min \|w\|_2 + C\sum_{i = 1}^{n}(1 - y_i(wx_i + w_0))_+ $$ Perceptron $$\min \sum_{i = 1}^{n}(- y_i(wx_i + w_0))_+ $$

We can see that SVM has almost the same goal as L2-regularized perceptron.

Since the objective is different, we also have different optimization schemes for these two algorithms, from the $\|w\|_2$, we see that it is the key reason for using quadratic programming for optimizing SVM.

Why does perceptron allow online update? If you see the gradient descent update rule for the hinge loss (hinge loss is used by both SVM and perceptron),

$$w^t = w^{t-1} + \eta\frac{1}{N}\sum_{i = 1}^{N}y^ix^i\mathbb{I}(y^iw^tx^i \leq 0)$$

Since all machine learning algorithms can be seen as the combination of loss function and optimization algorithm.

Perceptron is no more than hinge loss (loss function) + stochastic gradient descent (optimization)

$$w^t = w^{t-1} + y^{y+1}x^{t+1}\mathbb{I}(y^{t+1}w^{t}x^{t+1} \leq 0)$$

And SVM can be seen as hinge loss + l2 regularization (loss + regularization) + quadratic programming or other fancier optimization algorithms like SMO (optimization).

1
source | link

SVM: $$\min \|w\|_2 + C\sum_{i = 1}^{n}(1 - y_i(wx_i + w_0))_+ $$ Perceptron $$\min \sum_{i = 1}^{n}(- y_i(wx_i + w_0))_+ $$

We can see that SVM has almost the same goal as L2-regularized perceptron.

Since the objective is different, we also have different optimization schemes for these two algorithms, from the $\|w\|_2$, we see that it is the key reason for using quadratic programming for optimizing SVM.