In perceptron algorithm (the following analysis might apply to other classification algorithms), a smooth approximation of perceptron cost function $$\sum_i^n{\max(0, -y_i\mathbf{w}^T\mathbf{x}_i)}$$ is often used to find the separating hyperplane, where $y_i=\pm1$ (for simplicity, intercept term $b$ is included into $\mathbf{w}$). For example, one possible approximation to the above cost function might be LogSumExp, i.e., $$\sum_i^n{\log(1+e^{-y_i\mathbf{w}^T\mathbf{x}_i})}$$ is minimized instead.

Once we find the optimal solution $\mathbf{w}^*$, the accuracy (or the number of mislabeled instances) is usually calculated to see how good our classifier is. However, why don't we just start by minimizing the number of mislabeled instances (isn't this measure more direct?), for example, by minimizing the counting cost function as follows,

$$\sum_i^n{\max{(0, sign(-y_i\mathbf{w}^T \mathbf{x}_i))}}$$

I feel like this counting cost function can be approximated by LogSumExp (smooth max) plus a sigmoid-like function (smooth sign) as below,

$$\sum_i^n{\log{(1 + e^{\sigma(-y_i\mathbf{w}^T \mathbf{x}_i)})}}$$

where $\sigma(t)=\frac{1-e^{-t}}{1+e^{-t}}$ (note $\lim \limits_{c \to +\infty} \sigma(ct)$ is almost like sign function). Is there anything that prevents us from estimating this cost function instead (like numerical reason, perhaps)?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.