-2
$\begingroup$

I know the normal equation is the closed form way of solving the linear regression problem. But how are they related to support vector machines?

I was asked this in an interview and did not know the answer.

$\endgroup$
  • $\begingroup$ What are "normal equations"? $\endgroup$ – Harald Thomson Jul 23 '17 at 10:37
  • 1
    $\begingroup$ Could it be possible that the interviewer intended "normal equation" to mean the parameterization of a hyper-plane by its normal vector? $\endgroup$ – shimao Jul 23 '17 at 17:04
  • $\begingroup$ @shimao You are right, he said normal equation not equations. Could you elaborate a little bit more please? $\endgroup$ – LoveMeow Jul 23 '17 at 20:41
  • 1
    $\begingroup$ If you have a point $p_0$ on a plane, and a vector $v$ which is orthogonal to the plane, then the plane can be expressed as the set of all points $p$ such that $v^T(p-p_0) = 0$. The goal of SVM is to find such a $v$ and a $p_0$ which maximizes the margin. $\endgroup$ – shimao Jul 23 '17 at 20:44
  • $\begingroup$ @shimao could you write that as an answer so I can accept it? So this equation is the normal equation for an svm? $\endgroup$ – LoveMeow Jul 23 '17 at 20:46
1
$\begingroup$

If you have a point $p_0$ on a plane, and a vector $v$ which is orthogonal to the plane, then the plane can be expressed as the set of all points $p$ such that $v^T(p-p_0) = 0$. The goal of SVM is to find such a $v$ and a $p_0$ which maximizes the margin.

In fact, if $v$ is constrained to be a unit vector, then $|v^T(p-p_0)|$ is exactly the distance of point $p$ from the plane. Therefore the objective of SVM can be stated as finding the vector $v$ and $p_0$ such that all positive datapoints $p$ have

$$v^T(p-p_0) > \epsilon$$

and all negative datapoints $p$ have

$$v^T(p-p_0) < -\epsilon$$

for the largest possible $\epsilon$.

So to answer your question in a comment, $v^T(p-p_0)$ is the normal equation or normal form for a plane, which is different from what is called the normal equation for solving linear regression. It is also not the case that this is the normal equation for SVM -- it just happens to appear in the mathematical formulation of the SVM objective.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.