I know that you can solve an overdetermined input matrix with QR decomposition or using the normal equation for the least squares linear regression problem. https://notendur.hi.is/tpr/tutorials/svm/notes/chapter5_6.pdf
How would you do the same for an SVM?
I googled it and coudn't find an explanation I understand!
Thanks!
Ordinary least squares (OLS) regression, support vector machines (SVM), and many other machine learning methods are formulated as optimization problems. The loss function for each problem assigns a value to every possible choice of parameters. Fitting the model entails finding the parameters that minimize the loss.
The OLS loss function is convex, which implies that any parameter vector where the gradient of the loss function is zero is a solution. For this particular problem, you can take the gradient of the loss function, set it equal to zero, and solve for the parameters algebraically. This can be done in different ways, including writing the solution in terms of various matrix factorizations (e.g. the QR decomposition, as you mentioned). When the design/regressor matrix has full rank, the solution is unique.
The SVM optimization problem is slightly more complicated because it has constraints. There are multiple ways to formulate the problem. Like OLS, the problem is convex. But, unlike OLS, it's not possible to solve algebraically. Instead, iterative optimization algorithms are needed. This means it's not possible to fit an SVM using matrix decomposition, as you would for OLS.
