# Is Representer theorem valid with constraints on coefficients?

By the representer theorem, we have that in a Reproducing Kernel Hilbert Space the function being learnt in a regularizer + loss function problem under some conditions, can be represented as $\sum_i \alpha_i k(.,x_i)$. Then for the corresponding ML problem at hand an optimization is performed to learn the $\alpha$'s. If say there was a constraint put on $\alpha$'s during this optimization, does it make the representer theorem not hold, regardless of what the constraint be?

For SVM there already are constraints on $\alpha$, namely $0 < \alpha_i < C_i$ (see below for the origin of these constraints). This doesn't really affect the introduction of the representer theorem, provided that the constraints you add don't change the convex nature of the optimization problem.

I'll do a brief derivation for SVM, but the general idea translates to other kernel methods. The training problem for SVM:

\begin{align} \min_{\mathbf{w},\xi,b}\ &\frac{1}{2} ||\mathbf{w}||^2+C\sum_{i=1}^n \xi_i, \\ \mathtt{subject\ to}\ &y_i(\langle \mathbf{w},\varphi(\mathbf{x}_i)\rangle +b)\geq 1-\xi_i, \quad \xi_i \geq 0, \quad \forall i, \end{align} where $\mathbf{w}$ is the separating hyperplane in feature space (= the model), $\varphi(\cdot)$ the embedding function, $\mathbf{x}_i$ training instances and $y_i$ the associated labels.

The primal Langrange function is: $$L_p = \frac{1}{2}||\mathbf{w}||^2+C\sum_{i=1}^n\xi_i -\sum_{i=1}^n\alpha_i\Big[y_i\big(\langle\mathbf{w},\varphi(\mathbf{x}_i)\rangle+b\big)-(1-\xi_i)\Big]-\sum_{i=1}^n\mu_i\xi_i.$$

Finally, via the KKT conditions all partial derivatives must be zero, leading to:

\begin{align} \frac{\partial L_p}{\partial \xi_i}=0 \quad \rightarrow \quad &\alpha_i=C-\mu_i, \quad \forall i, \\ \frac{\partial L_p}{\partial \mathbf{w}}=0\quad \rightarrow \quad &\mathbf{w}=\sum_{i=1}^n \alpha_i y_i \varphi(\mathbf{x}_i), \end{align}

So at the end, the structure of $\mathbf{w}$ follows directly from the KKT conditions (i.e., from $\frac{\partial L_p}{\partial \mathbf{w}}=0$). If we add more constraints, the KKT conditions won't change, provided that the optimization problem remains convex.

Actually this answer seems to misunderstand what the constraints would be on. If there are generic constraints on the function in the overall RKHS, then the Representer Theorem as it exists today does not apply. Some recent efforts towards this direction have appeared at NIPS though:

https://www.ri.cmu.edu/pub_files/2015/0/Kernel-SOS.pdf