I apologize for the verbose description, but (after searching several places for an answer) maybe the best way to phrase it is to lay things out explicitly.
Say we are trying to build an SVM model for classification, and our data consists of $m$=1000 training examples with $n$=15 features.
We start with the 'Primal' version of the optimization problem, which is defined in terms of the values of the feature-vectors $x_i$.
The Lagrangian dual of the optimization problem is (from Wikipedia):
${\text{maximize}}:\ \ \ \sum _{i=1}^{m}c_{i}-{\frac {1}{2}}\sum _{i=1}^{m}\sum _{j=1}^{m}y_{i}y_{j}c_{i}c_{j}(x_{i}\cdot x_{j})\ ,$ ${\text{subject to }}:\ \ \sum _{i=1}^{m}c_{i}y_{i}=0,\,{\text{and }}0\leq c_{i}\leq {\frac {1}{2m\lambda }}\;{\text{for all }}i.$
Notably, this is defined only in terms of the pairwise dot-products of the $x_i$ vectors. This helps us use the kernel-trick by replacing $(x_i \cdot x_j$) in the optimization objective with $K(x_i,x_j)$, where $K$ is our choice of kernel function. Let's say we choose the RBF: $K(u,v)=\exp \left(-{\frac {\|u -v \|^2}{2\sigma ^2}}\right)$, and this brings us to:
APPROACH (A)
${\text{maximize}}:\ \ \ \sum _{i=1}^{m}c_{i}-{\frac {1}{2}}\sum _{i=1}^{m}\sum _{j=1}^{m}y_{i}y_{j}c_{i}c_{j}\ K(x_{i}\cdot x_{j})$
This approach works as if we had implicitly mapped all vectors $x_i \in \mathbb{R}^{15}$ to some higher-dimensional feature-set $\phi_i$(in this case, $\phi_i \in \mathbb{R}^\infty$ due to using the RBF), and found an optimal hyperplane to separate the points.
Now consider APPROACH (B), where we use the RBF in a different way:
We map the features $x_i$ to $\mathbb{R}^{m}$, by evaluating for each point its 'nearness' to every $x_i$ using the RBF.
i.e. Each $x_i \in \mathbb{R}^{15}$ is mapped to a $\phi_i \in \mathbb{R}^{1000}$,
where the value of the $d^{th}$ dimension of $\phi_i$ is $\text{RBF}(x_i,x_d)$.
This is not very different from using polynomial-features followed by linear regression.
In both approaches we are using the RBF as a measure of 'similarity'.
My questions are:
Is Approach (B) used in practice? Is there anything wrong with using the RBF to compute new features in this way, or does it make it inferior to approach (A)?
Intuitively, how is the handling of the pairwise 'similarity' of training points different in the two approaches? Why do 'similarity-functions' work well as choices for kernel functions in the first place?
