How does RVM achieve sparsity? I have read several textbook descriptions on RVM and none of them provide an adequate (plain English) explanation of how RVM achieves sparsity.
I am left feeling like the authors left out a paragraph of text that would have connected the dots and instead decided to replace (rather than supplement) it with mathematical derivations.
Could someone please explain the basic idea as to how RVM works in relation to learning sparse regression models?
 A: In Relevance vector machines (RVM) we have a prior on the weight vector $\mathbf{w}$ (which is $N+1$ dimensional, where $N$ is the number of examples) as shown in equation (5) of (1):
$$p(\mathbf{w}|\alpha) = \Pi_{i=0}^{N}\mathcal{N}(w_i|0,\alpha_i^{-1}),$$
where $\mathbf{\alpha}$ is the $N+1$ dimensional vector of hyperparameters.
This prior is supposed to ensure that the weight vector $\mathbf{w}$ (which represents the number of "support vectors" which are active) is "sparse" if we can integrate out all the nuisance parameters ($\alpha$). See paragraph preceding Section 2.2 in (1).
Potential points of confusion:


*

*the notation $\mathbf{w}$ is different from the $d$-dimensional linear model representation. Here, while comparing RVM with SVM, only think of the dual SVM formulation with the $N+1$ dimensional parameter $\mathbf{w}$.

*"Sparse" for (dual) SVMs means the number of support vectors is small. Do not confuse with number of non-zero coefficients in (the d-dimensional) linear models.

