What is the difference between Informative (IVM) and Relevance (RVM) vector machines

Question

I'm trying to understand if there is any specific difference between Informative IVMs and Relevance RVMs other than the terminology. I've not seen anything explicit.

When I'm reading about vector machines, it is easy to see the difference of IVM/RVMs from Support (SVM) vector machines [colloquially, for classification, the SVM finds those points (vectors) that define the DMZ (de-militatized zone ;-) between the categories, while the RVM finds those that are the 'middle' of the crowd, and an associated crowd 'size' (e.g. in gaussian globs)], but I don't see any special difference between I/R vector machines beyond a choice of terminology by their proponents.

Is there a difference?

Answer 1

The RVM places an Automatic Relevance Determination (ARD) prior on the weights in a regularized regression/logistic regression setup. (The ARD prior is a just a weak gamma prior on the precision of a gaussian random variable). Marginalizing out the weights and maximizing the likelihood of the data with respect to the precision causes many of the precision parameters to become large, which would push the associated weights to zero. If you use feature vectors given by a design matrix, then this strategy selects a small set of examples that predict the target variable well.

The IVM strategy is fundamentally different from the RVM's strategy. The IVM is a Gaussian Process method that selects a small set of points from the training set using a greedy selection criterion (based on change in entropy of the posterior GP) and combines this strategy with standard GP regression/classification on the sparse set of points.

Unlike the SVM, for both the IVM and RVM there is not an obvious geometric interpretation of relevant or informative vectors. Basically, both of the algorithms find sparse (the SVM and IVM are dual sparse, but the RVM should probably be considered primal sparse) solutions for regression/classification problems but they use different approaches to do so.

Answer 2

Some of the distinctions I have identified so far were hidden in (one of) the original papers by Neil Lawrence. There are two versions of "A Sparse Bayesian Compression Scheme - The Informative Vector Machine" [Kernel Workshop at NIPS 2001], one on the Microsoft research site, and one on Laurence's site.

In the MS version there is an extra sentence with the statement "the selected data points are close to the decision boundary, a characteristics shared with the SVM". So my original view that the IVM vectors represented the 'middle' was wrong.

The other point is that it is a compression scheme in the sense that it is looking for a 'sparse representation of the data set", so that the IVM also seeks to retain those vectors that provided the most information during the analysis, so they can be re-used, as a data set and any computation repeated.

This ability to reduce the size of the data set is useful when the computation is bigO(M^3)

The RVM does select the 'middle' vectors (and their weights) which act on the basis functions (e.g. Gaussians) (see Ch7.2 'Relevance Vector Machines' in Bishop's 'Pattern recognition and machine learning').

OK, so the explanation is a bit of a hand waving description and isn't fully complete, but hopefully it will help those who aren't as relaxed with compact matrix formulations. More feedback would still be welcome.