# Strange kernlab's relevance vector machine predictions

I am using a relevance vector machine as implemented in the kernlab-package in R, trained on a dataset with 360 continuous variables (features) and 60 examples (also continuous, so it's a relevance vector regression).

I have several datasets with equivalent dimensions from different subjects. Now it works fine for most of the subjects, but with one particular dataset, I get this strange results:

When using leave-one-out cross validation (so I train the RVM and try to subsequently predict one observation that was left out of the training), most of the predicted values are just around the mean of the example-values. So I really don't get good predictions, but just a slightly different value than the mean.

It seems like the SVM is not working at all; When I plot the fitted values against the actual values, I see the same pattern; predictions around the mean. So the RVM is not even able to predict the values it was trained on (for the other datasets I get correlations of around .9 between fitted and actual values).

It seems like, that I can at least improve the fitting (so that the RVM is at least able to predict the values it was trained on) by transforming the dependent variable (the example-values), for example by taking the square root of the dependent variable.

so this is the output for the untransformed dependent variable:

Relevance Vector Machine object of class "rvm" Problem type: regression

Linear (vanilla) kernel function.

Number of Relevance Vectors : 5
Variance :  1407.006
Training error : 1383.534902093


this, if I first transform the dependent variable by taking the square root:

Relevance Vector Machine object of class "rvm" Problem type: regression

Linear (vanilla) kernel function.

Number of Relevance Vectors : 55
Variance :  1.711355
Training error : 0.89601609


How is it, that the RVM-results change so dramatically, just by transforming the dependent variable? And what is going wrong, when an SVM just predicts values around the mean of the dependent variable (even for the values and observations it was trained on)?

-
 What kernel are you using? Linear or non-linear? I would encourage you to use a gaussian kernel and try tune the kernel parameter using a validation set derived from your training set (N-1). I suspect that your model is underperfoming as you have not tuned this parameter. All the best! – user9492 Feb 28 '12 at 15:07 I use a linear kernel. I assumed that you don't need to tune any hyperparameters in relevance vector machine, since these are 'tuned' iteratively already. – grueb Feb 28 '12 at 15:46