I am using a relevance vector machine as implemented in the kernlab-package in R, trained on a dataset with 360 continuous variables (features) and 60 examples (also continuous, so it's a relevance vector regression).
I have several datasets with equivalent dimensions from different subjects. Now it works fine for most of the subjects, but with one particular dataset, I get this strange results:
When using leave-one-out cross validation (so I train the RVM and try to subsequently predict one observation that was left out of the training), most of the predicted values are just around the mean of the example-values. So I really don't get good predictions, but just a slightly different value than the mean.
It seems like the SVM is not working at all; When I plot the fitted values against the actual values, I see the same pattern; predictions around the mean. So the RVM is not even able to predict the values it was trained on (for the other datasets I get correlations of around .9 between fitted and actual values).
It seems like, that I can at least improve the fitting (so that the RVM is at least able to predict the values it was trained on) by transforming the dependent variable (the example-values), for example by taking the square root of the dependent variable.
so this is the output for the untransformed dependent variable:
Relevance Vector Machine object of class "rvm" Problem type: regression
Linear (vanilla) kernel function.
Number of Relevance Vectors : 5
Variance : 1407.006
Training error : 1383.534902093
this, if I first transform the dependent variable by taking the square root:
Relevance Vector Machine object of class "rvm" Problem type: regression
Linear (vanilla) kernel function.
Number of Relevance Vectors : 55
Variance : 1.711355
Training error : 0.89601609
How is it, that the RVM-results change so dramatically, just by transforming the dependent variable? And what is going wrong, when an SVM just predicts values around the mean of the dependent variable (even for the values and observations it was trained on)?