For a given problem, one can determine the most relevant inputs for the LS-SVM within the Bayesian evidence framework (which works at three levels, where at level 3 one estimates the posterior probability of the kernel parameter). To do so, one assigns a different weighting parameter to each dimension in the kernel and optimizes this using the third level of inference. According to the used kernel, one can remove inputs corresponding the larger or smaller kernel parameters. This routine only works with the RBF kernel. In each step, the input with the largest optimal $\sigma^2$ is removed (backward selection). For every step, the generalization performance is approximated by the cost associated with the third level of Bayesian inference.

The ARD is based on backward selection of the inputs based on the $\sigma^2$ corresponding in each step with a minimal cost criterion. Minimizing this criterion can be done by 'continuous' or by 'discrete'. The former uses in each step continuous varying kernel parameter optimization, the latter decides which one to remove in each step by binary variables for each component (this can only applied for rather low dimensional inputs as the number of possible combinations grows exponentially with the number of inputs). If working with the RBF kernel, the kernel parameter $\sigma^2$ is rescaled appropriately after removing an input variable. The computation of the Bayesian cost criterion can be based on the SVD decomposition of the full kernel matrix.

My question is: how can you do input selection in a similar way as ARD using the crossvalidate function instead of the Bayesian framework?

Edit: ARD = Automatic Relevance Detection, LS-SVM = Least Squared Support Vector Machine, SVD = Singular Value Decomposition


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.