Clarify how to choose the kernel reference points (landmarks) to identify the non-linear boundary.

enter image description here


Going through SVM at Coursera ML - Support Vector Machine and trying to understand how to choose the landmarks to measure the distances to feed into the Gaussian Kernel.

It says "put the landmarks in the exact same locations as all the training examples".

enter image description here

enter image description here


Not clear why "the exact same locations of all the training data".

  1. Why using all the data?

    Number of features M and the number of data N is different and I suppose M << N. Then should we choose M number of data to use landmarks?
  2. Why not considering if a data to use as a landmark is classified positive or negative?

    I believe we would like to distinguish positive data (higher Gaussian probability), then why use the negative data as well as the landmarks?

    In the YouTube SVM with polynomial kernel visualization example (although it does not use Gaussian), the landmarks should be those that represent red points?

M is the number of data points, not the number of features. So we take all our (training) data, and for each (xi,yi), we get a landmark.

Notice that in the combined minimisation term, each fi is combined with its matching yi, so the minimisation takes account of which landmarks should be positive and which should be negative.

In the video each red dot AND each blue dot should be a landmark.

  • $\begingroup$ How come this approach doesn't lead to overfitting in situations where M is very large (is the regularization term in the minimization algorithm really enough to prevent overfitting ?!) ? $\endgroup$ – Tanguy Nov 20 '19 at 19:49
  • $\begingroup$ that's a good question -- you have to choose the regularisation parameter (C in the original question) carefully to avoid overfitting. When C is very large you are finding a tight fit to the landmarks (ie danger of overfitting). When C is small, you start to take account of the theta squared terms (ie danger of underfitting.) so C has to be chosen like any other hyperparameter. $\endgroup$ – Andrew Kay Nov 22 '19 at 8:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for?Browse other questions tagged or ask your own question.