11
$\begingroup$

Objective

Clarify how to choose the kernel reference points (landmarks) to identify the non-linear boundary.

enter image description here

Background

Going through SVM at Coursera ML - Support Vector Machine and trying to understand how to choose the landmarks to measure the distances to feed into the Gaussian Kernel.

It says "put the landmarks in the exact same locations as all the training examples".

enter image description here

enter image description here

Question

Not clear why "the exact same locations of all the training data".

  1. Why using all the data?

    Number of features M and the number of data N is different and I suppose M << N. Then should we choose M number of data to use landmarks?
  2. Why not considering if a data to use as a landmark is classified positive or negative?

    I believe we would like to distinguish positive data (higher Gaussian probability), then why use the negative data as well as the landmarks?

    In the YouTube SVM with polynomial kernel visualization example (although it does not use Gaussian), the landmarks should be those that represent red points?
Improve this question
$\endgroup$

2 Answers 2

Reset to default
2
$\begingroup$

M is the number of data points, not the number of features. So we take all our (training) data, and for each (xi,yi), we get a landmark.

Notice that in the combined minimisation term, each fi is combined with its matching yi, so the minimisation takes account of which landmarks should be positive and which should be negative.

In the video each red dot AND each blue dot should be a landmark.

Improve this answer
$\endgroup$
2
  • $\begingroup$ How come this approach doesn't lead to overfitting in situations where M is very large (is the regularization term in the minimization algorithm really enough to prevent overfitting ?!) ? $\endgroup$
    – Tanguy
    Nov 20, 2019 at 19:49
  • $\begingroup$ that's a good question -- you have to choose the regularisation parameter (C in the original question) carefully to avoid overfitting. When C is very large you are finding a tight fit to the landmarks (ie danger of overfitting). When C is small, you start to take account of the theta squared terms (ie danger of underfitting.) so C has to be chosen like any other hyperparameter. $\endgroup$
    – Andrew Kay
    Nov 22, 2019 at 8:57
1
$\begingroup$

Obviously very late, but as Andrew's answer mentions, each $f^{(i)}$ is paired with the same label as $x^{(i)}$. That is, if $x^{(i)}$ is paired with a corresponding label of $1$, then $f^{(i)}$ will also be paired with $1$. For this reason, if we only utilize the features $x^{(i)}$ with corresponding labels of $1$ (i.e. positive features), we'd end up with a transformed dataset in which each of the features $f^{(i)}$ have corresponding label of $1$. Training an SVM on this transformed dataset would therefore yield an SVM that could constantly predict $1$, and obtain a cost of $0$ each time, which obviously isn't very helpful at all.

This is all to say we need to pick both positive and negative feature as landmarks, otherwise we could end up with a transformed dataset so skewed that it becomes impossible to train an SVM with a reasonable test set performance on said transformed dataset.

Hope this helps.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.