I am playing around with Matlab's example which involves classifying whether data lie inside a circle of radius 1 (label: -1) or out of it (label: 1). I decided to experiment with things and flipped the label of one training point (call this $P$) outside the circle of radius 1 (from label: 1 to label: -1) to see how it would affect the boundary.
For different values of the regularization constant, I noticed that $P$ always ends up being a support vector. For smaller values of this constant, it's the only support vector far from the true boundary while for larger values, there are around 2 additional support vectors nearby.
My understanding is that especially in the latter case (large regularization constant), since there are support vectors from the 2 classes, I expect that there is a boundary between the data points in the region near $P$. However, after placing test points near $P$, the classifier predicted that they have label of 1 and none of them have a label of -1 even though $P$ is a support vector with label: -1.
Can anyone help me make sense of why these data points turned out to be support vectors?
Below are some plots of the training data with support vectors for various regularization constants.