# Non-linear kernel for classifying data points corresponding to two concentric circles [closed]

Have seen article, while doing self-study, on Non-linearly seperable problems, here. The images as given there are here, and here.

It deals a common text-book problem, where the data points are in two disjoint groups of concentric circles.

The method given is to have the kernel function as : $$φ(x) = φ((x_1, x_2)) = (x_1^2, √2 x_1x_2, x_2^2).$$

But, if take an example with the inner circle having radius diameter of $$2$$ units, while the outer having a radius diameter of $$4$$ units; then let us have two points on the inner circle given by : $$(2,2), (2,4)$$; denoted by $$x_{i1}$$, and $$x_{i2}$$ respectively.
Similarly, let there be two data points on the outer circle as: $$x_{o1}= (2,1),$$ and $$x_{o2}=(2,5).$$ The application of the kernel function, does not seem to separate the points on the two concentric circles with a linear SVC, as shown by the new coordinates, and Norms, as given below:

1. $$φ(x_{o1})=(2,1) = (4, 2\sqrt{2}, 1),$$ with norm $$||φ(x_{o1})||=16+1+8 = 25,$$
2. $$φ(x_{i1})=(2,2) = (4, 4\sqrt{2}, 4),$$ with norm $$||φ(x_{i1})||=64,$$
3. $$φ(x_{i2})=(2,4) = (4, 8\sqrt{2}, 16),$$ with norm $$||φ(x_{i2})||=16+256+128 = 400,$$
4. $$φ(x_{o2})=(2,5) = (4, 10\sqrt{2}, 25),$$ with norm $$||φ(x_{o2})||=481.$$

It is unclear how the given kernel function would transform/map these points into two linearly separable classes.

• Welcome to Cross Validated! I'm not sure why this got a close vote, let alone one for being off-topic. This seems completely within the scope of thie Stack.
– Dave
Commented May 23 at 15:27
• In what sense are $(2,2)$ and $(2,4)$ on the inner circle?? The radii are $\sqrt{2^2+2^2}=\sqrt{8}$ and $\sqrt{2^2+4^2}=\sqrt{20},$ neither of which are even close to $2.$ Same question for the outer circle. Moreover, in none of your links do I see any example with circles of radii $2$ and $4:$ I see only $1$ and $1/2.$ In this sense your example is incomplete: what are the centers of your circles?
– whuber
Commented May 23 at 17:09
• @whuber Please shift the centre to $(2,3),$ rather than the origin. Also, I have mistaken in stating 'radius', instead of 'diameter'. Commented May 24 at 3:13

For circles centred at $$(0, 0)$$, I think one or more of the points you listed wouldn't lie on the circles.

Consider a simple case where the radii of the circles are 5 and 10 respectively. The figure below highlights 2 points on each circle. The points are not linearly separable.

I've annotated each point with its 3D coordinates in $$\phi$$ space. Visualising this in 3D:

For this particular polynomial kernel and dataset, the separability comes from the combination of $$\phi_0=x_0^2$$ and $$\phi_2=x_1^2$$, which together represent $$\mathrm{radius^2}$$. At any angle, the radius of a blue point is larger than the radius of the red point. As a result, the blue points get pushed out further than the red points, leading to the following projection in the $$(\phi_0, \phi_2)$$:

The $$\phi_1$$ axis above moves the circles closer to, or further from, the screen, but for this centred dataset doesn't do anything as far as separating the classes.

Whilst $$\phi_1$$ has product terms, and therefore is sensitive to the size of each circle, it also carries information about the sign of each point. Both circles have $$xy$$ products with the same sign, so their $$\phi_1$$ is centred at the same point:

These are projections onto a 1D space, separated by label for clarity:

The first thing to note is that $$\phi_0$$ and $$\phi_2$$ care only about the magnitude, not the sign, as they are non-negative (rows 1 to 3). Taken together, $$\phi_0 + \phi_2$$ (row 4) define $$\mathrm{radius}^2$$, which enables separation of the two circles. In contrast, the last 2 rows track the sign of $$x_0$$ and $$x_1$$ because of the inclusion of $$\phi_1$$. The sign of $$x_0 x_1$$ is the same for both circles, causing points to overlap and become non-separable.

For this dataset, the only distinguishing feature in feature space is the radius of each circle, which is why we need a radius axis. Otherwise, we can't separate the classes. This dataset is an example of where a simpler 1D projection is adequate, so the extra complexity of projecting to 3D isn't necessarily useful.

You could include distance as a 4th dimension in the mapped space. I only used the 3D mapping $$\phi(\mathbf{x})=(x_0^2, \sqrt{2}x_0 x_1, x_1^ 2)$$. A useful distance is encoded by $$\phi_0$$ and $$\phi_2$$ because $$\phi_0+\phi_2=\mathrm{radius}^2$$.

Example data and code.

%matplotlib widget
import numpy as np
from matplotlib import pyplot as plt

from sklearn.datasets import make_circles

r_outer = 10
r_inner = 5

x_outer0 = [-8, 6]
x_outer1 = [6, -8]
x_inner0 = [3, 4]
x_inner1 = [-4, -3]

tfmd = [(c[0] ** 2, 2**0.5 * np.prod(c), c[1] ** 2) for c in coords]
return np.row_stack(tfmd)

#Create and plot the circles
factor = 0.6
X, y = make_circles(n_samples=1500, noise=0.05, factor=factor, random_state=0)

X[y==0] = X[y==0] * r_outer
X[y==1] = (X[y==1] / factor) * r_inner

#Scatter the data in 2D
plt.close('all')
plt.scatter(X[:, 0], X[:, 1], c=y, cmap='jet', marker='.', s=400, alpha=0.04, edgecolor='none')

#Highlight the test case points and their transformed values
for i, coord in enumerate([x_outer0, x_outer1, x_inner0, x_inner1]):
plt.scatter(coord[0], coord[1], marker='d', s=80, c='blue' if i <= 1 else 'red')

coord_str = f'({coord[0]}, {coord[1]})'
plt.annotate(
coord_str, coord,
xytext=(5, 0), textcoords='offset points',
va='bottom', ha='left',
fontsize=9, fontweight='bold',
)

coord_str = r'$$\phi(x, y)=$$' + f'({tfmd[0]}, {tfmd[1]}, {tfmd[2]})'
plt.annotate(
coord_str, coord,
xytext=(0, -8), textcoords='offset points',
va='top', ha='center',
fontsize=9
)
plt.gca().spines[:].set_visible(False)
plt.xlabel('x')
plt.ylabel('y')
plt.show()

#Transform cloud and view in 3D

projection='3d', proj_type='persp', focal_length=0.12
)

ax.scatter(
X_tfmd[::2, 0], X_tfmd[::2, 1], X_tfmd[::2, 2], c=y[::2],
edgecolor='none', cmap='jet', alpha=0.08, s=60
)

for i, coord in enumerate([x_outer0, x_outer1, x_inner0, x_inner1]):
ax.scatter(
tfmd[0], tfmd[1], tfmd[2],
marker='d', s=100, edgecolor='black',
c='blue' if i<=1 else 'red'
)

#Format plot
ax.view_init(elev=10, azim=-55)
ax.set_box_aspect(aspect=[2, 2, 1], zoom=1)
ax.set(xlabel=r'$$\phi_0$$', ylabel=r'$$\phi_1$$', zlabel=r'$$\phi_2$$')


1D projections:

plt.close('all')
f, axs = plt.subplots(nrows=6, figsize=(8, 6), layout='tight', sharex=True)
common_params = dict(c=y[np.argsort(y)], marker='.', s=5, cmap='jet', alpha=0.5)
y_constant = np.concatenate([np.full_like(y[y==0], 0), np.full_like(y[y==1], 1)])
y_constant = y_constant + np.random.uniform(-0.4, 0.4, y.size)

X_sort = X_tfmd[np.argsort(y)]

ax = axs[0]
ax.scatter(X_sort[:, 0], y_constant, **common_params)
ax.set_xlabel(r'$$\phi_0=x_0^2$$')

ax = axs[1]
ax.scatter(X_sort[:, 1], y_constant, **common_params)
ax.set_xlabel(r'$$\phi_1=\sqrt{2}x_0 x_1$$')

ax = axs[2]
ax.scatter(X_sort[:, 2], y_constant, **common_params)
ax.set_xlabel(r'$$\phi_2=x_1^2$$')

ax = axs[3]
ax.scatter(X_sort[:, [0, 2]].sum(axis=1), y_constant, **common_params)
ax.set_xlabel(r'$$\phi_0+\phi_2=x_0^2 + x_1^2 = \mathrm{radius}^2$$')

ax = axs[4]
ax.scatter(X_sort[:, [0, 1]].sum(axis=1), y_constant, **common_params)
ax.set_xlabel(r'$$\phi_1+\phi_0=\sqrt{2}x_0 x_1 + x_0^2$$')

ax = axs[5]
ax.scatter(X_sort[:, [1, 2]].sum(axis=1), y_constant, **common_params)
ax.set_xlabel(r'$$\phi_1 + \phi_2=\sqrt{2}x_0 x_1 + x_1^2$$')

#Formatting
[ax.spines[['top', 'left', 'right']].set_visible(False) for ax in axs]
[ax.tick_params(axis='y', labelleft=False, left=False) for ax in axs]

[ax.tick_params(axis='x', labelbottom=False, bottom=False) for ax in axs[:-1]]
axs[-1].axvline(0, 0, 10.3, color='black', lw=1, ls='--', clip_on=False)


This projects to a 4D space, where the 4th axis is $$||\phi||_2$$.

%matplotlib widget
import numpy as np
from matplotlib import pyplot as plt

from sklearn.datasets import make_circles

r_outer = 10
r_inner = 5

x_outer0 = [-8, 6]
x_outer1 = [6, -8]
x_inner0 = [3, 4]
x_inner1 = [-4, -3]

tfmd = [(c[0] ** 2, 2**0.5 * np.prod(c), c[1] ** 2) for c in coords]
tfmd = [tfmd_i + (np.linalg.norm(tfmd_i),) for tfmd_i in tfmd]
return np.row_stack(tfmd)

#Create and plot the circles
factor = 0.6
X, y = make_circles(n_samples=1500, noise=0.05, factor=factor, random_state=0)

X[y==0] = X[y==0] * r_outer
X[y==1] = (X[y==1] / factor) * r_inner

#Scatter the data in 2D
plt.close('all')
plt.scatter(X[:, 0], X[:, 1], c=y, cmap='jet', marker='.', s=400, alpha=0.04, edgecolor='none')

#Highlight the test case points and their transformed values
for i, coord in enumerate([x_outer0, x_outer1, x_inner0, x_inner1]):
plt.scatter(coord[0], coord[1], marker='d', s=80, c='blue' if i <= 1 else 'red')

coord_str = f'({coord[0]}, {coord[1]})'
plt.annotate(
coord_str, coord,
xytext=(5, 0), textcoords='offset points',
va='bottom', ha='left',
fontsize=9, fontweight='bold',
)

coord_str = r'$$\phi(x, y)=$$' + f'({tfmd[0]}, {tfmd[1]}, {tfmd[2]}, {tfmd[3]})'
plt.annotate(
coord_str, coord,
xytext=(0, -8), textcoords='offset points',
va='top', ha='center',
fontsize=9
)
plt.gca().spines[:].set_visible(False)
plt.xlabel('x')
plt.ylabel('y')
plt.show()

#Transform cloud

#View in 3D
f = plt.figure(figsize=(5, 10))
ax = f.add_subplot(211, projection='3d', proj_type='persp', focal_length=0.12)

ax.scatter(
X_tfmd[::2, 0], X_tfmd[::2, 1], X_tfmd[::2, 2], c=y[::2],
edgecolor='none', cmap='jet', alpha=0.08, s=60
)

for i, coord in enumerate([x_outer0, x_outer1, x_inner0, x_inner1]):
ax.scatter(
tfmd[0], tfmd[1], tfmd[2],
marker='d', s=100, edgecolor='black',
c='blue' if i<=1 else 'red'
)

#Format plot
ax.view_init(elev=10, azim=-55)
ax.set_box_aspect(aspect=[2, 2, 1], zoom=0.85)
ax.set(xlabel=r'$$\phi_0$$', ylabel=r'$$\phi_1$$', zlabel=r'$$\phi_2$$')

#
#Next plot: coloured by phi3
#
ax = f.add_subplot(212, projection='3d', proj_type='persp', focal_length=0.12)

ax.scatter(
X_tfmd[::2, 0], X_tfmd[::2, 1], X_tfmd[::2, 2], c=X_tfmd[::2, 3],
edgecolor='none', cmap='plasma_r', alpha=0.08, s=60
)

from matplotlib.cm import ScalarMappable
(sm := ScalarMappable(cmap='plasma_r')).set_array(X_tfmd[:, 3])
ax.figure.colorbar(
ax=ax, mappable=sm,
location='top', orientation='horizontal',
label=r'$$\phi_4$$'
)

for i, coord in enumerate([x_outer0, x_outer1, x_inner0, x_inner1]):
ax.scatter(
tfmd[0], tfmd[1], tfmd[2],
marker='d', s=100, edgecolor='black',
c='blue' if i<=1 else 'red'
)

#Format plot
ax.view_init(elev=10, azim=-55)
ax.set_box_aspect(aspect=[2, 2, 1], zoom=0.85)
ax.set(xlabel=r'$$\phi_0$$', ylabel=r'$$\phi_1$$', zlabel=r'$$\phi_2$$')

• I have updated my answer. Whilst $\phi_1$ has product terms, and therefore is sensitive to the size of each circle, it also carries information about the sign of each point. Since the circles are centred, the sign of $\phi_1$ is the same for both circles, so the points overlap and become non-separable. You could include distance as a 4th dimension in the mapped space. I only used the 3D mapping $(x_0^2, \sqrt{2}x_0 x_1, x_1^ 2)$. Distance (radius) is encoded by $\phi_0$ and $\phi_2$ because $\phi_0+\phi_1=\mathrm{radius}^2$. Commented May 24 at 10:48
• No problem. I am assuming that the circles are centred at $(0, 0)$, in which case $\phi_0+\phi_2=x_0^2+x_1^2=\mathrm{radius}^2$. Commented May 24 at 13:12
• Just to clarify...I thought the mapping function was $\phi=(x_0^2,\sqrt{2}x_0 x_1, x_1^2)$. Is the mapping function the one you've described in the comment above, which has $x_0^4$ and the two other 4th-order terms? @jiten Commented Jun 7 at 10:24
• Yes, that is correct mapping, with four dimensions, as extended from the 3 dimensions, to include norm (of the mapped data points) too. Commented Jun 7 at 11:05
• You're welcome. The %matplotlib is for interactive plots. If not supported, commenting it out will fall back on the default plot behaviour. Commented Jun 9 at 11:07