2
$\begingroup$

I am wondering if it is at all possible to plot a 4D perceptron line in 2D.

Obviously, it would be impossible to observe it with all of its original information, but is there a way for me to observe say, just 2 of the variables to make a plot of one being dependant on the other?

The reason I want to do this is I want to show how well a perceptron is classifying by looking at scatter plots of the data and seeing where it would guess the division is on that particular plot.

Intuition tells me there is some way to transform the line to do this as after all, we can observe data points in just two dimensions, or am I wrong—is it impossible because the variables are dependant on one another?

$\endgroup$
2
  • $\begingroup$ Would t-SNE be an option? And what does the perceptron have to do with anything, just that you have four variables feeding your neural net and want to plot them? $\endgroup$ – Dave May 5 '20 at 22:11
  • $\begingroup$ @Dave I've never encountered t-SNE before, I'll check it out! Basically I'm just programming machine learning algorithms on the iris dataset. I just wanted to try and observe how well a single perceptron splits the data. $\endgroup$ – Max May 5 '20 at 22:15
1
$\begingroup$

Unfortunately we can't plot 4D object, but we can project them into a smaller dimensions.

2D example

For example, we can get the projection from 2D to 1D. Let's plot original data and the projection. I just go naive way and throw away information about one dimension and plot data in 1D. It yields wrong results - we lost information that we can use to separate dots.

What can we do? We can use info from perceptron about how it's separated dots. We will project the data using angles of perceptron's separation line.

Perceptron has it's $w$ (weights vector) which is just normal vector to separation line (or, in general, separation hyperplane)

We will rotate all the data and $w$ to get $w$ parallel to one of the axes. This means that hyperplane will be perpendicular to this axis and one of $w$ components will become zero.

You can get an angle using scalar product: $$\alpha = \frac{w \cdot x}{||w|| \cdot ||x||}$$ where $x$ is a unit vector of x-axis , $||\cdot||$ is a length of vector.

Then, construct rotation matrix $r$ and multiply (matrix multiplication) your data vector and $w$ by it:

$$r = \begin{bmatrix}cos(\alpha) & \pm sin(\alpha)\\\mp sin(\alpha) & cos(\alpha)\end{bmatrix}$$

Note that you need to take into account the sing of $sin(\alpha)$, because angle that returned from formula above is always positive.

Now we can project all the data on the remaining axis without any loss of information or classification accuracy.

2D to 1D example

On the last figure you can see blue dot. It's the projection of separation line. If we doing this trick right, the line (hyperplane in general) is reducing it's dimension too.

You can have the code that makes this plots (see bottom of this answer):

Your case

As long as you have linear classification, you can do this trick from any dimension to 3D, 2D or even 1D.

An algorithm:

  1. Choice the axis to eliminate
  2. Rotate data and $w$ to get this axis parallel to $w$ (get an angle and construct the rotation matrix like I did in the code)
  3. Remove info about it from data (project to the remaining axis)
  4. Repeat until you can plot it

In your case you can project once and plot 3D scatter and the separation plane, or make it twice and plot 2D scatter and the separation line.

Python code with comments

from sklearn.datasets import make_blobs
from sklearn.linear_model import LogisticRegression
import matplotlib.pyplot as plt
import numpy as np

X, y = make_blobs(n_samples=1000, centers=[[0, 0], [-8, 3]])

model = LogisticRegression()
model.fit(X, y)

def plot_line(w, w0, X, ax):
    '''Plots sep.line given w and w0'''
    x0 = min(X[:,0])
    x1 = max(X[:,0])
    x = np.linspace(x0, x1, 2)

    # w is a normal vector (perpendicular to the line that we need)
    # let's rorate it by 90 degrees
    rot90 =  np.array([[0,1],[-1,0]])
    l = np.dot(rot90, w).flatten()

    # l[0], l[1] and w0 are the coeffs of a General form of equation.
    # We need to get slope-intercept form:
    k = l[1]/l[0]
    ax.plot(x, k*x - w0/l[0])
    ax.set_ylim(min(X[:,1]), max(X[:,1]))

def get_angle(a, b):
    '''Returns angle between two vectors'''
    # Here we use scalar product to get the angle
    res = np.sum(a*b)/(np.linalg.norm(a) * np.linalg.norm(b))
    return np.arccos(res)


# Obtaining weights
w = model.coef_.T
w0 = model.intercept_

# Unit vector of y axis
y_ax = np.array([[0,1]]).T

# Angle between y axis and w
angle = get_angle(w, y_ax)

# Rotation matrix with 'angle'
# We need to take into account the direction of the rotation
k = np.sign(w.prod())
rot_m = np.array([[np.cos(angle), k*np.sin(angle)],
                  [-k*np.sin(angle), np.cos(angle)]])

# Now we can rotate w and all of the X dots
new_X = np.dot(X, rot_m)
new_w = np.dot(w.T, rot_m).T

# Plotting original data
f, (a0, a1, a2, a3) = plt.subplots(1, 4, gridspec_kw={'width_ratios': [3,1,3,1]}, figsize=(10,4))
plot_line(w, w0, X, a0)
a0.scatter(X.T[0], X.T[1], c=y)
a0.set_title('Original data')

# Plotting 'bad' projection
# I used shifted list to prevent overlapping dots on the figure
step = 0.1
shifted = [-step if i==0 else step for i in y]
a1.scatter(shifted, X.T[1], c=y)
a1.set_title('Incorrect 1D\nprojection')
a1.set_xlim(-1,1)

# Plotting rotated version
plot_line(new_w, w0, new_X, a2)
a2.scatter(new_X.T[0], new_X.T[1], c=y)
a2.set_title('Rotated 2D')

# Plotting 'good' projection
step = 0.1
shifted = [-step if i==0 else step for i in y]
a3.scatter(shifted, new_X.T[1], c=y)
a3.scatter(0, -w0/new_w[1], s=100)
a3.set_title('Correct 1D\nprojection')
a3.set_xlim(-1,1)
plt.tight_layout()
$\endgroup$
1
  • $\begingroup$ This answer is incredibly well described and demonstrated thank you for that, it is more than I could have expected. It makes sense to me now that rotating the data such that one axis can be set to 0 for all values allows us to project it in lower dimensions. I think a nice brush up of projective geometry is in order! $\endgroup$ – Max May 11 '20 at 15:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.