Fisher linear discriminant analysis method projections of all the data samples to 1-dimensional space

Question

Background: I have a 2-dimensional 2-class classification problem, the following training sample vectors are given:

$$D_1 = \left\{ \begin{bmatrix} 0 \\ -1 \end{bmatrix}, \begin{bmatrix} 3 \\ -2\end{bmatrix}, \begin{bmatrix} 0 \\ 2 \end{bmatrix}, \begin{bmatrix} -2 \\ 1\end{bmatrix}, \begin{bmatrix} 2 \\ -1 \end{bmatrix} \right\} $$ and $$ D_2=\left\{ \begin{bmatrix} 6 \\ 0 \end{bmatrix}, \begin{bmatrix} 3 \\ 2\end{bmatrix}, \begin{bmatrix} 9 \\ 1 \end{bmatrix}, \begin{bmatrix} 7 \\ 4\end{bmatrix}, \begin{bmatrix} 5 \\ 5 \end{bmatrix} \right\}$$

I used Fisher linear discriminant analysis method to calculate a transform vector $w$.

I wrote the code and I got

w =

    -0.2308
    -0.2033

Question: How can I calculate the projections of all the data samples in the resulting 1-dimensional space?

Do I just multiply the data with the $w$ and get new $D_1$ and $D_2$?

I found a slide, can someone explain this to me?

The projection from a $d$-D to $(c\!-\!1)$-D can be expressed as
$y_i = w_i^t x \quad i=1, \dots, c-1 \quad \text{or}$
$y=W^t x$, where $w_i$ are columns of $d \times (c\!-\!1)$ matrix $W$.

Answer 1

"Projection" here can be an ambiguous term. It could mean projection onto a 1D space, or onto the subspace spanned by the line. The dot product, as you mention gives the projection in the former case. In the later case, for each data point, you are essentially finding the closest point on the line to it in the original 2D space. See here for calculation of the closest point.

Here is some python code to calculate both cases.

import matplotlib.pyplot as plt
import numpy as np

D = np.matrix("0 -1; 3 -2; 0 2; -1 1; 2 -1;" +
              "6 0; 3 2; 9 1; 7 4; 5 5")

w = np.array([-0.2308, -0.2033])
slope = w[1]/w[0]

# This gives the projection onto the 1D space
np.dot(D, w)

# Plot point
plt.plot(D[0:5, 0], D[0:5, 1], 'bo')
plt.plot(D[5:, 0], D[5:, 1], 'ro')
plt.plot([-1, 10], [-slope, 10*slope], color='k', linestyle='-', linewidth=1)

# Subspace Projection https://en.wikibooks.org/wiki/Linear_Algebra/Orthogonal_Projection_Onto_a_Line
DP = dot(D, w).T * w / (dot(w,w))

# Plot projected points
plt.plot(DP[0:5, 0], DP[0:5, 1], 'bs')
plt.plot(DP[5:, 0], DP[5:, 1], 'rs')
plt.gca().set_aspect('equal', adjustable='box')
plt.draw()

Answer 2

import numpy as np import matplotlib.pyplot as plt

Define the two classes

C1 = np.array([[0, -1], [3, -2], [0, 2], [-2, 1], [2, -1]]) C2 = np.array([[6, 0], [3, 2], [9, 1], [7, 4], [5, 5]])

Calculate the mean of each class

mean_C1 = np.mean(C1, axis=0) mean_C2 = np.mean(C2, axis=0)

Calculate the within-class scatter matrix

Sw = np.dot((C1 - mean_C1).T, (C1 - mean_C1)) + np.dot((C2 - mean_C2).T, (C2 - mean_C2))

Calculate the between-class scatter matrix

Sb = np.dot((mean_C1 - mean_C2).reshape(2, 1), (mean_C1 - mean_C2).reshape(1, 2))

Calculate the optimal projection direction V

eigenvalues, eigenvectors = np.linalg.eig(np.dot(np.linalg.inv(Sw), Sb)) V = eigenvectors[:, np.argmax(eigenvalues)]

Project the data onto the optimal line of projection

proj_C1 = np.dot(C1, V) proj_C2 = np.dot(C2, V)

Calculate the minimum and maximum values of the 1D data

min_val = min(np.min(proj_C1), np.min(proj_C2)) max_val = max(np.max(proj_C1), np.max(proj_C2))

Plot the 1D data and the optimal line of projection

plt.scatter(proj_C1, np.zeros_like(proj_C1), color='blue') plt.scatter(proj_C2, np.zeros_like(proj_C2), color='red') plt.plot([min_val, max_val], [0, 0], color='green') plt.show()