1
$\begingroup$

Background: I have a 2-dimensional 2-class classification problem, the following training sample vectors are given:

$$D_1 = \left\{ \begin{bmatrix} 0 \\ -1 \end{bmatrix}, \begin{bmatrix} 3 \\ -2\end{bmatrix}, \begin{bmatrix} 0 \\ 2 \end{bmatrix}, \begin{bmatrix} -2 \\ 1\end{bmatrix}, \begin{bmatrix} 2 \\ -1 \end{bmatrix} \right\} $$ and $$ D_2=\left\{ \begin{bmatrix} 6 \\ 0 \end{bmatrix}, \begin{bmatrix} 3 \\ 2\end{bmatrix}, \begin{bmatrix} 9 \\ 1 \end{bmatrix}, \begin{bmatrix} 7 \\ 4\end{bmatrix}, \begin{bmatrix} 5 \\ 5 \end{bmatrix} \right\}$$

I used Fisher linear discriminant analysis method to calculate a transform vector $w$.

I wrote the code and I got

w =

    -0.2308
    -0.2033

Question: How can I calculate the projections of all the data samples in the resulting 1-dimensional space?

Do I just multiply the data with the $w$ and get new $D_1$ and $D_2$?

I found a slide, can someone explain this to me?

The projection from a $d$-D to $(c\!-\!1)$-D can be expressed as
$y_i = w_i^t x \quad i=1, \dots, c-1 \quad \text{or}$
$y=W^t x$, where $w_i$ are columns of $d \times (c\!-\!1)$ matrix $W$.

$\endgroup$
3
  • $\begingroup$ Did you attempt to search for answers already published? $\endgroup$
    – ttnphns
    Commented Mar 27, 2017 at 18:00
  • $\begingroup$ I did. I am just not sure about what it means to project to 1D space? will we have a single number for each point instead of [x1 x2] $\endgroup$
    – Pete
    Commented Mar 27, 2017 at 18:02
  • $\begingroup$ I actually found something and I added it to the question, can you explain this? $\endgroup$
    – Pete
    Commented Mar 27, 2017 at 18:10

2 Answers 2

2
$\begingroup$

"Projection" here can be an ambiguous term. It could mean projection onto a 1D space, or onto the subspace spanned by the line. The dot product, as you mention gives the projection in the former case. In the later case, for each data point, you are essentially finding the closest point on the line to it in the original 2D space. See here for calculation of the closest point.

Here is some python code to calculate both cases.

import matplotlib.pyplot as plt
import numpy as np

D = np.matrix("0 -1; 3 -2; 0 2; -1 1; 2 -1;" +
              "6 0; 3 2; 9 1; 7 4; 5 5")

w = np.array([-0.2308, -0.2033])
slope = w[1]/w[0]

# This gives the projection onto the 1D space
np.dot(D, w)

# Plot point
plt.plot(D[0:5, 0], D[0:5, 1], 'bo')
plt.plot(D[5:, 0], D[5:, 1], 'ro')
plt.plot([-1, 10], [-slope, 10*slope], color='k', linestyle='-', linewidth=1)

# Subspace Projection https://en.wikibooks.org/wiki/Linear_Algebra/Orthogonal_Projection_Onto_a_Line
DP = dot(D, w).T * w / (dot(w,w))

# Plot projected points
plt.plot(DP[0:5, 0], DP[0:5, 1], 'bs')
plt.plot(DP[5:, 0], DP[5:, 1], 'rs')
plt.gca().set_aspect('equal', adjustable='box')
plt.draw()

enter image description here

$\endgroup$
1
  • 1
    $\begingroup$ Why is the slope = w[1] / w[0]? $\endgroup$
    – thenac
    Commented Dec 29, 2019 at 19:46
0
$\begingroup$

import numpy as np import matplotlib.pyplot as plt

Define the two classes

C1 = np.array([[0, -1], [3, -2], [0, 2], [-2, 1], [2, -1]]) C2 = np.array([[6, 0], [3, 2], [9, 1], [7, 4], [5, 5]])

Calculate the mean of each class

mean_C1 = np.mean(C1, axis=0) mean_C2 = np.mean(C2, axis=0)

Calculate the within-class scatter matrix

Sw = np.dot((C1 - mean_C1).T, (C1 - mean_C1)) + np.dot((C2 - mean_C2).T, (C2 - mean_C2))

Calculate the between-class scatter matrix

Sb = np.dot((mean_C1 - mean_C2).reshape(2, 1), (mean_C1 - mean_C2).reshape(1, 2))

Calculate the optimal projection direction V

eigenvalues, eigenvectors = np.linalg.eig(np.dot(np.linalg.inv(Sw), Sb)) V = eigenvectors[:, np.argmax(eigenvalues)]

Project the data onto the optimal line of projection

proj_C1 = np.dot(C1, V) proj_C2 = np.dot(C2, V)

Calculate the minimum and maximum values of the 1D data

min_val = min(np.min(proj_C1), np.min(proj_C2)) max_val = max(np.max(proj_C1), np.max(proj_C2))

Plot the 1D data and the optimal line of projection

plt.scatter(proj_C1, np.zeros_like(proj_C1), color='blue') plt.scatter(proj_C2, np.zeros_like(proj_C2), color='red') plt.plot([min_val, max_val], [0, 0], color='green') plt.show()

$\endgroup$
1
  • 2
    $\begingroup$ Code in answers is fine, commented code is better - but we are looking for answers on the conceptual question, so if you can edit your post to address the statistical question itself, with the code as an illustration, that would be most useful. $\endgroup$ Commented Mar 29, 2023 at 10:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.