So how can I project the data with the eigenvectors from LDA?

Question

I have data that look like this.

And my goal is to reduce this 3D dimension into 2D dimension so it might looks like this. Turning the angle so the distance between all classes becomes maximum.

So therefore I have made a MATLAB-code to use:

function [W] = lda(varargin)
  % Check if there is any input
  if(isempty(varargin))
    error('Missing inputs')
  end

  % Get impulse response
  if(length(varargin) >= 1)
    X = varargin{1};
  else
    error('Missing data X')
  end

  % Get the sample time
  if(length(varargin) >= 2)
    y = varargin{2};
  else
    error('Missing class ID y');
  end

  % Get the sample time
  if(length(varargin) >= 3)
    c = varargin{3};
  else
    error('Missing amount of components');
  end

  % Get size of X
  [row, column] = size(X);

  % Create average vector mu_X = mean(X, 2)
    mu_X = mean(X, 2);

    % Count classes
    amount_of_classes = y(end) + 1;

    % Create scatter matrices Sw and Sb
    Sw = zeros(row, row);
    Sb = zeros(row, row);

    % How many samples of each class
    samples_of_each_class = zeros(1, amount_of_classes);
    for i = 1:column
      samples_of_each_class(y(i) + 1) = samples_of_each_class(y(i) + 1) + 1; % Remove +1 if you are using C
    end

    % Iterate all classes
    shift = 1;
    for i = 1:amount_of_classes
      % Get samples of each class
      samples_of_class = samples_of_each_class(i);

      % Copy a class to Xi from X
      Xi = X(:, shift:shift+samples_of_class - 1);

      % Shift
      shift = shift + samples_of_class;

      % Get average of Xi
      mu_Xi = mean(Xi, 2);

      % Center Xi
      Xi = Xi - mu_Xi;

      % Copy Xi and transpose Xi to XiT and turn XiT into transpose
      XiT = Xi';

      % Create XiXiT = Xi*Xi'
      XiXiT = Xi*XiT;

      % Add to Sw scatter matrix
      Sw = Sw + XiXiT;

      % Calculate difference
      diff = mu_Xi - mu_X;

      % Borrow this matrix and do XiXiT = diff*diff'
      XiXiT = diff*diff';

      % Add to Sb scatter matrix - Important to multiply XiXiT with samples of class
      Sb = Sb + XiXiT*samples_of_class;
    end

    % Use cholesky decomposition to solve generalized eigenvalue problem Ax = lambda*B*v
  Sw = Sw + eye(size(Sw));
    L = chol(Sw, 'lower');
    Y = linsolve(L, Sb);
    Z = Y*inv(L');
    [V, D] = eig(Z);

    % Sort eigenvectors descending by eigenvalue
    [D, idx] = sort(diag(D), 1, 'descend');
    V = V(:,idx);

    % Get components W
    W = V(:, 1:c);
end

And a working example

% Data for the first class
x1 = 2*randn(50, 1);
y1 = 50 + 5*randn(50, 1);
z1 = (1:50)';

% Data for the second class
x2 = 5*randn(50, 1);
y2 = -4 + 2*randn(50, 1);
z2 = (100:-1:51)';

% Data for the third class
x3 = 15 + 3*randn(50, 1);
y3 = 50 + 2*randn(50, 1);
z3 = (-50:-1)';

% Create the data matrix
X = [x1, y1, z1, x2, y2, z2, x3, y3, z3];

% Create class ID, indexing from zero
y = [0 0 0 1 1 1 2 2 2];

% How many dimension 
c = 2;

% Plot original data
close all
scatter3(X(:, 1), X(:, 2), X(:, 3), 'r')
hold on
scatter3(X(:, 4), X(:, 5), X(:, 6), 'g')
hold on
scatter3(X(:, 7), X(:, 8), X(:, 9), 'b')

% Do LDA - Now what?
W = lda(X, y, c);

The $W$ matrix contains a lot of eigenvectors. What I need to do is to multiply $W$ with $X$, but the problem is that It's not possible. I can make the $W$ into transpose, but still, I don't think that's the right method to use.

So how can I project the data with the eigenvectors from LDA?

Answer 1

I'm late here, but this link may be helpful to you if you're still exploring this.

Modified from Alexander Jamieson's answer there:

Mdl = fitcdiscr(X,y)
[W, LAMBDA] = eig(Mdl.BetweenSigma, Mdl.Sigma) %Must be in the right order! 
lambda = diag(LAMBDA);
[lambda, SortOrder] = sort(lambda, 'descend')
W = W(:, SortOrder);
Y = X*W;

You may be looking instead for X*W.