Using canonical correlation analysis (CCA) to find matches

Question

I have a training dataset of images: X (Visual) and Y (Infrared). Each set has $300$ training examples. I extract feature vectors from both sets of images. Thus my X and Y datasets are respectively $300\times 1920$ and $300\times 1536$ where $300$ is the number of sample observations (in this case images) and $1920$ & $1536$ are respectively the length of the feature vectors in visual and infrared spectrum.

A testing dataset consists of images of different subjects in both visual spectrum and infrared spectrum. I will have the visual spectrum data as my gallery. I will have the infrared images as the probe data. For each probe image, I need to retrieve the corresponding visual image from the gallery by using some kind of similarity measure.

Basic algorithm idea:

1. Start Training Phase. Read the images from visual and infrared spectrum dataset.
2. Get the feature vectors by using desired descriptors and populate the matrices X and Y.
3. Use CCA for subspace learning. Get the projection matrices Wx and Wy.
4. Start Testing Phase. Given the gallery images (visual), read these images and use the Wx transformation matrix to convert them into the CCA subspace.
5. For each probe image, convert it into the CCA subspace by using the Wy transformation and compare with each images of the gallery and compute a matching score.
6. The image (or its label) in the gallery having the maximum score is returned.

Could anyone tell me if my approach is correct or not? Pointing me in the right direction would also be helpful. Please see the following paper for reference: Yi et al. 2007, Face Matching Between Near Infrared and Visible Light Images.

---

I work in Matlab and use the following command to perform CCA:

[Wx,Wx,r,U,V] = canoncorr(X,Y);   %// DO CCA

The output I get is this :

  Name           Size             Bytes   Class     Attributes

  Wx         1920x297            890880  double              
  Wx         1536x297            712704  double              
  U           300x297             27840  double              
  V           300x297             27840  double              
  r             1x297               464  double       

As was explained to me on StackOverflow:

1. The projection matrices are Wx and Wy since they transform X and Y into the new space.

2. The resulting projections of X and Y into the new space are U and V, respectively.

3. The r vector represents the entries of the correlation matrix between U and V, which is a diagonal matrix.

Answer 1

This looks like a possible approach.

CCA will find pairs of vectors $(\mathbf w, \mathbf v)$ such that projections $\mathbf X \mathbf w$ and $\mathbf Y \mathbf v$ have maximal possible correlations (the pairs will be ordered in the order of decreasing correlations). Projection vectors are normalized such that the variance of $\mathbf X \mathbf w$ and of $\mathbf Y \mathbf v$ is equal to $1$. This means that projections are not only correlated, but "on the same scale" and hence can be directly compared.

Some things to keep in mind is that: (1) you can only center your test data with the mean of the training data; (2) in high dimensions CCA is prone to overfitting and it will be a good idea either to use regularized CCA or to preprocess the data with PCA.

Here is a very simple Matlab script implementing this approach:

% // Using Fisher Iris data.
% // X will be petal measurements, Y will be sepal measurements
load fisheriris
trainN = 75; %// using half of the data for training and half for testing

centerTrain = mean(meas(1:trainN,:));
X = bsxfun(@minus, meas(:,1:2), centerTrain(1:2));
Y = bsxfun(@minus, meas(:,3:4), centerTrain(3:4));

% // This computes CCA on the training data
[A,B,r] = canoncorr(X(1:trainN,:), Y(1:trainN,:));

% // Projecting the test data
Xtestpr = X(trainN+1:end, :) * A;
Ytestpr = Y(trainN+1:end, :) * B;

% // Loop over all train samples
correct = 0;
for i=1:size(Xtestpr,1)
    % // Using only the first CCA projection, find the sample in Y
    % // closest to the one in X. Euclidean distance is used as a
    % // similarity measure.
    [~, ind] = min(sum((Xtestpr(i,1) - Ytestpr(:,1)).^2, 2));

    % // if classified correctly
    if ind==i
        correct = correct+1;
    end    
end

%// compute the probability that so many correct matchings
%// could be obtained by chance
pval = 1 - binocdf(correct-1, size(Xtestpr,1), 1/size(Xtestpr,1));

%// compute confidence interval on correct matching rate
[~, cinf] = binofit(correct, size(Xtestpr,1));

This gives me $6$ correct classifications out of $75$, which does not sound like a lot, but still is clearly significant with a p-value of $0.0005$. Confidence interval on matching probability is $(0.03, 0.17)$.