I was presently working on Canonical Correlation Analysis which maximizes the correlation between two variables $X$ and $Y$ of two modalities in some transformed domain using transformations $T_1$ and $T_2$. Please refer to this paper hardoon cca where the $X$ and $Y$ modality are taken to be the image and text domain and the task is to retrieve some images based on some given text data.

Specifically it solves the problem of cross-modal retrieval by mathematically solving this problem $T1,T2 = \arg \max corr(T_1X,T_2Y)$. I have already read up some variants on this such as cluster cca which is a supervised approach and can work with even unpaired data with either modality.

I am trying to solve the problem $T = \arg \max corr(X,TY)$ which essentially means finding out a single transformation $T$ which will maximize the correlation of $X$ with the transformed data $TY$. I do not want to use two transformations. Does any such method exist in the literature ?

I understand ordinary least squares (OLS) regression is a possible choice but in OLS we seek a $T$ so as to minimize the difference/error $T = \arg \min ||TY-X||^2$. My objective instead is to instead maximize the correlation but by using a single transformation ? More specifically perhaps a better question is it possible to maximize the correlation between two sets of variables using a single transformation ? Why should we always require two transformations ?

  • 1
    $\begingroup$ "Variables ... of two modalities ... in some transformed domain" is so vague and general that it would help to explain what kinds of mathematical objects $X$, $Y$, and $T$ are intended to be. If one were to guess based on the implied connection with CCA, then the answer would be to use ordinary least squares regression. $\endgroup$ – whuber Feb 25 '16 at 20:40
  • 1
    $\begingroup$ @whuber I edited the question. Perhaps it is clear now. I know about OLS but instead of minimizing the error between two domains - I wanted to maximize the correlation but using a single transformation. Could you please tell me if it can be done ? $\endgroup$ – roni Feb 26 '16 at 5:06
  • $\begingroup$ OLS is still a solution. I write "a" because there must be a family of solutions, since rescaling and recentering $Y$ will not change the correlation. $\endgroup$ – whuber Feb 26 '16 at 13:53

The answer is ordinary least squares regression (OLS).

Because most readers will be more familiar, and comfortable with, regressing $Y$ on $X$, allow me to change notation slightly. If we switch the roles of $X$ and $Y$, take transposes, and rename $T^\prime$ as $\beta$, then in these terms the question asks:

Find $\beta$ for which the correlation between the vectors $y$ and $X\beta$ is as great as possible.

Specifically, $X$ is an $n\times k$ matrix, $y$ is an $n$-vector, and $\beta$ is a $k$-vector. Because correlation does not change when values are shifted, then without any loss of generality we may assume the average of $y$ is zero and the average of each column of $X$ likewise is zero.

The geometry of least squares immediately answers the question. By definition, the least squares solution $\hat\beta$ yields the projection $\hat y = X\hat \beta$ for which $|e|^2 = |y - \hat y|^2$ is as small as possible. Assuming $\hat y \ne 0$, we may normalize $\hat y$ to unit length (by dividing it by its own length). Let $f$ be the difference $y - \hat y / |\hat y|$. Notice that its length is the distance between $y$ and a point on the unit sphere in the space spanned by the columns of $X$. Finally, let $\theta$ be the angle between $y$ and $\hat y$. Its cosine, as is well known, is the correlation coefficient $\rho$ between any nonzero multiple of $y$ and any nonzero multiple of $\hat y$.


Here are two simple results from two-dimensional Euclidean geometry:

  1. Since $|e| = |y|\sin\theta$ and OLS minimizes $|e|^2$, $|e|^2/|y|^2 = \sin^2\theta$ must be as small as possible. Consequently $\rho^2 = \cos^2\theta = 1-\sin^2\theta$ is as large as possible.

  2. Since the Law of Cosines implies $|f|^2 = |y|^2 + 1 - 2|y| \cos\theta$, $|f|^2$ must be as small as possible.

These results show that the OLS solution simultaneously gives the largest possible value of $\rho^2$ and the shortest distance from $y$ to the unit sphere in the column space of $X$.

Therefore, to maximize $\rho$ itself, use either $\hat y = X\hat \beta$ or $-\hat y = X(-\hat\beta)$: exactly one of these will produce a positive value of $\rho$. Note that any positive multiple of such a solution still solves the problem of maximizing $\rho$ (but is no longer the OLS solution).

Finally, $\hat y = 0$ (which was excluded from this analysis) implies all columns of $X$ are orthogonal to $y$, whence $X\beta$ is orthogonal to $y$ for all possible values of $\beta$. In this case, the maximum value of $\rho^2$ is $0$ and all possible values of $\hat \beta$ are solutions.

Why did $y$ need to be a vector and not also a matrix $Y$? Principally because then "the" correlation between $Y$ and $X\beta$ is not defined: there are many possible correlations between vectors in the column space of $Y$ and the column space of $X$. But we could play the same game, anyway, by considerating correlations between linear combinations $X\beta$ of columns of $X$ and $Y\alpha$ of columns of $Y$. Note that the unit spheres in the $X$-space and the $Y$-space are both closed and compact. This implies the minimum distance between those spheres actually is attained, say at points $X\hat\beta$ and $Y\hat\alpha$. The same geometric analysis given above implies these two points have maximal squared correlation: that's the CCA result.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.