The problem is: For a pair of random variable $(X,Y)$, where $X \in \mathbb{R}^p$, $Y \in \mathbb{R}^q$, write $\Sigma_{XX}=var(X)$, $\Sigma_{YY}=var(Y)$, $\Sigma_{YX}=cov(Y,X)$.
Define $(\alpha^*, \beta^*)=$ argmax $\alpha^T\Sigma_{YX}\beta$ s.t. $\alpha^T\Sigma_{YY}\alpha=1$, $\beta^T\Sigma_{XX}\beta=1$.
Assume that $\Sigma_{YY}>0$, $\Sigma_{XX}>0$. How can I solve for $(\alpha^*, \beta^*)$ if $\Sigma_{YY}$, $\Sigma_{XX}$, $\Sigma_{YX}$ are known?
Recall that the maximum singular value of a matrix $A$ is the maximizer of $$ \sigma_{\max}=\max_{x,y}x^{T}Ay $$
over the set $\left\Vert x\right\Vert =\left\Vert y\right\Vert =1$. The constraint sets your care about are a little different though, as they are ellipsoids rather than spheres.
The way to solve the problem you state is to apply a change of variables, changing $a\rightarrow x\Sigma_{XX}^{-1/2}$, and similarly for $b\rightarrow\Sigma_{YY}^{-1/2}y$ (we use the (principal) matrix square root here, which is well defined for positive definite covariance matrices). You then get a singular value problem of the above form which can be solved in terms of $x$ and $y$, and the solution mapped back to $a$ and $b$ vectors.
[self-study]tag (you'll have to remove an existing tag) & read its wiki. Then tell us what you understand thus far, what you've tried & where you're stuck. We'll provide hints to help you get unstuck. $\endgroup$