I have a dataset $\{Y_i\}$, and a set of datasets $\{\{X_j\}_Z\}$ that depend on $Z=1,2,\ldots,26$ (and $j_{max}=100$; $i_{max}=26\times 100$), meaning that there are sets, numbered by $Z$, $\{X_j\}_1$, $\{X_j\}_2$ etc., which are in general different. Each $X_{j,Z}$ corresponds to some $Y_i$.

I'd want to find values of $Z$, possibly different for each $X_j$, that will maximize the correlation (Spearman's $\rho$, preferably) between $X$ and $Y$. So I'd want to find a set of $\{Z_k\}_{k=1}^{100}$ that says to take $X_1$ from the $Z_1$-st set of $Xs$, $X_2$ from the $Z_2$-nd, $X_3$ from the $Z_3$-rd etc.

A brute force approach would be to create $26^{100}$ sets of $Z$, compute the $\rho$ for each and chose such $Zs$ that they maximize the correlation, but as $26^{100}\approx 10^{141}$ that's absurd. Even with taking $Z$ in steps of 5 would result in $5^{100}\approx 10^{70}$ possibilities, which is still unreasonable.

On the other hand, I don't necessarily need the best set of $Zs$; I'd be satisfied with just a good correlation - what is good here is not well precised, though.

Are there any ways to find such a set of $Zs$ in a more efficient way than with a random search? Some steepest descent, or anything like that?


Let $\{Y_i\}=\{a,b,c,d,e,f,g,h,i,j,k,l\}$, so $i_{max}=12$; let $\{\{X_j\}_Z\}=\{\{A_1,B_1,C_1,D_1\}_{Z=1},\{A_2,B_2,C_2,D_2\}_{Z=2},\{A_3,B_3,C_3,D_3\}_{Z=3}\}$, so $j_{max}=4$ and $Z\in\{1,2,3\}$. I need to find the $Zs$ for each $A,B,C,D$ such that the correlation with corresponding elements of $\{Y_i\}$ is best. E.g., let's say $\{A_1,B_2,C_3,D_1\}$ -- with the corresponding $\{a,f,k,d\}$ -- give the highest correlation. ($\{Y_i\}$ can be partitioned into $Z_{max}=3$ non-overlapping subsets of size $J_{max}=4$ each.) The $A,B,C,D$ can be thought of as particles, and $Z$ is some characteristic of every one of them. How can I find the $Zs$?

After writting this example I see that maybe restructuring the data into $\{X_Z\}_j\equiv\{\{A_1, A_2, A_3\}, \{B_1, B_2, B_3\}, \{C_1, C_2, C_3\}, \{D_1, D_2, D_3\}\}$ and $\{Y_i\}$ accordingly might be better.

  • $\begingroup$ You sound unclear. In the title, you want "a set of parameters" (which make one think of regressions), but in the text, you want to pick values. $\endgroup$ – ttnphns Jan 18 '18 at 13:19
  • $\begingroup$ @ttnphns Well, $Z$ are integers in some range, so I'd say that's a constraint on the possible values of the parameters. But if you have an idea how to improve the question, I'd welcome your suggestions. $\endgroup$ – corey979 Jan 18 '18 at 14:32

Canonical correlation analysis. See

Lai & Xing (2008). Statistical Models and Methods for Financial Markets,

for example.


We're looking for long answers that provide some explanation and context. Don't just give a one-line answer; explain why your answer is right, ideally with citations. Answers that don't include explanations may be removed.

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.