# How to find a set of parameters that maximize the correlation between two sets?

I have a dataset $\{Y_i\}$, and a set of datasets $\{\{X_j\}_Z\}$ that depend on $Z=1,2,\ldots,26$ (and $j_{max}=100$; $i_{max}=26\times 100$), meaning that there are sets, numbered by $Z$, $\{X_j\}_1$, $\{X_j\}_2$ etc., which are in general different. Each $X_{j,Z}$ corresponds to some $Y_i$.

I'd want to find values of $Z$, possibly different for each $X_j$, that will maximize the correlation (Spearman's $\rho$, preferably) between $X$ and $Y$. So I'd want to find a set of $\{Z_k\}_{k=1}^{100}$ that says to take $X_1$ from the $Z_1$-st set of $Xs$, $X_2$ from the $Z_2$-nd, $X_3$ from the $Z_3$-rd etc.

A brute force approach would be to create $26^{100}$ sets of $Z$, compute the $\rho$ for each and chose such $Zs$ that they maximize the correlation, but as $26^{100}\approx 10^{141}$ that's absurd. Even with taking $Z$ in steps of 5 would result in $5^{100}\approx 10^{70}$ possibilities, which is still unreasonable.

On the other hand, I don't necessarily need the best set of $Zs$; I'd be satisfied with just a good correlation - what is good here is not well precised, though.

Are there any ways to find such a set of $Zs$ in a more efficient way than with a random search? Some steepest descent, or anything like that?

Example:

Let $\{Y_i\}=\{a,b,c,d,e,f,g,h,i,j,k,l\}$, so $i_{max}=12$; let $\{\{X_j\}_Z\}=\{\{A_1,B_1,C_1,D_1\}_{Z=1},\{A_2,B_2,C_2,D_2\}_{Z=2},\{A_3,B_3,C_3,D_3\}_{Z=3}\}$, so $j_{max}=4$ and $Z\in\{1,2,3\}$. I need to find the $Zs$ for each $A,B,C,D$ such that the correlation with corresponding elements of $\{Y_i\}$ is best. E.g., let's say $\{A_1,B_2,C_3,D_1\}$ -- with the corresponding $\{a,f,k,d\}$ -- give the highest correlation. ($\{Y_i\}$ can be partitioned into $Z_{max}=3$ non-overlapping subsets of size $J_{max}=4$ each.) The $A,B,C,D$ can be thought of as particles, and $Z$ is some characteristic of every one of them. How can I find the $Zs$?

After writting this example I see that maybe restructuring the data into $\{X_Z\}_j\equiv\{\{A_1, A_2, A_3\}, \{B_1, B_2, B_3\}, \{C_1, C_2, C_3\}, \{D_1, D_2, D_3\}\}$ and $\{Y_i\}$ accordingly might be better.

• You sound unclear. In the title, you want "a set of parameters" (which make one think of regressions), but in the text, you want to pick values. – ttnphns Jan 18 '18 at 13:19
• @ttnphns Well, $Z$ are integers in some range, so I'd say that's a constraint on the possible values of the parameters. But if you have an idea how to improve the question, I'd welcome your suggestions. – corey979 Jan 18 '18 at 14:32