# CCA on feature maps: Gradient w.r.t to Jacobian

Assume I have two neural networks, abstracted as two feature maps, parametrized by $$\theta_x,\theta_y$$ respectively. $$\phi_x(x;\theta_x) \in \mathbb{R}^{h_1}$$, $$\phi_y(x;\theta_y) \in \mathbb{R}^{h_2}$$ and we would like to perform canonical correlation analysis on the results of those feature maps, which means we want to maximize:

$$\max_{\theta_x, \theta_y, w_x, w_y} w_x^T C_{xy} w_y + \alpha [\text{min} (0, 1 - w_x^T C_{xx} w_x) + \text{min}(0, 1 - w_y^T C_{yy} w_y)]$$

where $$C_{xy} = \mathbb{E}[\phi_x \phi_y^T]$$, $$C_{xx} = \mathbb{E}[\phi_x\phi_x^T]$$, $$C_{yy} = \mathbb{E}[\phi_y \phi_y^T]$$

I now have to calculate the gradient of this function w.r.t. $$\theta_x$$ as a function of the Jacobian matrix $$\frac{\partial \phi_x}{\partial \theta_x}$$

Unfortunately, I don't really now to approach this. For example, what is the gradient of an expectation as a function of the Jacobian?