I'm reading a paper on deep learning-based recommender systems: Neural Collaborative Filtering. There are two sub-networks, GMF and MLP, which are fused into a unified model, by a concatenation layer. For the concatenation of the hidden layers of the two models, the authors add a hyper-parameter $\alpha$ for "determining the trade-off between the two models.":

enter image description here

enter image description here

I am wondering is this weighted concatenation necessary? I have an intuition that if there's a certain kind of proportion or weights w.r.t the two combined models, the back-propagation algorithm should be able to automatically learn and incorporate the proportion/weights into the weight parameters. So such a proportion hyper-parameter is not needed.

For example, $$\begin{bmatrix}0.3n_1 & 0.7n_2\end{bmatrix} \cdot \begin{bmatrix} w_{11} & w_{12} \\ w_{21} & w_{22}\end{bmatrix} = \begin{bmatrix}n_1 & n_2\end{bmatrix} \cdot \begin{bmatrix} 0.3w_{11} & 0.3w_{12} \\ 0.7w_{21} & 0.7w_{22}\end{bmatrix}$$

What do you think?

  • $\begingroup$ If the question is "What do you think?", then it's rather a candidate for closing as opinion based... Is it useful? The authors of the mentioned work thought so, although I agree with your reasoning that a network could probably learn that itself... $\endgroup$ – Jan Kukacka Apr 18 at 15:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.