Hypothetical scenario: I have a pool of millions of items, each of which has two associated values: its weight $x$ and its price $y$. I am training a network that has to predict some value $Z$ based on a collections of $3$ items. In other words,


I am training it through an unsupervised learning technique, where I select random items in a random order for every training sample. I know for a fact that the order of items has no meaning whatsoever, in other words:

$$f(x_1,y_1,x_2,y_2,x_3,y_3) = f(x_2,y_2,x_3,y_3,x_1,y_1)=f(x_3,y_3,x_1,y_1,x_2,y_2)=\ ...$$

Should I in this case share the weights between the first and second layer? So the connections from $x_1$ to the next layer have the same weights as $x_2$ and $x_3$ to the next layer (likewise for $y_1$). Or is there a reason I should not share weights?


I just simply tested if the performance of the network is the same when using weight sharing, but it is far far worse. My conclusion is that just because the function should behave symmetrically, it does not mean you can apply weight sharing. By weight sharing you lose information on the difference between the products it seems. It is important to realize that when sharing weights $\mathbf{w}$ for both input $a$ and $b$,

$$a\cdot \mathbf{w} + b \cdot \mathbf{w} = (a+b) \cdot \mathbf{w}$$

In other words, your network only knows the sum of $a$ and $b$. But I would love for someone to explain this in more detail (or debunk it).

Example: you want to train some network that can calculate multiplication $f(a,b) = a\cdot b$. You know that $f(a,b)=f(b,a)$, so you are tempted to use weight sharing. But then your network only knows $a+b$, so that it can no longer deduce what $a$ and $b$ were and cannot calculate the multiplication.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.