3
$\begingroup$

I am currently studying this paper (page 53) (mirror), in which the suggest convolution to be done in a special manner.

This is the formula:

\begin{equation} \tag{1}\label{1} q_{j,m} = \sigma \left(\sum_i \sum_{n=1}^{F} o_{i,n+m-1} \cdot w_{i,j,n} + w_{0,j} \right) \end{equation}

Here is their explanation:

As shown in Fig. 4.2 (mirror), all input feature maps (assume I in total), $O_i (i = 1, · · · , I)$ are mapped into a number of feature maps (assume $J$ in total), $Q_j (j = 1, · · · , J)$ in the convolution layers based on a number of local filters ($I × J$ in total), $w_{ij}$ $(i = 1, · · · , I; j = 1, · · · , J)$. The mapping can be represented as the well-known convolution operation in signal processing. Assuming input feature maps are all one dimensional, each unit of one feature map in the convolution layer can be computed as equation $\eqref{1}$ (equation above).

where $o_{i,m}$ is the $m$-th unit of the $i$-th input feature map $O_i$, $q_{j,m}$ is the $m$- th unit of the $j$-th feature map $Q_j$ of the convolution layer, $w_{i,j,n}$ is the $n$th element of the weight vector, $w_{i,j}$, connecting the $i$th feature map of the input to the $j$th feature map of the convolution layer, and $F$ is called the filter size which is the number of input bands that each unit of the convolution layer receives.

So far so good:

What i basically understood from this is what I've tried to illustrate in this image.

enter image description here

So I am trying to implement this, and this network structure is called Full weight sharing..

We call the weight sharing scheme in Fig. 4.3, as described in the previous section, the full weight sharing (FWS) scheme. This is the standard CNN scheme used in image processing since the same patterns may appear at any location in an image. However, speech signals typically behave quite differently in various frequency bands. Using different sets of weights for different frequency bands may be more suitable since it allows the detection of differ- ent feature patterns at different frequencies. (page 62)

So apperently should the weight be reused, but that doesn't seem to be the case looking at equation $\eqref{1}$, as the weight has a index that changes with the summations..

and weight matrix illustrated in the paper at page 58 (figure 4.5), shows that the matrix has $(I \cdot F \times J )$, so basically three dimensions.

enter image description here

So where is the weight being shared?.. It looks more like that each weight is unique, and every datapoint is processed uniquely?

My idea of weight sharing would be something like this:

enter image description here

In which the same filter/weight are passed along the input data with some stride.

This would also be the case here, if weight matrix didn't have the $j$ index, which make the processing each units unique.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.