2
$\begingroup$

How does one write the mathematical formula for conv1d used in PyTorch, including parameters like stride length and padding?

For instance, I can write

import torch

input1d = torch.tensor([[[1,2,3,4,5]]])
filter1d = torch.tensor([[[4,3]]])

torch.nn.functional.conv1d(input1d, filter1d)

Output:

tensor([[[10, 17, 24, 31]]])

This is equivalent to

torch.einsum("abcd,abd->abc", input1d.unfold(2,2,1), filter1d)

How does this translate to mathematical operations? I am particularly uncertain about input1d.unfold(2,2,1).

I have the following, but I am quite sure it is incorrect $$ x = \mathrm{input1d} \in \mathbb{R}^{1 \times 1 \times 5} \\ w = \mathrm{filter1d} \in \mathbb{R}^{1 \times 1 \times 2} \\ \int_{-\infty}^{\infty} \sum_{d} x_{abcd}\,w_{abd} $$

$\endgroup$

1 Answer 1

5
+50
$\begingroup$

So a few things here:

Firstly, it is worth mentioning for the sake of transparency that torch.nn.functional.conv1d is more strictly cross-correlation rather than convolution, which involves flipping the filter, in a more broad usage. However, for CNN applications, the distinction is not important, and so the term convolution is overwhelmingly overloaded to mean cross-correlation (no flip).

The second thing that is worth mentioning is that ``mathematical operations'' can mean a few things as well. A relatively straightforward approach would be using

$$(\vec{a}\star\vec{b})[i] = \vec{a}\left[i:i+\text{len}(\vec{b})\right]\cdot \vec{b}$$

where $\vec{a}\star\vec{b}$ is the desired result of cross-correlating $\vec{a}$ and $\vec{b}$ (given as a vector-describing function $\vec{a}\star\vec{b}:\mathbb{Z} \rightarrow\mathbb{R}$ that maps the vector indices in $\mathbb{Z}$ to results in $\mathbb{R}$). $:$ and $\cdot$ are range-indexing and dot product operations, which can be broken down into summation notation as desired.

Incorporating stride, this is

$$\text{corr}(\vec{a}, \vec{b}, \text{stride})[i] = \vec{a}\left[i\cdot\text{stride}:i\cdot\text{stride}+\text{len}(\vec{b})\right]\cdot \vec{b}$$

After breaking this down into summation notation, you should be able to extend the dot product operation to consider padded zeros (or more precisely, only sum/multiply existing elements). I will circle back to address this and your question involving matrix multiplication and reindexing when I'm free from work.

$\endgroup$
4
  • 2
    $\begingroup$ (+1) Thank you for mentioning that torch.nn.functional.conv1d is the cross-correlation. Too bad they have a misleading naming convention. $\endgroup$
    – Galen
    Commented Dec 7, 2021 at 17:05
  • $\begingroup$ Definitely misleading, but I think there's decent reason for it. IIRC, the seminal paper(s) for CNNs (or possibly for vision, IDR) actually did use the more broad usage of convolution. In the context of CNNs, though, the two have the same utility, and so since you can swap out convolution and cross-correlation as long as you're consistent, people did. Thus, in pretty much all the stuff out there about CNNs you actually /can/ use convolution, it's just easier to think with cross-correlation. (Personally, however, going from ECE -> ML, I was very surprised by the convention at first) $\endgroup$ Commented Dec 7, 2021 at 17:40
  • $\begingroup$ @DavidMcKnight Great explanation. I admit that writing convolution as a mathematical operation is a bit ambiguous. But the indexing operation on $\vec{a}$ is clear to me, it reminds me of slice notation in python. The convolution can be simulated by using a circulant matrix, which I thought was a bit more more intuitive because the numbers in $\vec{a}$ and $\vec{b}$ can be written out. However, I am confused on how to describe a function for generating the circulant matrix. $\endgroup$
    – Kevin
    Commented Jan 3, 2022 at 15:53
  • $\begingroup$ @DavidMcKnight I posted a question about this here: stats.stackexchange.com/questions/557703/… $\endgroup$
    – Kevin
    Commented Jan 3, 2022 at 15:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.