Understand neural network in a 'mathematical' way

Question

This is a soft question. But as I read papers/reports about neural network used for pictures. Often there are comments like 'the first layer of the neural network captures the edge/shape information' etc.

I am just wondering what exactly the 'edge/shape' is if we just view an image as a matrix. Meanwhile, is there any intuition that a neural network could capture the 'edge/shape' if we just view a neural network as a composition of linear and non-linear functions?

Thanks so much!

Answer 1

I am just wondering what exactly the 'edge/shape' is if we just view an image as a matrix.

Basically, neural networks are learning so-called filters, which is a concept from traditional computer vision. In traditional computer vision, these were developed manually. For instance, if you have a convolutional neural network with a 3x3 kernel that has the following weights:

$$ k = \begin{bmatrix} -1 & -2 & -1 \\ 0 & 0 & 0\\ 1 & 2& 1 \end{bmatrix} $$

then, you have a so-called horizontal Sobel filter that can detect vertical edges if you use it for convolution (sliding it over the input image). For instance, see the illustration below:

(Image source: https://i.stack.imgur.com/RBFN4.png)

Let's take a practical example using the vertical Sobel (vertical edge detector):

import torch
import numpy as np
from torchvision import datasets
from torchvision import transforms
from torch.utils.data import DataLoader


conv = torch.nn.Conv2d(1, 1, kernel_size=3)
conv.weight = torch.nn.Parameter(torch.tensor([[[[-1., 0., 1.],
                                                 [-2., 0., 2.],
                                                 [-1., 0., 1.]]]]))

Let's load an input image first:

train_dataset = datasets.MNIST(root='data', 
                               train=True, 
                               transform=transforms.ToTensor(),
                               download=True)

train_loader = DataLoader(dataset=train_dataset, 
                          batch_size=10, 
                          shuffle=False)

for images, labels in train_loader:
    break


import matplotlib.pyplot as plt

nhwc_img = np.transpose(images[7], axes=(1, 2, 0))
nhw_img = np.squeeze(nhwc_img.numpy(), axis=2)
plt.imshow(nhw_img, cmap='Greys');

And now apply the vertical Sobel filter we defined above:

with torch.set_grad_enabled(False):
    nhwc_img = conv(images[7].view(1, 1, 28, 28))
    plt.imshow(torch.squeeze(nhwc_img), cmap='Greys')

As you can see, the signal in the output is primarily there where the input had vertical edges. Of course, the network will probably not learn Sobel filters but learn to detect filters in a way that are useful for minimizing a loss function on the given task. Hopefully, this provides some intuition though.