Can any one explain why dot product is used in neural network and what is the intitutive thought of dot product

Question

I am new to neural networks can any one explain what is the intitutive thought of dot product and why it is used in neural network

Answer 1

Dot products describe part of how neural nets work, conceptually. I'll describe the concept first using scalars, and then show how this can be re-written using the dot product.

Let's take a look at a single unit in a typical neural net. It receives inputs $\{x_1, \dots, x_n\}$ from other units, and produces an output $y$. To compute the output, we multiply each input by a corresponding weight $\{w_1, \dots, w_n\}$. The weights determine the strength of the connection from each input. We sum the weighted inputs to obtain the total amount of input, then add a bias term $b$. The final output is obtained by running this sum through an activation function $f$, which describes the way that the unit responds to the total input. The activation function is typically nonlinear, e.g. a sigmoidal or rectified linear function. So we have:

$$y = f \left ( \sum_{i=1}^n w_i x_i + b \right )$$

The weighted sum can be re-written as a dot product, which is more convenient notation, and can be computed more efficiently. Let the vector $x = [x_1, \dots, x_n]$ contain the inputs, and the vector $w = [w_1, \dots, w_n]$ contain the corresponding weights. By the definition of the dot product:

$$\sum_{i=1}^n w_i x_i = w \cdot x$$

Plug this back into the equation for the output:

$$y = f \left ( w \cdot x + b \right )$$

In practice, you wouldn't compute the outputs one by one, but for an entire layer of the neural net simultaneously. This would use matrix multiplication rather than individual dot products, which can be implemented more efficiently using numerical linear algebra libraries.

Answer 2

To answer this question we need to go back to one of the earliest neural networks the Rosenblatt’s perceptron where using vectors and the property of dot product to split hyperplanes of feature vectors were first used.

This may be familiar to many, but it for some a refresher may help

Q. What does a vector mean?

A Vector is meaningless¹ unless you specify the context - Vector Space. Assume we are thinking about something like force vector, the context is a 2D or 3D Euclidean world

Source: 3Blue1Brown’s video on Vectors

From https://towardsdatascience.com/perceptron-learning-algorithm-d5db0deab975

¹Maths is really abstract and meaningless unless you apply it to a context- this is a reason why you will get tripped if you try to get just a mathematical intuition about the neural network

The easiest way to understand it is in a geometric context, say 2D or 3D cartesian coordinates, and then extrapolate it. This is what we will try to do here.

Q. What is the connection between Matrices and Vectors?

Vectors are represented as matrices. Example here is a Euclidean Vector in three-dimensional Euclidean space (or $R^{3}$), represented as a column vector (usually) or row vector

$$ a = \begin{bmatrix} a_{1}\\a_{2}\\a_{3}\ \end{bmatrix} = \begin{bmatrix} a_{1} & a_{2} &a_{3}\end{bmatrix} $$

Q. What is a Dot product? and what does it signify ?

First the dry definitions. Algebraically, the dot product is the sum of the products of the corresponding entries of the two sequences of numbers.

if $ \vec a = \left\langle {{a_1},{a_2},{a_3}} \right\rangle $ and $\vec b = \left\langle {{b_1},{b_2},{b_3}} \right\rangle $, Then

$$ \begin{equation}\vec a\centerdot \vec b = {a_1}{b_1} + {a_2}{b_2} + {a_3}{b_3}\label{eq:eq1}\end{equation} $$

Geometrically, it is the product of the Euclidean magnitudes of the two vectors and the cosine of the angle between them

$$ \begin{equation}\vec a\centerdot \vec b = \left\| {\vec a} \right\|\,\,\left\| {\vec b} \right\|\cos \theta \label{eq:eq2} \end{equation} $$

These definitions are equivalent when using Cartesian coordinates. Here is a simple proof that follows from trigonometry - http://tutorial.math.lamar.edu/Classes/CalcII/DotProduct.aspx (You may need this article too -https://sergedesmedt.github.io/MathOfNeuralNetworks/VectorMath.html#learn_vector_math_diff)

Now to the meat of the answer, the intuition part.

If two vectors are in the same direction the dot product is positive and if they are in the opposite direction the dot product is negative.

(Try it here - https://sergedesmedt.github.io/MathOfNeuralNetworks/DotProduct.html#learn_dotproduct)

So you could use the dot product as a way to find out if two vectors are aligned or not.

That is for any two distinct sets of input feature vectors in a vector space (say we are classifying if a leaf is healthy or not based on certain features of the leaf), we can have a weight vector, whose dot product with one input feature vector of the set of input vectors of a certain class (say leaf is healthy) is positive and with the other set is negative. In essence, we are using the weight vectors to split the hyper-plane into two distinctive sets.

The initial neural network - the Rosenblatt's perceptron was doing this and could only do this - that is finding a solution if and only if the input set was linearly separable. (that constraint led to an AI winter and frosted the hopes/hype generated by the Perceptron when it was proved that it could not solve for XNOR not linearly separable)

Here is how the Rosenblatt's perceptron is modelled

Image source https://maelfabien.github.io/deeplearning/Perceptron/#the-classic-model Inputs are $x_1$ to $x_n$ , weights are some values that are learned $w_1$ to $w_n$. There is also a bias (b) which in above is -$\theta$ If we take the bias term out, the equation is

$$ \begin{equation} f(x) = \begin{cases} 1, & \text{if}\ \textbf{w}\cdot\textbf{x}+b ≥ 0 \\ 0, & \text{otherwise} \\ \end{cases} \end{equation} $$

If we take a dummy input $x_0$ as 1, then we can add the bias as a weight $w_0$ and then this bias can also fit cleanly to the sigma rule

$ y = 1 \textbf{if } \sum_i w_i x_i ≥ 0 \text{ Else } y=0 $

This is the dot product of weight and input vector w.x

(Note that dot product of two matrices (representing vectors), can be written as that transpose of one multiplied by another - you will see this way in some articles)

$$ \begin{equation} \sigma(w^Tx + b)= \begin{cases} 1, & \text{if}\ w^Tx + b ≥ 0 \\ 0, & \text{otherwise} \\ \end{cases} \end{equation} $$

Basically, all three equations are the same.

Taking from https://sergedesmedt.github.io/MathOfNeuralNetworks/RosenblattPerceptronArticle.html

So, the equation $ \bf w⋅x>b $ defines all the points on one side of the hyperplane, and $ \bf w⋅x<=b$ all the points on the other side of the hyperplane and on the hyperplane itself. This happens to be the very definition of “linear separability” Thus, the perceptron allows us to separate our feature space in two convex half-spaces

Please also see the above article from sergedesmedt. It explains also how the weights are trained.

So you can see how important dot product, and the representation of inputs and weights as vectors, are in neural networks.

This concept comes into play in modern neural networks as well.

Answer 3

The reason we use dot products is because lots of things are lines.

One way of seeing it is that the use of dot product in a neural network originally came from the idea of using dot product in linear regression.

The most frequently used definition of a line is $y = ax+b$. But this is the same as saying $b = y-ax$, which is the same as saying $b = (y,x) \cdot (1,-a)$.

So mathematically, a line is expressed with a dot product between the coordinate axes $y,x$ and some other vector. And lines are useful for linear regression. And you can view neural networks as a linear model with a nonlinear activation tacked on top.

Answer 4

It's very rough and imprecise, but I think of the dot product between two matrices or vectors as: "how much are they pulling in the same direction".

If the dot product is 0, they are pulling at a 90 degree angle. If the dot product is positive, then are pulling in the same general direction. If the dot product is negative, they are pulling away from each other.

If the dot product of normalized vectors is 1, they are the same.