To answer this question we need to go back to one of the earliest neural networks the Rosenblatt’s perceptron where using vectors and the property of dot product to split hyperplanes of feature vectors were first used.
This may be familiar to many, but it for some a refresher may help
Q. What does a vector mean?
A Vector is meaningless¹ unless you specify the context - Vector Space. Assume we are thinking about something like force vector, the context is a 2D or 3D Euclidean world
Source: 3Blue1Brown’s video on Vectors
From https://towardsdatascience.com/perceptron-learning-algorithm-d5db0deab975
¹Maths is really abstract and meaningless unless you apply it to a context- this is a reason why you will get tripped if you try to get just a mathematical intuition about the neural network
The easiest way to understand it is in a geometric context, say 2D or 3D cartesian coordinates, and then extrapolate it. This is what we will try to do here.
Q. What is the connection between Matrices and Vectors?
Vectors are represented as matrices. Example here is a Euclidean Vector in three-dimensional Euclidean space (or $R^{3}$), represented as a column vector (usually) or row vector
$$
a = \begin{bmatrix}
a_{1}\\a_{2}\\a_{3}\
\end{bmatrix} = \begin{bmatrix} a_{1} & a_{2} &a_{3}\end{bmatrix}
$$
Q. What is a Dot product? and what does it signify ?
First the dry definitions.
Algebraically, the dot product is the sum of the products of the corresponding entries of the two sequences of numbers.
if $ \vec a = \left\langle {{a_1},{a_2},{a_3}} \right\rangle $ and $\vec b = \left\langle {{b_1},{b_2},{b_3}} \right\rangle $, Then
$$
\begin{equation}\vec a\centerdot \vec b = {a_1}{b_1} + {a_2}{b_2} + {a_3}{b_3}\label{eq:eq1}\end{equation}
$$
Geometrically, it is the product of the Euclidean magnitudes of the two vectors and the cosine of the angle between them
$$
\begin{equation}\vec a\centerdot \vec b = \left\| {\vec a} \right\|\,\,\left\| {\vec b} \right\|\cos \theta \label{eq:eq2} \end{equation}
$$
These definitions are equivalent when using Cartesian coordinates.
Here is a simple proof that follows from trigonometry -
http://tutorial.math.lamar.edu/Classes/CalcII/DotProduct.aspx
(You may need this too -https://sergedesmedt.github.io/MathOfNeuralNetworks/VectorMath.html#learn_vector_math_diff)
Now to the meat of the answer, the intuition part.
Till above all this is plain maths and correct. From this point on, it is from my understanding;
If two vectors are in the same direction the dot product is positive and if they are in the opposite direction the dot product is negative.
Test it here -
https://sergedesmedt.github.io/MathOfNeuralNetworks/DotProduct.html#learn_dotproduct
So you could use the dot product as a way to find out if two vectors are aligned or not. That is for any two distinct sets of input feature vectors in a vector space ( say we are classifying if leaf is healthy or not based on certain features of the leaf), we can have a weight vector, whose dot product with one input feature vector of the set of input vectors of a certain class (say leaf is healthy) is positive and with the other set is negative. In essence, we are using the weight vectors to split the hyper-plane into two distinctive sets.
The initial neural network - the Rosenblatt's perceptron was doing this and could only do this - that is finding a solution if and only if the input set was linearly separable. (that constraint led to an AI winter and frosted the hopes/hype generated by the Perceptron when it was proved that it could not solve for XNOR not linearly separable)
Here is how the Rosenblatt's perceptron is modeled
Image source https://maelfabien.github.io/deeplearning/Perceptron/#the-classic-model
Inputs are $x_1$ to $x_n$ , weights are some values that are learned $w_1$ to $w_n$. There is also a bias (b) which in above is -$\theta$
If we take the bias term out, the equation is
$$
\begin{equation}
f(x) =
\begin{cases}
1, & \text{if}\ \textbf{w}\cdot\textbf{x}+b ≥ 0 \\
0, & \text{otherwise} \\
\end{cases}
\end{equation}
$$
If we take a dummy input x0 as 1, then we can add the bias as a weight $w_0$ and then this bias can also fit cleanly to the sigma rule
$ y = 1 \textbf{if } \sum_i w_i x_i ≥ 0 \text{ Else } y=0 $
This is the dot product of weight and input vector w.x
Note that dot product of two matrices (representing vectors), can be written as that transpose of one multiplied by another
$$
\begin{equation}
\sigma(w^Tx + b)=
\begin{cases}
1, & \text{if}\ w^Tx + b ≥ 0 \\
0, & \text{otherwise} \\
\end{cases}
\end{equation}
$$
Basically, all three equations are the same.
Taking from https://sergedesmedt.github.io/MathOfNeuralNetworks/RosenblattPerceptronArticle.html
So, the equation $ \bf w⋅x>b $ defines all the points on one side of
the hyperplane, and $ \bf w⋅x<=b$ all the points on the other side of
the hyperplane and on the hyperplane itself. This happens to be the
very definition of “linear separability” Thus, the perceptron allows
us to separate our feature space in two convex half-spaces
Please also see the above article. It explains also how the weights are trained. So you can see how integral dot product and the representation of inputs and weights as vectors are in neural networks. This concept comes into play in modern neural networks as well, where gradient descent is basically using the
