Dot product vs Element-wise multiplication

Question

What is the different between the dot product "$\cdot$" and the element-wise multiplication notation $\odot$ in Statistics? I referred to Hamilton's Time-Series Analysis, and these seem to have the same definition. For instance, for two $n\times 1$ vectors $\vec{a}$ and $\vec{b}$, is \begin{equation} \vec{a}\odot \vec{b}\overset{?}{\equiv} \vec{a} \cdot \vec{b}=\sum\limits_{i=1}^n a_ib_i \end{equation}

Answer 1

The difference operationally is the aggregation by summation. With the dot product, you multiply the corresponding components and add those products together. With the Hadamard product (element-wise product) you multiply the corresponding components, but do not aggregate by summation, leaving a new vector with the same dimension as the original operand vectors. And on that point, the dot product of two vectors gives a scalar number while the Hadamard product of two vectors gives a vector.

$$c = \vec{x}^T \cdot \vec{y} = \begin{bmatrix} x_{1} \\ x_{2} \\ \vdots \\ x_{n} \end{bmatrix}^T \cdot \begin{bmatrix} y_{1} \\ y_{2} \\ \vdots \\ y_{n} \end{bmatrix} = [x_1, x_2, \cdots, x_n] \cdot \begin{bmatrix} y_{1} \\ y_{2} \\ \vdots \\ y_{n} \end{bmatrix} = \sum_{i=1}^n x_iy_i \in \mathbb{R}$$ $$\vec{z} = \vec{x} \odot \vec{y} = \begin{bmatrix} x_{1} \\ x_{2} \\ \vdots \\ x_{n} \end{bmatrix} \odot \begin{bmatrix} y_{1} \\ y_{2} \\ \vdots \\ y_{n} \end{bmatrix} = \begin{bmatrix} x_1y_{1} \\ x_2y_{2} \\ \vdots \\ x_ny_{n} \end{bmatrix} \in \mathbb{R}^n$$

Note that $\vec{x}, \vec{y}$ must have the same dimension. The Hadamard product is often defined in terms of matrices in many sources, but for general tensors it suffices that they have the same shape. For two vectors, this is tantamount to having the same dimension.

So in general $\vec{x} \cdot \vec{y} \neq \vec{x} \odot \vec{y}$.

Furthermore, $\|\vec{x} \odot \vec{y}\| \leq \|\vec{x}\|\|\vec{y}\|$ and $\|\vec{x} \cdot \vec{y}\| \leq \|\vec{x}\| \|\vec{y}\|$. But since $Pr\left( \|\vec{x} \cdot \vec{y}\| \leq \|\vec{x} \odot \vec{y}\| \right) \approx .67$ for some choice of probability space (see link on $\|\vec{x} \odot \vec{y}\| \leq \|\vec{x}\|\|\vec{y}\|$), there isn't a direct inequality between their norms.

Footnote: The dot product is also an example of an inner product while the Hadamard product is not. This is important for certain forms of rotational invariance, but that is not really a 'statistics' SE point.

Answer 2

Element-wise(Hadamard) multiplication(product) doesn't need summation after multiplication while dot multiplication(product) needs.

<Element-wise(Hadamard) multiplication (product)>

For example, element-wise multiplication doesn't need summation after multiplication so the result is [12, 21, 20] as shown below.

• Element-wise multiplication is the multiplication of 0D or more D tensors(arrays).

• The rule which you must follow to do element-wise multiplication is 2 tensors(arrays) must have the same number of rows and columns.

[a, b, c] x [d, e, f] = [ad, be, cf]
1 row       1 row 
3 columns   3 columns

[2, 7, 4] x [6, 3, 5] = [12, 21, 20]
                       [2x6, 7x3, 4x5]
 [2, 7, 4]
  x  x  x 
 [6, 3, 5]
     ||
[12, 21, 20]

In PyTorch with * or mul(). ** or mul() can multiply 0D or more D tensors(arrays) by element-wise multiplication:

import torch

tensor1 = torch.tensor([2, 7, 4])
tensor2 = torch.tensor([6, 3, 5])

tensor1 * tensor2 # tensor([12, 21, 20])
torch.mul(tensor1, tensor2) # tensor([12, 21, 20])

<Dot multiplication(product)>

For example, dot multiplication needs summation after multiplication so the result is 53 as shown below.

• Dot multiplication is the multiplication of 1D tensors(arrays).
• The rule which you must follow to do dot multiplication is the number of the rows of A and B tensors(arrays) is 1 and the number of the columns must be the same.

   <A>         <B>
[a, b, c] x [d, e, f] = ad+be+cf
1 row       1 row
3 columns   3 columns

[2, 7, 4] x [6, 3, 5] = 53
                 (2x6)+(7x3)+(4x5)
  [2, 7, 4]
   x  x  x
  [6, 3, 5]
      ||
 [12, 21, 20]
 12 + 21 + 20
      ||
      53

In PyTorch with @, dot() or matmul():

import torch

tensor1 = torch.tensor([2, 7, 4])
tensor2 = torch.tensor([6, 3, 5])

tensor1 @ tensor2 # tensor(53)
torch.dot(tensor1, tensor2) # tensor(53)
torch.matmul(tensor1, tensor2) # tensor(53)

*Memos:

• dot() can multiply 1D tensors(arrays) by dot multiplication.
• @ or matmul() can multiply 1D or more D tensors(arrays) by matrix multiplication.