Implementing batch gradient descent

Question

I'm trying to understand this notation:

Specifically, what does the summation compute at each iteration? Looks like for each training sample, I compute the error (difference between target and output for that sample) and multiple that by each term in that same sample. Is this a nested summation?

Example data:

// Samples
[
  [1, 2, 3],
  [4, 5, 6],
]

// Predicted output
[
  [23],
  [42],
]

// Target output
[
  [15],
  [16],
]

What does that summation expand into?

Answer 1

As far as I understand, you first compute the difference between target and output such as

$E_j = \sum_{i} target_{ji} - output_{ji}$

Here

$E_1 = 15 - 23 = -8$ and $E_2 = 16 - 42 = -26$

Now for each weight, you will take into account the input dimension associated with the weight thus:

$\Delta w_1 = \eta * ( (-8 * 1) + (-26 * 4))$

$\Delta w_2 = \eta * ( (-8 * 2) + (-26 * 5))$

$\Delta w_2 = \eta * ( (-8 * 3) + (-26 * 6))$

So the $x_{ij}$ in the formula is the data dimension $j$ for sample $i$.

We might prefer though to compute everything much more compact such as:

$\vec{\Delta w} = \eta * \vec{E} \times \vec{x}$, where $\vec{E}$ is an (1xn) array and $\vec{x}$ is the input data such as (nx3).