I'm reading Why Momentum Really Works, a post from the new distill journal. I'll paraphrase the main equations leading to the part which confuses me, the post describes the intuition in more detail.
The gradient descent algorithm is given by the following iterative process $$w^{k+1} = w^k-\alpha \nabla f(w^k)$$ where $w^k$ is the value of iteration $k$, the learning rate is $\alpha$ and $\nabla f(w)$ is the gradient of the function $f$ evaluated at $w$. The function $f$ you wish to minimize.
Gradient descent with momentum is given by adding "memory" to the descent, this is described by the pair of equations:
\begin{align} z^{k+1} &= \beta z^k + \nabla f(w^k) \\ w^{k+1} &= w^k - \alpha z^{k+1} \end{align}
In the next section "First Steps: Gradient Descent" the author considers a convex quadratic function $$f(w) = \frac12w^TAw-b^Tw, \quad w \in \mathbb{R}^n, A \in \mathbb{R}^{n,n}$$ which has gradient $$\nabla f(w) = Aw-b$$ If we assume $A$ is symmetric and invertable then $f$ has optimal solution $w^\star = A^{-1}b$.
If we were to use gradient descent then we would iterate towards this optimal solution in the following way \begin{align} w^{k+1} &= w^k - \alpha \nabla f(w) \\ &= w^k - \alpha (Aw^k -b) \end{align}
Then the article goes on to say "There is a very natural space to view gradient descent where all the dimensions act independently — the eigenvectors of $A$". I think this makes sense, although my intuition is kind of fuzzy.
Every symmetric matrix $A$ has an eigenvalue decomposition where $$A = Q\text{diag}(\lambda_1,\ldots,\lambda_n)Q^T.$$
Where $\lambda_1 > \ldots > \lambda_n$ and $Q$ is the vector with the corresponding eigenvectors as columns (right?).
This next part is where I don't understand what is going on:
If we perform a change of basis, $x^k = Q^T(w^k - w^\star)$, the iterations break apart, becoming:
\begin{align} x_i^{k+1} &= x_i^k - \alpha \lambda_i x_i^k \\ &=(1-\alpha\lambda_i)x_i^k &= (1- \alpha\lambda_i)^{k+1}x_i^0 \end{align}
Moving back to our original space $w$, we can see that
$$w^k - w^\star = Qx^k = \sum\limits_{i}^n = x_i^0(1-\alpha\lambda_i)^kq_i$$
What is going on here? Where is the motivation of taking $w^k - w^\star$ into the eigendomain? What is $x^k$? Why are we now looking at invidual elements of the vector? I tried to follow the caculations through, but $x^{k+1}$ depends on $w^{k+1}$ which depends on $z^k$, which I thought we were trying to eliminate. My question is can someone expand on these few steps with some intuition and calculations? Thanks.