What is the relationship between gradient boosting and gradient decent? What is the relationship between gradient boosting and gradient decent? Are the closely related and how are they related to each other?
 A: Short answer:
In gradient boosting you need to minimize some loss, and gradient descent is an algorithm that allows you to minimize functions $L: R^n \to R$.
Long answer:
Notation for regression problems
In general regression problems, you are given some observations $(x_1,y_1),..,(x_n,y_n)$ and would like to predict a function $F$ such that $F(x_i) \approx y_i$.
Deciding what is the criteria for $`\approx`$ is usually done using a loss function $L$ (for example $L(x,y) = (x-y)^2$).
Your task is to minimize the total loss for a function $L_{tot}(F)$ depending on the data $(x_i),(y_i)$ and the choice of regressor $F$. So you would like to do
$$
\min_{F} L_{tot}(F) = \min_{F} \sum_{i=1}^N L(y_i, F(x_i))
$$
and find the function $F^*$ that realizes the minimium. $F$ is a function so it lives in an infinite dimensional space, which is impossible to solve numerically without making further assumptions.
Gradient boosting
The gradient boosting algorithm seeks a minimum $F^*$ of the type
$$
F^*(x) = \sum_{i=1}^M a_i h_i(x) + a_0 
$$
where $a_0,..,a_M$ are constants, and $h_1,..,h_M$ are some pre-determined basis functions. 
The way it is done is by starting with a constant function $F = a_0$ and proceed with incremental updates of the type $ a_i h_i$ at each step i, for $i=1,..,M$. Each update at step $i$ is a minimization procedure on the coefficient $a_i$ only, and hence is a one-dimensional problem.
Connection with gradient descent
The gradient descent algorithm allows you to find a (local) minimum for a minimization problem. In this case, we are going to apply it to solve the one-dimensional minimization problems. 
Assume that we have the function $F_{i-1}$ at step $i-1$, and we are seeking an update at step $i$ of the type
$$
F_{i} = F_{i-1} + a_i h_i
$$
such that the update lowers the total loss;
$$
L_{tot} (F_i) \leq L_{tot} (F_{i-1})
$$
The idea is to start at $F_{i-1}$, and locally follow the direction of the greatest decrease of the function $a \mapsto L_{tot}(F_{i-1} + a h_i)$. 
Since we are solving a one dimensional problem, let's simplify the notation: call $f(a) = L_{tot}(F_{i-1} + a h_i)$. We are starting at $a^0=0$, and are looking for a minimum of $f$. The direction of greatest (local) decrease is given by minus the derivative $-f'(a^0)$.
So $f(a^0 - \delta f'(a^0)) \leq f(a^0)$ for some $\delta$ small enough, and gives an new value $a^1 = a^0 - \delta f'(a^0)$. One can repeat the procedure starting at $a^1$ to get a better point $a^2$, and repeat again and again until you are confident that you are close to the minimum. This procedure is the gradient descent algorithm, and will yield at the end of the day some number $a_i$ for step $i$ such that $f(a_i) = L_{tot} (F_{i-1} + a_i h_i) \leq L_{tot} (F_{i-1} + a h_i)$ for all $a$.
Let me know if you want more details about the gradient boosting algorithm. 
