I think you are confusing the true objective function with the learner's objective function. Often, in machine learning and data science, we approximate a function using a model by following the gradient of the loss function with respect to the parameters of the model.
For example, let us approximate a true objective function $t(x)$ using a model $f(x,\alpha)$, given a dataset of points $(x_i,y_i)$ (with $i \in \{0\ldots n\}$). We first define the loss function as the sum of the loss with respect to each training point:
$\qquad L(x) = \sum_{i=1}^n (t(x_i) - f(x_i,\alpha))^2 = \sum_{i=1}^n (y_i - f(x_i,\alpha))^2$
And then use an optimization procedure to minimize the loss. One way to solve this, is to use gradient descent, which would require you to follow the gradient with respect to the loss function in an online manner:
$\text{while not stop:}\\
\quad\text{pick a parameter index i}\\
\quad\alpha \leftarrow \alpha + \eta \cdot \nabla f(x_i,\alpha)
$
Now if we have a form of the model, we can calculate this in a straightforward way. For instance, if $f(x,\alpha)$ is a one-dimensional plane:
$
\qquad f(x,\alpha) = \alpha_1 \cdot x + \alpha_0\\
\qquad \nabla L(x,\alpha) = \nabla(y_i - f(x_i,\alpha))^2 = \nabla(y_i - \alpha_1 \cdot x + \alpha_0)^2 = \ldots
$
Plug in the second equation into the algorithm, and you're good to go. This is, of course, just one example. There exist many different formulations of loss functions and many different associated optimization procedures. Enumerating all of them would require an enumeration of the discipline and is simply too broad (although I can say many of the optimization procedure encoutered in machine learning are either linear or convex).
Note also, that we don't always optimize with respect to the loss immediatly. Often, other terms are introduced in the optimization problem such as regularization terms. Furthermore, in some cases we want to avoid working with the loss function directly alltogether, one famous example being support vector machines.
