Does KNN have a loss function? I didn't find a definition of loss function on wiki in the context of machine learning.
this one is less formal though, it is clear enough.
At its core, a loss function is incredibly simple: it’s a method of evaluating how well your algorithm models your dataset. If your predictions are totally off, your loss function will output a higher number. If they’re pretty good, it’ll output a lower number. As you change pieces of your algorithm to try and improve your model, your loss function will tell you if you’re getting anywhere.
it seems that the error rate of KNN is not the function that could guide the model itself optimize, such as Gradient Descent.
so, Does KNN have a loss function?
 A: $k$-NN does not have a loss function that can be minimized during training. In fact, this algorithm is not trained at all. The only "training" that happens for $k$-NN, is memorising the data (creating a local copy), so that during prediction you can do a search and majority vote. Technically, no function is fitted to the data, and so, no optimization is done (it cannot be trained using gradient descent).
A: As an alternative to the accepted answer:
Every stats algorithm is explicitly or implicitly minimizing some objective, even if there are no parameters or hyperparameters, and even if the minimization is not done iteratively. The kNN is so simple that one does not typically think of it like this, but you can actually write down an explicit objective function:
$$ \hat{t} = \text{argmax}_\mathcal{C} \sum_{i: x_i \in N_k(\{x\}, \hat{x})} \delta(t_i, \mathcal{C}) $$
What this says it that the predicted class $\hat{t}$ for a point $\hat{x}$ is equal to the class $\mathcal{C}$ which maximizes the number of other points $x_i$ that are in the set of $k$ nearby points $N_k(\{x\}, \hat{x})$ that also have the same class, measured by $\delta(t_i, \mathcal{C})$ which is $1$ when $x_i$ is in class $\mathcal{C}$, $0$ otherwise.
The advantage of writing it this way is that one can see how to make the objective "softer" by weighting points by proximity. Regarding "training," there are no parameters here to fit. But one could tune the distance metric (which is used to define $N_k$) or the weighting of points in this sum to optimize some additional classification objective. This leads into Neighborhood Component Analysis: https://www.cs.toronto.edu/~hinton/absps/nca.pdf which learns a distance metric.
