What does it mean for the Bayes Classifier to be optimal? The Bayes classifier is always called the 'optimal' classifier. What does this actually mean? In particular does optimal mean the Bayes classifier will never make a mistake when predicting the label of some data. And if the Bayes classifier perfectly predicts all labels, does this mean it has perfectly estimated the probability density distribution of the data?
 A: For a classification problem on $\mathcal X\times\mathcal Y$ with feature space $\mathcal X = \mathbb R^d$ and $K$ classes $\mathcal Y = \{1,\ldots,K\}$. The Bayes classifier $\eta : \mathcal X \to \mathcal Y$ is defined as
$$\eta(x) :=\arg\max_{k\in\mathcal Y}\ \mathbb P(Y=k\ |\ X=x) $$
With this definition, if we define the risk of a classifier (i.e. any function $f : \mathcal X \to \mathcal Y$) as the quantity
$$\mathcal R(f):=\mathbb P(f(X)\ne Y) $$
which simply represents the probability that the classifier $f$ makes a mistake, it is straightforward to prove that $\eta$ minimizes $\mathcal R$ (see the proof on the wiki article). Which means that for any classifier $f$, you always have
$$\mathcal R(f) \ge \mathcal R(\eta) $$
So as you can see from this definition of optimality, the Bayes classifier is only optimal in the sense that it performs better than any other classifier. In particular, there are no guarantees that the Bayes classifier achieves zero error (in fact, it is rarely the case).
To see how the Bayes classifier might misclassify, some examples, look at this picture (taken from Prof. Xiaohui Xie's lecture slides) :

In this picture, the prediction variable $X$ lies on the real axis, and you have two classes ($0$ and $1$). For convenience, we let $\eta(x):=\mathbb P(Y=1\ |\ X=x)$ in this case. As you can see, you have a decision boundary $\delta$ for which $\eta(x)\ge 1 - \eta(x)$ for all $x\ge\delta$ and vice-versa.
That means that for any example $X$ that lands on the right side of the decision boundary, the Bayes classifier will assign to it the class $1$. However, as you can see from the graph, examples from class $0$ can still land on the right side on the decision boundary, in which case they will be wrongly classified.
For the Bayes classifier to have perfect accuracy, we need the different classes to be "well separated" and far from the decision boundary, which is not often the case. This can be precisely formulated using the concept of "margin condition", as introduced by Audibert and Tsybakov in this paper : Fast learning rates for plug-in classifiers under the margin condition
(2005).

If the Bayes classifier perfectly predicts all labels, does this mean it has perfectly estimated the probability density distribution of the data?

First, it's incorrect to say that the Bayes classifier "estimates" anything. After all, it is just a function of the random variable $(X,Y)$, it does not rely on any sample at all (and that's why it can't be computed in practice). Second, the Bayes classifier does not really estimate any probability distribution, it is more of a separating rule or boundary between the different classes (and the best possible one). If it makes no error, it simply means that the datapoints are well separated in the feature space.
