I come to the solution of my own question. The answer is in the affirmative. I prove it in two ways.
Proof 1:
The mean $E[x]=\frac{a\alpha+1}{\alpha+2}\rightarrow a$ and the variance $\text{var}[x] = \frac{(a\alpha+1)(b\alpha+1)}{(\alpha+2)^2(\alpha+3)}\rightarrow 0$, as $\alpha\rightarrow 0$.
By Chebyshev's inequality, together with boundedness of $f(x,\alpha)$, it can be shown $f(x,\alpha)$ uniformly approaches $0$ for $|x-a|>\epsilon$, for arbitrary fixed positive $\epsilon$.
Proof 2:
Use Stirling's approximation to write the asymptotics of Beta function
$$B(u,v) \sim \sqrt{2\pi}\frac{u^{u-\frac{1}{2}}v^{v-\frac{1}{2}}}{(u+v)^{u+v-\frac{1}{2}}},$$
for large $u$ and $v$.
$$f(x,\alpha) \sim \sqrt{\frac{\alpha+2}{2\pi ab}}g(x)^\alpha$$
where
$$g(x) := \Big(\frac{x}{a}\Big)^a\Big(\frac{1-x}{b}\Big)^b.$$
By the strict concavity of the natural logarithmic function, so long as $x\ne a$,
$$\ln g(x)<\ln\Big(a\frac{x}{a}+b\frac{1-x}{b}\Big)=0.$$
I will fill in the detail and finish the derivation later.