# Hoes does laplace smoothing in Naive Bayes control high bias and high variance?

I'm trying to understand how laplace smoothing exactly helps to balance between overfitting and underfitting.

I know that Laplace smoothing is used as a fail safe probability if there's a any unknown words but I couldn't exactly figure how exactly it helps in controlling bias and variance tradeoff.

• It might be helpful to think about this in the context of Bayesian analysis of proportion data. A thorough discussion, including its connection to Laplacian smoothing, can be found in Gleman et al., Bayesian Data Analysis 3rd edition. – Sycorax says Reinstate Monica Jul 16 '19 at 20:18

Laplace smoothing is a way to move probabilities towards uninformed mean. Suppose you have a multinomial variable with sample counts $$c_1, c_2,..,c_d$$, where $$d$$ is the number of dimensions. A Laplace smoothed version of estimated probabilities has the form: $$(c_i + \alpha)/(N + d\alpha)$$, where $$\alpha$$ is positive. If $$\alpha$$ is $$0$$ then we have non smoothed estimator. Often $$1$$ is used to solve the problem of missing categories. But values greater then one can be used. If large values are used than the influence of observed counts will be lower because estimated probabilities for the same number of observations will be lower. A direct consequence is that the variance of the model (estimates) is lower, possibly with larger bias. You can see the value of $$\alpha$$ as the strength of a Bayesian uninformed prior.
Suppose you $$d=2$$, and you have a training dataset with a single example, of the first category. Using Laplace smoothing with $$\alpha=1$$ we would estimate the probabilities as $$0.66, 0.33$$ If use $$\alpha=10$$ we get $$0.52,0.48$$, much closer to equal probabilities $$0.5,0.5$$