0
$\begingroup$

I'm trying to understand how laplace smoothing exactly helps to balance between overfitting and underfitting.

I know that Laplace smoothing is used as a fail safe probability if there's a any unknown words but I couldn't exactly figure how exactly it helps in controlling bias and variance tradeoff.

$\endgroup$
  • 1
    $\begingroup$ It might be helpful to think about this in the context of Bayesian analysis of proportion data. A thorough discussion, including its connection to Laplacian smoothing, can be found in Gleman et al., Bayesian Data Analysis 3rd edition. $\endgroup$ – Sycorax says Reinstate Monica Jul 16 '19 at 20:18
4
$\begingroup$

Laplace smoothing is a way to move probabilities towards uninformed mean. Suppose you have a multinomial variable with sample counts $c_1, c_2,..,c_d$, where $d$ is the number of dimensions. A Laplace smoothed version of estimated probabilities has the form: $(c_i + \alpha)/(N + d\alpha)$, where $\alpha$ is positive. If $\alpha$ is $0$ then we have non smoothed estimator. Often $1$ is used to solve the problem of missing categories. But values greater then one can be used. If large values are used than the influence of observed counts will be lower because estimated probabilities for the same number of observations will be lower. A direct consequence is that the variance of the model (estimates) is lower, possibly with larger bias. You can see the value of $\alpha$ as the strength of a Bayesian uninformed prior.

[address comment]

Suppose you $d=2$, and you have a training dataset with a single example, of the first category. Using Laplace smoothing with $\alpha=1$ we would estimate the probabilities as $0.66, 0.33$ If use $\alpha=10$ we get $0.52,0.48$, much closer to equal probabilities $0.5,0.5$

$\endgroup$
  • $\begingroup$ "If large values are used than the influence of observed counts will be lower because estimated probabilities for the same number of observations will be lower" I was not able to understand it. Please can you provide an example here so I can intuitively get it $\endgroup$ – user_6396 Jul 16 '19 at 19:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.