Skip to main content
added 1393 characters in body
Source Link
Tim
  • 141.2k
  • 26
  • 270
  • 512

There's no such a thing as estimating parameters "through training data". To get point estimates, you can train a probabilistic model by maximizing the likelihood function (MLE) alone, or maximizing the posterior probabilities (MAP). The above description says that it uses MAP, so it considers not only the data but also a prior. In naive Bayes, the common choice is to use Laplace smoothing (uniform prior) to prevent probabilities of zeroes for the unobserved cases, which would zero-out everything in the calculations.

You can just use maximum likelihood estimator, i.e. calculate probability as number of occurrences / sample size, but there are two potential problems:

  1. For every $(x_i, y)$ pair you would need to observe enough data to be able to calculate the $P(x_i|y)$ probability reliably. If $x_i$ is rare or is rare for the particular subgroup $y=c$, you would need to collect a lot of data to calculate it reliably. This is why we often use procedures as Laplace smoothing, or assume other priors so that we can correct the empirical estimates. With Laplace smoothing, we can make the empirical probabilities smoother (more uniform-ish) or more extreme. If you have reasonable prior information to peak an informative prior, you can use it to "correct" the information contained with data based on other information or prior results.

  2. If for some $(x_i, y)$ pair the number of occurrences of $x_i|y$ would be equal to zero, then the estimated probability $P(x_i|y)$ would be also equal to zero 0 / sample size = 0. In such case

    $$ P(y) \prod_{i=1}^n P(x_i | y) $$

    would also zero-out no matter what the other probabilities are. Laplace smoothing fixes this problem by forcing the probabilities to be non-zero.

Using the Bayesian approach (MAP) solves the two problems at the cost of making the estimates of the individual probabilities slightly biased.

There's no such a thing as estimating parameters "through training data". To get point estimates, you can train a probabilistic model by maximizing the likelihood function (MLE) alone, or maximizing the posterior probabilities (MAP). The above description says that it uses MAP, so it considers not only the data but also a prior. In naive Bayes, the common choice is to use Laplace smoothing (uniform prior) to prevent probabilities of zeroes for the unobserved cases, which would zero-out everything in the calculations.

There's no such a thing as estimating parameters "through training data". To get point estimates, you can train a probabilistic model by maximizing the likelihood function (MLE) alone, or maximizing the posterior probabilities (MAP). The above description says that it uses MAP, so it considers not only the data but also a prior. In naive Bayes, the common choice is to use Laplace smoothing (uniform prior) to prevent probabilities of zeroes for the unobserved cases, which would zero-out everything in the calculations.

You can just use maximum likelihood estimator, i.e. calculate probability as number of occurrences / sample size, but there are two potential problems:

  1. For every $(x_i, y)$ pair you would need to observe enough data to be able to calculate the $P(x_i|y)$ probability reliably. If $x_i$ is rare or is rare for the particular subgroup $y=c$, you would need to collect a lot of data to calculate it reliably. This is why we often use procedures as Laplace smoothing, or assume other priors so that we can correct the empirical estimates. With Laplace smoothing, we can make the empirical probabilities smoother (more uniform-ish) or more extreme. If you have reasonable prior information to peak an informative prior, you can use it to "correct" the information contained with data based on other information or prior results.

  2. If for some $(x_i, y)$ pair the number of occurrences of $x_i|y$ would be equal to zero, then the estimated probability $P(x_i|y)$ would be also equal to zero 0 / sample size = 0. In such case

    $$ P(y) \prod_{i=1}^n P(x_i | y) $$

    would also zero-out no matter what the other probabilities are. Laplace smoothing fixes this problem by forcing the probabilities to be non-zero.

Using the Bayesian approach (MAP) solves the two problems at the cost of making the estimates of the individual probabilities slightly biased.

Source Link
Tim
  • 141.2k
  • 26
  • 270
  • 512

There's no such a thing as estimating parameters "through training data". To get point estimates, you can train a probabilistic model by maximizing the likelihood function (MLE) alone, or maximizing the posterior probabilities (MAP). The above description says that it uses MAP, so it considers not only the data but also a prior. In naive Bayes, the common choice is to use Laplace smoothing (uniform prior) to prevent probabilities of zeroes for the unobserved cases, which would zero-out everything in the calculations.