Each book I read propose a different fitting method for Logistic Regression.

The general idea is to maximize this expression.

$$ Pr\left(\beta|y,X,M\right) = \frac{Pr\left(y|\beta,X,M\right) Pr(\beta,M)}{Pr(y|X,M)} = \frac{Pr\left(y|\beta,X,M\right) Pr(\beta,M)}{\int Pr\left(y|\beta,X\right) Pr(\beta) d\beta} $$

The book A first course in Machine Learning says the following.

Mathematically the integral at the denominator has no close solution, unless an ad-hoc prior distribution is chosen. When we cannot directly compute the posterior density (due to the denominator), we have three options:

  • we can maximize the numerator, i.e., maximizing the likelihood times the prior. This is called Maximum A-posteriori estimation. This uses the Newton-Raphson mathematical method to estimate a single point corresponding to the maximum. However this is not a "Bayesian" method because we have a single point but we do not know how confident we are in this model.

  • The second method is the Laplace approximation. The idea is to approximate the density of interest with a Gaussian. The log-likelihood is approximated using Taylor expansion.

  • The third method is sampling from data. A popular method is Metropolis-Hastings.

The Elements of Statistical Learning propose a method similar to the first one except that instead of maximizing all the numerator it is maximizing the likelihood. The result however is similar to the first method (isn't it?).

Finally, the book Hands On Machine Learning define use a classical cost function

$$J(\theta) = -\frac{1}{m} \sum_{i=1}^m \left[ y^{(i)}\log(\hat{p}^{(i)}) +(1-y^{(i)})\log(1-\hat{p}^{(i)})\right]$$

and then uses the one of the gradient descent techniques (batch, stochastic, mini).

I can see similarities in all the methods that rely on the Bayes theorem but I cannot see any relationship between anyone of these methods and the Gradient descent technique: in one case you are maximizing a conditional probability while in the second you are minimizing a cost-function. Is there any similarity?

Is the result obtained using Gradient Descent the same as the one obtained by maximizing the numerator or the likelihood ? If not what are the differences?

Which estimation methods do machine learning libraries such as sklearn use?

Can someone please clarify?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.