12
$\begingroup$

I'm learning logistic regression and got confused when I saw the equation of the textbook. I knew that for a continuous distribution, to calculate the probability, the pdf $f(x)$ is meaningless. Instead the cumulative density function $F(x)$ shall be used. Thus, since we're maximizing the probability, shouldn't we use the product of cdfs rather than pdfs on the right side of the MLE equation? Thank you!

enter image description here

UPDATE, and further questions:

This question brings up an interesting point about why we don't often use the fact that $Y=F(X)\sim U(0,1)$ and then try to minimise the KL divergence between $Y$ and $U$:

$$\text{KL}(Y, U) = \int_0^1 f_y(y) \ln f_y(y) \text{d}y$$

Typically we have easy access to the form of $f$ (the original pdf) but $F$ might be less tractable and $f_Y$ is basically something we would need to estimate using empirical CDFs based on the samples $F(X_i), i=...$. The question is, are the two formulations (the usual MLE and the KL version above) very different in their results?

$\endgroup$
3
  • 3
    $\begingroup$ Several misconceptions here: (1) the pdf is far from meaningless; (2) $F$ is a cumulative distribution, not a cumulative density; and (3) we're not maximizing a probability, but only a likelihood. The likelihood ignores differential elements ($dx$). $\endgroup$
    – whuber
    Feb 8, 2018 at 23:40
  • $\begingroup$ Thank you @whuber, I did make a confusion of probability and likelihood. $\endgroup$
    – Yujian
    Feb 9, 2018 at 1:13
  • $\begingroup$ With censored data cdf's are definitely used in maximum likelihood estimation: stats.stackexchange.com/questions/49443. $\endgroup$
    – JimB
    Apr 6, 2019 at 21:50

2 Answers 2

3
$\begingroup$

How can a CDF be used to rank two possible parametrizations for a model? It is a cumulative probability, so it can only tell us the probability of obtaining such a result or a lower value given a probability model. If we took $\theta$ to predict the smallest possible outcomes, the CDF is nearly 1 at every observation and this would be the most "likely" in the sense that "yup, if the mean height were truly -99 I am very confident that repeating my sample would produce values smaller than the ones I observed".

We could balance the left cumulative probability with the right cumulative probability. Consider the converse in our calculation: a median unbiased estimator satisfies:

$$P(X < \theta) = P(X > \theta)$$

Here the best value of $\theta$ is the one for which $X$ is equally likely to be greater or less than it's predicted value (assuming $\theta$ is a mean here). But that certainly doesn't correspond with our idea of being able to rank alternate parameterizations as more likely for a particular sample.

Perhaps, on the other hand you wanted to be sure $X$ was very probable in a small interval of the value, that is maximize this probability:

$$P(\theta - d < X < \theta + d)/d = \left(F(X+d) - F(X-d)\right)/d$$

But how big should $d$ be? Well if $d$ is taken to be arbitrarily small:

$$\lim_{d \rightarrow 0} \left(F(X+d) - F(X-d)\right)/d = f(X)$$

And you get the density. It is the instantaneous probability function that best characterizes the likelihood of a specific observation under a parametrization.

$\endgroup$
8
  • $\begingroup$ Thank you @AdamO. Could you please explain a little more why "If we took θ to predict the smallest possible outcomes, the CDF is nearly 1 at every observation"? $\endgroup$
    – Yujian
    Feb 9, 2018 at 1:06
  • $\begingroup$ @Yujian a random sample of weight of kittens: if I set the "true mean" as 0 with an SD of 0.01, but their weights are 10, 13, and 15 oz. P(X<10) $\approx$ 1, P(X<13) $\approx$ 1, P(X<15) $\approx$ 1. These are very high CDF values, and thus are close to the "most likely" if we replaced PDF with CDF, but the true mean of 0 is nonsense; it makes no biologic sense and has no relevance to the sample. $\endgroup$
    – AdamO
    Feb 9, 2018 at 3:44
  • $\begingroup$ I think people are missing something interesting here: if $X$ has a CDF $F$, $Y=F(X)\sim U(0,1)$. So shouldn't one be able to minimal $KL(P_Y, P_{uniform})$ which is basically just the entropy of the random variable $Y=F(X)$. Maybe this is equivalent to the usual max-likelihood? $\endgroup$
    – mathtick
    Apr 6, 2019 at 17:30
  • $\begingroup$ @mathtick What do you mean by "$P_Y$"? If it's the empirical distribution, then unfortunately the KL distance to the Uniform distribution is always infinite. If it's not the empirical distribution, then how is it related to the data? $\endgroup$
    – whuber
    Apr 6, 2019 at 21:26
  • $\begingroup$ @whuber $P_Y$ is the probability distribution for $Y=F(X)$. If $F$ is the the distribution function that generated the sample $X_i$ then $F(X_i)$ will not be uniformly distributed. $\endgroup$
    – mathtick
    Apr 6, 2019 at 21:36
0
$\begingroup$

You have an empirical dataset and want to find the best fitting parameters of a hypothetical distribution. Say your empirical is Gaussian with mean 50, sd 10.

Let's let the algorithm make a guess... mean 0, sd 1. Your real points will be far in the tail of this guess, but we can summarize it by multiplying all the probabilities of your values based on an assumption of mean 0, sd 1. Actually, instead of multiplying, let's sum the log, since that's more manageable. Also, since our algorithm likes to minimize, not maximize, we'll flip the sign, so you end up with -logLiklihood.

Turns out that when you make a good guess for mean and sd, -LogLiklihood will be smaller than a bad guess. Rinse and repeat until the change in -logLiklihood is small enough, and there's your fit.

The CDF doesn't natively lend itself to this kind of objective function. Multiplying out the product of the PDF (or summing the log) tells you quite literally, the likelihood of your data under the hypothesis of a particular parameter set.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.