I know that the PDF is the first derivative of the CDF for a continuous random variable, and the difference for a discrete random variable. However, I would like to know why this is, why are there two different cases for discrete and continuous?
- Anybody can ask a question
- Anybody can answer
- The best answers are voted up and rise to the top
I am gonna be a bit imprecise, but hopefully intuitive.
Discrete and continuous probability distributions must be treated differently. For any value in a discrete distribution there is a finite probability. With a fair coin, the probability of heads is 0.5, with a fair six sided die, the probability of a 1 is one sixth, etc. However, the probability of any specific value in a continuous distribution is zero, because one specific value is only one value out of an infinite number of possible values, and if specific values had a >0 probability, then they would not sum up to 1. Hence, with continuous distributions we talk about the probability of ranges of values.
"Sum up to" is key in answering your question. If you are at all familiar with calculus and its history, you understand that the integral sign—that elongated 'S': $\int$—is a special kind of summation: one describing the limiting case as we approach summing an infinite number of vanishingly small values between points $a$ and $b$ on some function. If that function is a PDF, we can integrate it (sum up) to produce a CDF, and conversely differentiate (difference) the CDF to obtain the PDF.
In the discrete case, we can simply perform standard arithmetic summation (hence, big '$\Sigma$', rather than the tall 'S' notation) and arithmetic differencing.
The difference is for the convenience and understanding of people who have not had to endure Ph.D. level theory courses where you derive and prove "Integral with respect to Counting Measure". Which shows there really is no difference between discrete and continuous distributions, that a sum is really an integral (and as @Alexis already mentioned, an integral is essentially a sum) and a difference is really a derivative (it is a little simpler to see that a derivative is a difference scaled appropriately).
Textbooks and courses will treat them different because it is simpler to teach/understand early on rather than requiring the math that shows there is not a difference.
(At least at introductory levels) the term density refers only to continuous random variables.
Discrete random variables have a probability mass function, sometimes called a probability function (pmf or pf, not pdf). This doesn't return density but actual probability.
Some random variables don't have either (but they still have a cdf).
Think about what the definition of a cdf is ($F_X(x) = P(X\leq x)$), and then what happens as $x$ moves a little bit in both cases.
Now consider that any jumps in the cdf imply that a particular value has non-zero probability (that $P(X\leq x)>P(X<x)$ and the difference is $P(X=x)$). That non-zero probability for particular $x$-values is what the pmf records $p_X(x)=P(X=x)$.
(In more advanced treatments, the distinction disappears.)
Actually, you can treat continuous and discrete distributions similarly, but in order to do this you have introduce Dirac's delta functions, left limits and other "advanced" concepts.
So, the easy way to answer your question is that discrete CDF jumps, it's discontinuous. You can't differentiate it everywhere because of that.
Again, if you know delta function, all is possible!