On his blog, Larry Wasserman has a post about what he planned to cover in his course last fall. He notes that he was abandoning some classical topics in favor of more modern issues. One topic that he mentions is Hoeffding’s inequality. What makes this result especially important for students and practitioners?
Hoeffding's inequality provides an upper bound on the probability that the sum of independent bounded random variables deviates from its expected value. This inequality provides a simple way to create a confidence interval for the binomial parameter $p$ corresponding to the probability of success. It can be used in the context of empirical risk minimization to estimate the error rate in classification.