I am still learning, but let me present my current state of understanding. I was much helped by Christoph Molner's E-Mail Course On Conformal Prediction. I focus here entirely on the goals of the CP outcomes and completely skip how various CP algorithms might meet these goals. The intuition of the "how" is very important, but that varies with different algorithms, so there are many variations. I hope that other answers might fill in with that. For now, I find it much easier to understand the "how" if I first intuitively understand the "why", as I will attempt here.
The core idea is that instead of giving a single prediction for each row of data, a range of predictions is offered with a qualified guarantee that the true value will be within a specified range or set of options. For example, a 90% requirement guarantees that 90% of the true values will fall within the range or set given by the conformal prediction. The analyst can specify any required guarantee they want, such as 95% or 80% depending on their real-world application requirements. The
Let me give the basic idea of three kinds of conformal prediction: regression, binary classification, and multiclass classification.
Regression. Conformal prediction for regression is similar to prediction intervals for point predictions of the target variable (not to be confused with confidence intervals for coefficient estimates). However, prediction intervals have at least two limitations:
- They assume a given distribution of the residuals, e.g., a normal distribution. In contrast, conformal prediction has no assumption about the distribution. It works for any model that produces predictions and works for any distribution of errors.
- Even if distributional assumptions are met, there is no guarantee that new data will actually fall within the expected range. For example, a 90% prediction interval might result in 86% of actual values falling into the specified interval, or it might result in 93% in that interval. The 90% is just an approximation. In contrast, conformal prediction guarantees that for a 90% requirement, at least 90% of the data will fall within the interval that it specifies.
Binary classification. Conformal prediction for binary classification is best thought of as giving well-calibrated probability scores such that when all cases whose probabilities are above the requirement are combined, they are collectively guaranteed to meet the requirement. For example, if the requirement is 90%, then when considering new data, when all positive cases whose probabilities are > 90% are combined, then the conformal prediction guarantees that at least 90% of such cases are indeed positive.
Crucially, if the CP cannot guarantee either class (positive or negative) to be 90%, then it will predict both [positive, negative], which in CP terms means that the result is guaranteed to be one of the two. Since it would be too easy to achieve the guarantee by simply always predicting [positive, negative], two CP predictions that both meet the guarantee are compared by preferring the prediction that gives more single results (that is, either [positive] or [negative]) and fewer "any-of-the-above" results (that is, fewer [positive, negative]).
Multiclass classification. Conformal prediction for multiclass classification is best thought of as giving well-calibrated probability scores. However, rather than giving a single prediction of the most likely class, the result is a set of classes guaranteed to meet the requirement. For example, suppose that the requirement is that 90% of predictions will be correct and there are three classes A, B, and C. When considering new data, if the conformal prediction predicts any of the three classes (e.g., A) with certainty >= 90%, then its result will be that class with high certainty (i.e., [A]). But if none of its probabilities reaches the requirement, then the class probabilities will be cumulatively added and listed in order until the requirement is met. So, for example, if the calibrated probability for A is 70% and that for B is 23% (meaning that C is 7%), then the result will be [A, B], meaning that the true value is guaranteed to be either A or B, with 90% certainty. If no two probabilities meet the requirement (e.g., A = 70%, B = 18%, C = 12%), then the result will be all the options (i.e., [A, B, C]).
Again, as with CP for binary classification, to avoid the trivial case of achieving a guarantee by always predicting [A, B, C], the best CP algorithm is the one that has the lowest average set size. Understanding that all CP algorithms meet the required guarantee (e.g., 90% guaranteed), then a perfect CP algorithm would have an average size of 1 (that is, only one of the three classes is ever predicted, i.e. [A], [B], or [C]) and a useless CP algorithm would have an average size of 3 (that is, it always predicts [A, B, C], not necessarily in that order). Real-world CP algorithms will always have a value between 1 and 3, meaning that they very often give results with two values to meet the 90% requirement (e.g., [C, A] or [B, C]).
(Again, note that 90% is just an example; the analyst can specify any required guarantee they want, such as 95% or 80% depending on their real-world application requirements.)
Adaptive versus non-adaptive conformal prediction
There is at least one very important conceptual variation to note. By default, CP guarantees the requirement on average over all predicted data points. For example, if there are 100 lines of real data, suppose the predicted values could be split into a range of 10 data points each (you can think of this as 10 bins of a histogram). It is possible that for some ranges or bins, the guarantee is not met, but CP guarantees that it is met on average over all 10 bins. For example, maybe a 90% probability for the bins at the extreme ends only have 80% of positive cases (for binary classification) or have 80% of values that fall in the conformal interval (for regression), but for the middle bins, 95% of the values meet the guarantee, so the overall guarantee is indeed met when everything is averaged out. This default situation is called non-adaptive CP.
However, if we want every range of the data (that is, each of the 10 bins individually across the full range of predicted values) to meet the CP guarantee, then we need a CP algorithm that implements adaptive CP. In that case, not only do all the output data on average meet the CP guarantee, but every one of the 10 bins will individually also meet the CP guarantee. The point is that if we want to assure that the CP results adapt to different ranges of the data, then we must be sure that our chosen CP algorithm has this adaptive guarantee feature.
This is the core idea as I understand it. There are other forms (such as for time series analysis with sequentially ordered data), but that is beyond the scope of my intuitive primer.
