Intuitive explanation of conformal prediction

I have recently started learning about conformal prediction. I am a programmer without a strong mathematical background, but with a strong intuitive, applied background in statistics. I am trying to understand the intuition underlying conformal prediction and it is difficult to find good sources on this. Even the classic "A Gentle Introduction to Conformal Prediction and Distribution-Free Uncertainty Quantification" quickly gets too mathematical for me to intuitively follow.

Could someone please give an intuitive explanation of the key concepts underlying conformal prediction with minimal mathematical details?

• Are you interested in what it does or how it works (or both)? Also, see this excellent recent series. (The author now also has a book on the subject.) Commented Mar 9, 2023 at 12:27
• @EikeP., I am interested in both. Actually, I have just completed the series that you linked and it is because I am not fully satisfied that I am asking (and answering) this question. (I had to go through the email series twice to get a grasp on it.) As my answer below indicates, I am offering my answer to do a hopefully better job at answering the "what it does" question; I agree that that series is excellent for the intuition of "how it works". Commented Mar 9, 2023 at 15:48
• I think following these two tutorials from Swedish groups could be the'first contact' to conformal prediction. - Henric Linusen youtu.be/0MsGri8nmJQ - Ulf Johansson youtu.be/KsQpkjl7u1w Commented Jun 3, 2023 at 17:34

I am still learning, but let me present my current state of understanding. I was much helped by Christoph Molner's E-Mail Course On Conformal Prediction. I focus here entirely on the goals of the CP outcomes and completely skip how various CP algorithms might meet these goals. The intuition of the "how" is very important, but that varies with different algorithms, so there are many variations. I hope that other answers might fill in with that. For now, I find it much easier to understand the "how" if I first intuitively understand the "why", as I will attempt here.

The core idea is that instead of giving a single prediction for each row of data, a range of predictions is offered with a qualified guarantee that the true value will be within a specified range or set of options. For example, a 90% requirement guarantees that 90% of the true values will fall within the range or set given by the conformal prediction. The analyst can specify any required guarantee they want, such as 95% or 80% depending on their real-world application requirements. The

Let me give the basic idea of three kinds of conformal prediction: regression, binary classification, and multiclass classification.

Regression. Conformal prediction for regression is similar to prediction intervals for point predictions of the target variable (not to be confused with confidence intervals for coefficient estimates). However, prediction intervals have at least two limitations:

• They assume a given distribution of the residuals, e.g., a normal distribution. In contrast, conformal prediction has no assumption about the distribution. It works for any model that produces predictions and works for any distribution of errors.
• Even if distributional assumptions are met, there is no guarantee that new data will actually fall within the expected range. For example, a 90% prediction interval might result in 86% of actual values falling into the specified interval, or it might result in 93% in that interval. The 90% is just an approximation. In contrast, conformal prediction guarantees that for a 90% requirement, at least 90% of the data will fall within the interval that it specifies.

Binary classification. Conformal prediction for binary classification is best thought of as giving well-calibrated probability scores such that when all cases whose probabilities are above the requirement are combined, they are collectively guaranteed to meet the requirement. For example, if the requirement is 90%, then when considering new data, when all positive cases whose probabilities are > 90% are combined, then the conformal prediction guarantees that at least 90% of such cases are indeed positive.

Crucially, if the CP cannot guarantee either class (positive or negative) to be 90%, then it will predict both [positive, negative], which in CP terms means that the result is guaranteed to be one of the two. Since it would be too easy to achieve the guarantee by simply always predicting [positive, negative], two CP predictions that both meet the guarantee are compared by preferring the prediction that gives more single results (that is, either [positive] or [negative]) and fewer "any-of-the-above" results (that is, fewer [positive, negative]).

Multiclass classification. Conformal prediction for multiclass classification is best thought of as giving well-calibrated probability scores. However, rather than giving a single prediction of the most likely class, the result is a set of classes guaranteed to meet the requirement. For example, suppose that the requirement is that 90% of predictions will be correct and there are three classes A, B, and C. When considering new data, if the conformal prediction predicts any of the three classes (e.g., A) with certainty >= 90%, then its result will be that class with high certainty (i.e., [A]). But if none of its probabilities reaches the requirement, then the class probabilities will be cumulatively added and listed in order until the requirement is met. So, for example, if the calibrated probability for A is 70% and that for B is 23% (meaning that C is 7%), then the result will be [A, B], meaning that the true value is guaranteed to be either A or B, with 90% certainty. If no two probabilities meet the requirement (e.g., A = 70%, B = 18%, C = 12%), then the result will be all the options (i.e., [A, B, C]).

Again, as with CP for binary classification, to avoid the trivial case of achieving a guarantee by always predicting [A, B, C], the best CP algorithm is the one that has the lowest average set size. Understanding that all CP algorithms meet the required guarantee (e.g., 90% guaranteed), then a perfect CP algorithm would have an average size of 1 (that is, only one of the three classes is ever predicted, i.e. [A], [B], or [C]) and a useless CP algorithm would have an average size of 3 (that is, it always predicts [A, B, C], not necessarily in that order). Real-world CP algorithms will always have a value between 1 and 3, meaning that they very often give results with two values to meet the 90% requirement (e.g., [C, A] or [B, C]).

(Again, note that 90% is just an example; the analyst can specify any required guarantee they want, such as 95% or 80% depending on their real-world application requirements.)