This might be a vague question, but I'm wondering how is Scikit-Learn's quantile transformation implemented?
I'm wondering how can a skewed dataset be transformed into a normal distribution like this?
Usually scikit-learn provides a link to the wiki, but not this transformation.
Can someone point me to the right direction?
Thanks
Yes, it appears to be described in a few different places, with no link to any papers.
The class documentation summarises the algorithm as follows:
The transformation is applied on each feature independently. The cumulative density function of a feature is used to project the original values. Features values of new/unseen data that fall below or above the fitted range will be mapped to the bounds of the output distribution. Note that this transform is non-linear. It may distort linear correlations between variables measured at the same scale but renders variables measured at different scales more directly comparable.
And the user guide adds some new information:
However, by performing a rank transformation, it smooths out unusual distributions and is less influenced by outliers than scaling methods. It does, however, distort correlations and distances within and across features.
Specifically, for a Normal transformation:
Thus the median of the input becomes the mean of the output, centered at 0. The normal output is clipped so that the input’s minimum and maximum — corresponding to the 1e-7 and 1 - 1e-7 quantiles respectively — do not become infinite under the transformation.
The GitHub pull request for this estimator references an older one that shows it was originally going to be named a "rank scaler".
In broader terms, this paper provides a good summary of the various ways that such "inverse normal transformations (INTs)" may be implemented:
INTs are ways of transforming the sample distribution of a continuous variable to make it appear more normally distributed. There are several types of INTs. The first distinction we make is between rank-based and non-rank-based INTs. Non-rank-based INTs entail assuming a particular cumulative distribution function (CDF) for the observed data, estimating the parameters of that distribution, converting observed scores to estimated quantiles from the CDF, and then converting these quantiles to standard normal deviates using the inverse normal (or probit function). Such non-rank-based INTs are usually referred to as copulas (Basrak et al. 2004; Li et al. 2006) and will not be considered further. It is worth noting, however, that the rank-based INTs can be expressed as a special case of the copula method in which the empirical CDF is used instead of restricting the CDF to some family of distributions. That is, every moment is in effect estimated from the data and the quantiles become simple functions of the ranks.
Rank-based INTs involve a preliminary step of converting a variable to ranks and can be further subdivided into two classes: those that involve a stochastic element and those that are deterministic. We are aware of only one INT that involves a stochastic element and this approach has been referred to as the use of “random normal deviates” (Conover 1980). One deterrent to this approach is that each investigator applying the same method to the same dataset will obtain a slightly different answer, which might be unsatisfying to some. This approach has the theoretical advantage of avoiding the granularity in the distribution of P values, an issue which often plagues many nonparametric tests. Nevertheless, the stochastic nature of these INTs seems to discourage researchers and they are rarely, if ever, used.
Deterministic rank-based INTs can be classified into those that use expected normal scores (Fisher and Yates 1938) versus those that use back transformation of sample quantile (or fractional rank) to approximate the expected normal scores. Using numerical integration, Harter (1961) has provided the most complete table of expected normal scores. INTs that involve back transformation of fractional ranks to approximate the expected normal scores of Fisher and Yates (Maritz 1982) appear to be the most commonly used in genetic research and will be the primary focus of attention. In back-transforming ranks, a fractional offset is needed to avoid having the minimum and maximum observations transformed to negative and positive infinity, respectively. Perhaps the most commonly used rank-based INT transformation entails creating a modified rank variable and then computing a new transformed value of the phenotype for the ith subject.
Looking at the QuantileTransformer code, it looks like the very last item in the list: a deterministic rank-based INT that calculates a modified rank variable.
However, it's a relatively simple implementation:
numpy.percentilenumpy.interpscipy.stats.norm.ppftaking care to deal with bounds at the extremities.
Representing as a very simplified mapping, i.e. ignoring the interpolation and bounding logic, it would just be $y_i = \Phi^{-1}(F(x_i))$, where $F$ and $\Phi$ represent the CDFs of an empirical and standard Normal distribution, respectively.
if a is qth qutantile, q in [0, 1]. Then after mapping, the value corresponding to a is norm.ppf(q). ppf is the inverse of cdf.
