# How to find the transform from an empiric distribution to another empiric distribution

I have two datasets, each dataset is a pool of samples, and each sample contains multiple observations. Data-wise, we can represent one sample as a vector of floats, and a dataset as a matrix where each row is a sample.

I would like to find a transform to project the first dataset into the "space" of the second dataset. I think I am looking for a way to estimate and then project an empirical distribution into another empirical distribution. There might be a dependency between samples of one dataset, but not between datasets.

Bonus reward if the answer also suggests a way to account for the number of samples (because there is a big dataset and the other one is much smaller, thus a distribution estimation will likely be much more approximate for the latter).

Clarification: I have (and can't get) no apriori on the underlying distribution of any of the datasets. They are generated by a natural phenomenon, and no theory model correctly these distributions yet.

/EDIT: I will try to clarify with a concrete example close to what I am working on.

Let's say I am working on a machine learning application to automatically recognize objects shapes in a scanned image. To make it simpler, let's say these images are only in grayscale (no color, only pixel intensity remains).

The problem I am facing is that I have images coming from several different scanners, and each scanner has a different "intensity space", so that it is difficult to transpose a machine learning model learnt on one scanner to the next scanner, because of the scanner-induced variability in intensity.

More formally, using pseudo-Python code, my data and code would look like this:

# Images are represented as a row vector, each value is one pixel intensity
# They are the samples
scanner1_im1 = [1, 255, 3, 73, ...]
scanner1_im2 = [4, 83, 2, 190, ...]
scanner2_im1 = [8, 29, 1, ...]
scanner2_im2 = [37, 29, 4, ...]

# We can constitute a pool of image samples for each scanner, which I call a dataset
scanner1_images = [scanner1_im1, scanner1_im2]
scanner2_images = [scanner2_im1, scanner2_im2]

# TODO: normalization step, where the scanner2_images gets normalized/projected into scanner1_images space, or whatever technique that would allow to represent both datasets in an approximately similar space
scanner2_images_norm = norm_project_images(scanner2_images, scanner1_images)

# Machine learning modeling on normalized images
ml.learn(scanner1_images, scanner2_images_norm)


What statistical/probabilistic techniques may I use to project all scanners/datasets into a similar space of intensities (and thus reduce inter-scanner variability while retaining inter-images/intra-scanner variability as much as possible)?

• The question is unclear: what do you mean by a pool of sample? Do you seek a distribution $F(x)$ such that $x$ is one sample? – Xi'an Feb 11 '18 at 18:27
• I tried to clarify with the second sentence, but with your notation, it would be F(X1, X2, ..., Xn) where Xi is a vector. – gaborous Feb 11 '18 at 18:34
• To clarify, because this might be a difficult edge case, X1, ..., Xn might all have some intrinsic variability, but I am interested in the whole variability, the parameters of the "parent" distribution that is shared across all Xs. If I am not clear enough, please tell me. – gaborous Feb 11 '18 at 18:43
• Also I expect the difference between Xi and Yi (2nd dataset) to be greater than any Xi vs Xj or Yi vs Yj. This is this (extrinsic) difference that I would like to quantify and reduce, even if approximately. – gaborous Feb 11 '18 at 19:06
• @gaborous: I hope you don't mind, but I posted a related question here, in an attempt to clarify some aspects of this question. If my question correctly interprets yours, you may feel free to steal any of it for an edit. – Ben Feb 28 '18 at 0:07

I believe that the field that you are interested in is called Calibration transfer or more generally Transfer learning. A lot of litterature can be found in the Spectroscopy field which I believe is quite close to image processing in terms of methods.

There are two different situations:

Either you have common images that have been scanned by the different scanners, in that case, I will just describe a bit the simpler and the most used methods. A review of all different methods can be found here [1]

• One of the easiest and oldest one: A method that has been developed by Wang et al [2]. The underlying principle is quite simple: a multi response and multi variate regression between the pixels response of the original instrument and the second.

• I believe the most used one now is the double window piecewise direct standardization (DWPS). Which is somehow similar to the simplest one but you using a size defined window to regress each pixel. [3]

If you don't have common images scanned by each scanner (all the images are different) there are two different approaches:

(1) The samples from the initial scanner can be reweighted in order to have the closest features distribution than the samples from the new scanner. A numerous of methods have been developed to find the appropriate weights [4,5,6,7,8]. Then you will have to re-train the model with the reweighted initial samples to predict on the new scanner.

(2) Then I think it's the approach you are most interested in: you can re-estimated a feature representation where samples form the old and new scanner are in closer proximity. Some methods have been developed to find the new representation space, you can find them here:[9,10,11]

The source 11 is a good review of all methods!

These sources in general are not specifically for image processing (and I changed the term 'wavelength' to pixel) but I belive the methods can be used for it. I don't think they explicitely take into account the size of each dataset. Hope this helps anyway!

References

[1] Feudale, R. N., Woody, N. A., Tan, H., Myles, A. J., Brown, S. D., & Ferré, J. (2002). Transfer of multivariate calibration models: a review. Chemometrics and Intelligent Laboratory Systems, 64(2), 181-192.

[2] Y.D. Wang, D.J. Veltkamp, B.R. Kowalski, Anal. Chem. 63 (1991) 2750–2756.

[3] Greensill, C. V., Wolfs, P. J., Spiegelman, C. H., & Walsh, K. B. (2001). Calibration transfer between PDA-based NIR spectrometers in the NIR assessment of melon soluble solids content. Applied spectroscopy, 55(5), 647-653.

[4] Huang, J., Smola, A. J., Gretton, A., Borgwardt, K. M., and Scholkopf, B.: Correcting Sample Selection Bias by Unlabeled Data, in: Pro- 5 ceedings of the 19th International Conference on Neural Information Processing Systems, NIPS’06, pp. 601–608, MIT Press, Cambridge, MA, USA, http://dl.acm.org/citation.cfm?id=2976456.2976532, 2006.

[5] Sugiyama, M., Nakajima, S., Kashima, H., Buenau, P. V., and Kawanabe, M.: Direct Importance Estimation with Model Selection and Its Application to Covariate Shift Adaptation, in: Advances in Neural Information Processing Systems 20, edited 63 by Platt, J. C., Koller, D., Singer, Y., and Roweis, S. T., pp. 1433–1440, Curran Associates, Inc., http://papers.nips.cc/paper/3248-direct-importance-estimation-with-model-selection-and-its-application-to-covariate-shift-adaptation.pdf, 2008.

[6]Kim, S., Kano, M., Nakagawa, H., and Hasebe, S.: Estimation of active pharmaceutical ingredients content using locally weighted partial least squares and statistical wavelength selection, International Journal of Pharmaceutics, 421, 269 – 274, https://doi.org/https://doi.org/10.1016/j.ijpharm.2011.10.007, 2011.

[7] Hazama, K. and Kano, M.: Covariance-based locally weighted partial least squares for high-performance adaptive modeling, Chemometrics 25 and Intelligent Laboratory Systems, 146, 55 – 62, https://doi.org/https://doi.org/10.1016/j.chemolab.2015.05.007, 2015.

[8]Zhang, X., Kano, M., and Li, Y.: Locally weighted kernel partial least squares regression based on sparse nonlinear fea30 tures for virtual sensing of nonlinear time-varying processes, Computers & Chemical Engineering, 104, 164 – 171, https://doi.org/https://doi.org/10.1016/j.compchemeng.2017.04.014, 2017.

[9]Culp, M. and Michailidis, G.: An Iterative Algorithm for Extending Learners to a Semi-Supervised Setting, Journal of Computational and 20 Graphical Statistics, 17, 545–571, https://doi.org/10.1198/106186008X344748, 2008.

[10] Gujral, P., Amrhein, M., Ergon, R., Wise, B. M., and Bonvin, D.: On multivariate calibration with unlabeled data, Journal of Chemometrics, 25, 456–465, https://doi.org/10.1002/cem.1389, 2011.

[11] an, S. J. and Yang, Q.: A Survey on Transfer Learning, IEEE Transactions on Knowledge and Data Engineering, 22, 1345–1359, https://doi.org/10.1109/TKDE.2009.191, 2010.

• Thank you very much, I will review all these references! I spotted spectroscopy methods before but I was not sure they were applicable to other domains, I will have a look! – gaborous Mar 1 '18 at 20:06