How to find the transform from an empiric distribution to another empiric distribution

Question

I have two datasets, each dataset is a pool of samples, and each sample contains multiple observations. Data-wise, we can represent one sample as a vector of floats, and a dataset as a matrix where each row is a sample.

I would like to find a transform to project the first dataset into the "space" of the second dataset. I think I am looking for a way to estimate and then project an empirical distribution into another empirical distribution. There might be a dependency between samples of one dataset, but not between datasets.

Bonus reward if the answer also suggests a way to account for the number of samples (because there is a big dataset and the other one is much smaller, thus a distribution estimation will likely be much more approximate for the latter).

Clarification: I have (and can't get) no apriori on the underlying distribution of any of the datasets. They are generated by a natural phenomenon, and no theory model correctly these distributions yet.

/EDIT: I will try to clarify with a concrete example close to what I am working on.

Let's say I am working on a machine learning application to automatically recognize objects shapes in a scanned image. To make it simpler, let's say these images are only in grayscale (no color, only pixel intensity remains).

The problem I am facing is that I have images coming from several different scanners, and each scanner has a different "intensity space", so that it is difficult to transpose a machine learning model learnt on one scanner to the next scanner, because of the scanner-induced variability in intensity.

More formally, using pseudo-Python code, my data and code would look like this:

# Images are represented as a row vector, each value is one pixel intensity
# They are the samples
scanner1_im1 = [1, 255, 3, 73, ...]
scanner1_im2 = [4, 83, 2, 190, ...]
scanner2_im1 = [8, 29, 1, ...]
scanner2_im2 = [37, 29, 4, ...]

# We can constitute a pool of image samples for each scanner, which I call a dataset
scanner1_images = [scanner1_im1, scanner1_im2]
scanner2_images = [scanner2_im1, scanner2_im2]

# TODO: normalization step, where the scanner2_images gets normalized/projected into scanner1_images space, or whatever technique that would allow to represent both datasets in an approximately similar space
scanner2_images_norm = norm_project_images(scanner2_images, scanner1_images)

# Machine learning modeling on normalized images
ml.learn(scanner1_images, scanner2_images_norm)

What statistical/probabilistic techniques may I use to project all scanners/datasets into a similar space of intensities (and thus reduce inter-scanner variability while retaining inter-images/intra-scanner variability as much as possible)?

Answer 1

I believe that the field that you are interested in is called Calibration transfer or more generally Transfer learning. A lot of litterature can be found in the Spectroscopy field which I believe is quite close to image processing in terms of methods.

There are two different situations:

Either you have common images that have been scanned by the different scanners, in that case, I will just describe a bit the simpler and the most used methods. A review of all different methods can be found here [1]

• One of the easiest and oldest one: A method that has been developed by Wang et al [2]. The underlying principle is quite simple: a multi response and multi variate regression between the pixels response of the original instrument and the second.

• I believe the most used one now is the double window piecewise direct standardization (DWPS). Which is somehow similar to the simplest one but you using a size defined window to regress each pixel. [3]

If you don't have common images scanned by each scanner (all the images are different) there are two different approaches:

(1) The samples from the initial scanner can be reweighted in order to have the closest features distribution than the samples from the new scanner. A numerous of methods have been developed to find the appropriate weights [4,5,6,7,8]. Then you will have to re-train the model with the reweighted initial samples to predict on the new scanner.

(2) Then I think it's the approach you are most interested in: you can re-estimated a feature representation where samples form the old and new scanner are in closer proximity. Some methods have been developed to find the new representation space, you can find them here:[9,10,11]

The source 11 is a good review of all methods!

These sources in general are not specifically for image processing (and I changed the term 'wavelength' to pixel) but I belive the methods can be used for it. I don't think they explicitely take into account the size of each dataset. Hope this helps anyway!

References

[1] Feudale, R. N., Woody, N. A., Tan, H., Myles, A. J., Brown, S. D., & Ferré, J. (2002). Transfer of multivariate calibration models: a review. Chemometrics and Intelligent Laboratory Systems, 64(2), 181-192.

[2] Y.D. Wang, D.J. Veltkamp, B.R. Kowalski, Anal. Chem. 63 (1991) 2750–2756.

[3] Greensill, C. V., Wolfs, P. J., Spiegelman, C. H., & Walsh, K. B. (2001). Calibration transfer between PDA-based NIR spectrometers in the NIR assessment of melon soluble solids content. Applied spectroscopy, 55(5), 647-653.

[4] Huang, J., Smola, A. J., Gretton, A., Borgwardt, K. M., and Scholkopf, B.: Correcting Sample Selection Bias by Unlabeled Data, in: Pro- 5 ceedings of the 19th International Conference on Neural Information Processing Systems, NIPS’06, pp. 601–608, MIT Press, Cambridge, MA, USA, http://dl.acm.org/citation.cfm?id=2976456.2976532, 2006.

[5] Sugiyama, M., Nakajima, S., Kashima, H., Buenau, P. V., and Kawanabe, M.: Direct Importance Estimation with Model Selection and Its Application to Covariate Shift Adaptation, in: Advances in Neural Information Processing Systems 20, edited 63 by Platt, J. C., Koller, D., Singer, Y., and Roweis, S. T., pp. 1433–1440, Curran Associates, Inc., http://papers.nips.cc/paper/3248-direct-importance-estimation-with-model-selection-and-its-application-to-covariate-shift-adaptation.pdf, 2008.

[6]Kim, S., Kano, M., Nakagawa, H., and Hasebe, S.: Estimation of active pharmaceutical ingredients content using locally weighted partial least squares and statistical wavelength selection, International Journal of Pharmaceutics, 421, 269 – 274, https://doi.org/https://doi.org/10.1016/j.ijpharm.2011.10.007, 2011.

[7] Hazama, K. and Kano, M.: Covariance-based locally weighted partial least squares for high-performance adaptive modeling, Chemometrics 25 and Intelligent Laboratory Systems, 146, 55 – 62, https://doi.org/https://doi.org/10.1016/j.chemolab.2015.05.007, 2015.

[8]Zhang, X., Kano, M., and Li, Y.: Locally weighted kernel partial least squares regression based on sparse nonlinear fea30 tures for virtual sensing of nonlinear time-varying processes, Computers & Chemical Engineering, 104, 164 – 171, https://doi.org/https://doi.org/10.1016/j.compchemeng.2017.04.014, 2017.

[9]Culp, M. and Michailidis, G.: An Iterative Algorithm for Extending Learners to a Semi-Supervised Setting, Journal of Computational and 20 Graphical Statistics, 17, 545–571, https://doi.org/10.1198/106186008X344748, 2008.

[10] Gujral, P., Amrhein, M., Ergon, R., Wise, B. M., and Bonvin, D.: On multivariate calibration with unlabeled data, Journal of Chemometrics, 25, 456–465, https://doi.org/10.1002/cem.1389, 2011.

[11] an, S. J. and Yang, Q.: A Survey on Transfer Learning, IEEE Transactions on Knowledge and Data Engineering, 22, 1345–1359, https://doi.org/10.1109/TKDE.2009.191, 2010.