I have two datasets, each dataset is a pool of samples, and each sample contains multiple observations. Data-wise, we can represent one sample as a vector of floats, and a dataset as a matrix where each row is a sample.
I would like to find a transform to project the first dataset into the "space" of the second dataset. I think I am looking for a way to estimate and then project an empirical distribution into another empirical distribution. There might be a dependency between samples of one dataset, but not between datasets.
Bonus reward if the answer also suggests a way to account for the number of samples (because there is a big dataset and the other one is much smaller, thus a distribution estimation will likely be much more approximate for the latter).
Clarification: I have (and can't get) no apriori on the underlying distribution of any of the datasets. They are generated by a natural phenomenon, and no theory model correctly these distributions yet.
/EDIT: I will try to clarify with a concrete example close to what I am working on.
Let's say I am working on a machine learning application to automatically recognize objects shapes in a scanned image. To make it simpler, let's say these images are only in grayscale (no color, only pixel intensity remains).
The problem I am facing is that I have images coming from several different scanners, and each scanner has a different "intensity space", so that it is difficult to transpose a machine learning model learnt on one scanner to the next scanner, because of the scanner-induced variability in intensity.
More formally, using pseudo-Python code, my data and code would look like this:
# Images are represented as a row vector, each value is one pixel intensity # They are the samples scanner1_im1 = [1, 255, 3, 73, ...] scanner1_im2 = [4, 83, 2, 190, ...] scanner2_im1 = [8, 29, 1, ...] scanner2_im2 = [37, 29, 4, ...] # We can constitute a pool of image samples for each scanner, which I call a dataset scanner1_images = [scanner1_im1, scanner1_im2] scanner2_images = [scanner2_im1, scanner2_im2] # TODO: normalization step, where the scanner2_images gets normalized/projected into scanner1_images space, or whatever technique that would allow to represent both datasets in an approximately similar space scanner2_images_norm = norm_project_images(scanner2_images, scanner1_images) # Machine learning modeling on normalized images ml.learn(scanner1_images, scanner2_images_norm)
What statistical/probabilistic techniques may I use to project all scanners/datasets into a similar space of intensities (and thus reduce inter-scanner variability while retaining inter-images/intra-scanner variability as much as possible)?