Suppose we have dataset $A$ with several categorical and numerical features: $A_{cat_1}$, $A_{cat_2}$, $\ldots;$ $A_{num_1}$, $A_{num_2}$, $\ldots;$

Also we have another dataset $B$ with the same features, but probably with less categories (i.e. unique values) in features.

We want to sample from dataset $B$ according to joint distribution of $A$. How can one do it?

I was thinking in the following direction:

  1. We can LabelEncode categorical features. After that we can use Kernel Density Estimation on both numerical and encoded categorical features. Is it the right way to estimate distribution of $A$ or maybe there is more correct procedure to estimate distribution in presence of categorical variables?
  2. After we obtain KDE of $A$, we need to sample from $B$ according to that distribution. Could you describe (or maybe provide some code) how can one do it?

I admit this path itself could not be the best solution, so I am also open to better suggestions or any sources.


So I think if I understand you correctly, what you are trying to do is to sample points from $B$ according to the distribution of $A$. In which case I would take a different approach to what you have laid out here.

First of all you talked about label encoding $A$ and then doing a KDE. I don't think this makes sense unless your categorical variables can really present something continuous ie colour $\in$ [red, orange, yellow, green] as opposed to pet $\in$ [cat, frog, horse]. The problem here is that if you label encode these and apply the KDE filter, unless you set a small width for the filter you will end up with a lot of probability distribution between theses discrete categories. You will also end up blurring your distributions for the other variables.

I think a better approach would be this:

  1. Randomly sample a datapoint $a$ from $A$
  2. Find the datapoint $b$ from $B$ which minimises some distance $D(a,b)$
  3. Repeat

Then all you need to do is define a distance function $D$. This could take into account both continuous values and the categorical values.

For categorical variables perhaps you decide the partial distance $d_{cat_1} = 0\textit{ if }(a_{cat_1}=b_{cat_1})\textit{ else } \infty$

and for continuous variables maybe you could use $d_{num_1} = (a_{num_1}-a_{num_2})/\sigma_{num_1}$ where $\sigma_{num_1}$ is the standard deviation of $[A_{num_1}, B_{num_1}]$ and so on.

Then you could set $D = \sum_x d_x^2$

Here is a minimum toy example in python

import matplotlib.pyplot as plt
A = np.random.normal(0,1,(1000,2))
B = np.random.uniform(-3,3,(1000,2))

def distance(a,b):
    return np.sum((a-b)**2)

def resample(A, B, N):
    newsample = np.zeros((N,2))
    for i in range(N):
        # step 1
        a = A[np.random.randint(0,len(A))]
        #step 2
        mindist = np.inf
        mindistind = None
        for j in range(len(B)):
            d = distance(a,B[j])
            if d < mindist:
                mindist = d
                mindistind = j
    return newsample

C = resample(A, B, 300)

fig, (ax1, ax2, ax3) = plt.subplots(3,1, sharex=True, sharey=True)
ax1.scatter(A[:,0], A[:,1], label='A')
ax2.scatter(B[:,0], B[:,1], label='B')
ax3.scatter(C[:,0], C[:,1], label='B from A')
[ax.legend() for ax in (ax1, ax2, ax3)]
  • $\begingroup$ Thank you for your answer ;) $\endgroup$ – xxxxx Feb 18 at 22:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.