Multivariate sampling bias correction for data analysis I have a dataset of properties for 20,000 samples and 200 of these samples have a categorical label (A, B or C); and I'm trying to assess the relationships between the properties and the labels. I am doing this with Chi2 and Kendal's Tau tests for the discrete and continuous properties respectively. However, there is a significant sampling bias i.e. the 200 samples which have been assigned labels are not representative of the underlying population as assigning a label requires the sample to be tested and there are various factors influencing the decision to test a sample.
I have two questions:

*

*Does this sampling bias invalidate the Chi2 and Kendal Tau tests of feature importance?

*If so, are there methods for correcting this sampling bias? The underlying distributions are known for all the properties as the unlabelled + labelled data is the entire population, and I feel like there should be some way of using this knowledge to correct the bias in the labelled data but I can't seem to find any appropriate methods.

Edit: The method described in "Huang, Jiayuan, et al. "Correcting sample selection bias by unlabeled data." Advances in neural information processing systems 19 (2006)." appears to be one way of correcting this bias; though ideally there would be a robust implementation of this somewhere, which I am struggling to find.
 A: Upfront, a few words about how to find the approach that might be most beneficial to you:
First point:: In general, if you fit a predictor (aka "model") for your labels, you can presume that, in most of the cases, the independent variables (aka properties, features, ...) that are used by your predictor are also those that your labels depend on, otherwise they wouldn't be contained in the predictor.
Indeed, many fitting algorithms, like e.g. lm() in R, even give you, for each fitted parameter, a p-value for the significance of this parameter. And parameters are often linked to independent variables, like e.g. in linear models:
$$
y = p_1 x_1 + p_2 x_2 + \ldots + p_m x_m + b,
$$
where $y$ is the value you want to predict, the $x_i$ are your independent variables, and $p_i$ is the parameter "linked" to $x_i$. Then the significance of the fitted parameter can be understood as the significance of the belonging independent variable.
Having said that, there are lots of more complex models out there, like e.g. deep neural networks, GBMs, or random forest, where it is pretty much impossible to "see" the relevance of an independent variable just by looking at the model.
Second point: The problem of finding the most important variables, i.e. those that are most relevant for predicting, has been addressed by many papers and there are lots of approaches to tackle this. Ideally, you would try each of them and then pick the best for your situation, but that is not doable, since there are too many of them. For a shortcut, I would suggest, rather than using independence tests like Chi2 and Kendal's Tau, to have a look at random forest or mutual information based methods, which also have ready to use implementations, e.g. here and here.
Third point: Kendall's tau test is only working if both of the variables you want to compare are ordinal, i.e. can be ordered. So, unless you have an ordering for your three labels A, B, and C, you cannot use Kendall's tau test.
Note that, if you choose an ordering for your labels, it does have to make sense: E.g., if you choose A < B < C, then you are implying that for instance A and B are more similar than A and C, and this implication is relevant. For example, the labels A="red", B="green", C="blue" are not ordinal, and neither are A="house", B="car", c="tree". On the other hand, the five choices on a questionnaire ranging from "I completely disagree" all the way to "I completely agree" would be an example of ordinal labels.
Now, let's get to answering your two questions:
Question One: Yes, selection bias can alter your results of independence tests or other methods of detecting feature importance.
Question Two: There are methods for detecting whether and how much selection bias is influencing your results, at least to a certain extent, and how you could recover from that. If you want to pursue this road, you might want to start with this paper:

*

*Bareinboim, Elias, Jin Tian, and Judea Pearl. "Recovering from selection bias in causal and statistical inference." Proceedings of the AAAI Conference on Artificial Intelligence. Vol. 28. No. 1. 2014.

I could not find any pertinent implementations.
The paper you mention, "Huang, Jiayuan, et al. "Correcting sample selection bias by unlabeled data.", is concerned with one particular type of selection bias, which is arguably the most common one, namely "covariate shift".
Basically, covariate shift means, that the selection process creates a bias on the choice of your samples, but that it does not change the probability of the label given the sample. I don't know the details of your situation, but from what you describe, I would think that covariate shift is the selection bias you experience.
Unfortunately, I couldn't find an implementation for this method either.
So what should you do?
Given the bleak outlook concerning usable implementations for the methods described above, I would first try to find a way to remove the selection bias from the experiments. If this is not possible, see whether you can figure out yourself how the selection mechanism is biasing your data. Often, expert knowledge and common sense can get you quite far. Third, use the results as they are and report them always with the additional information about the selection bias. This still contains information which might be sufficient for some purposes.
A: According to [1], $P(s=1|z)$ --- the probability that an instance is included in the sample given its feature values --- can be used to weight the biased data.
Fortunately in my case, the factors which influence the decision to test a sample are discrete variables. This means that, again following the suggestion in [1], there is a simple method to calculate $P(s=1|z)$.
Let $X$ be the input/predictor space and $Y$ be the label space and $z = (x,y)$ where $x\in X$ and $y\in Y$. If we assume the sampling probability is independent of the labels we can calculate $P(s=1|z)$ as $m_{x}/n_{x}$ where $m_{x}$ is the number of times $x$ appears in the biased sample and $n_{x}$ is the number of times $x$ appears in the underlying population/unbiased sample (in my case Labelled + Unlabelled Data).
[1] Cortes, Corinna, Mehryar Mohri, Michael Riley, and Afshin Rostamizadeh. ‘Sample Selection Bias Correction Theory’. In Algorithmic Learning Theory, edited by Yoav Freund, László Györfi, György Turán, and Thomas Zeugmann, 5254:38–53. Lecture Notes in Computer Science. Berlin, Heidelberg: Springer Berlin Heidelberg, 2008. https://doi.org/10.1007/978-3-540-87987-9_8.
Something like the following in Python will add a weight column to a dataframe
import pandas as pd
from typing import List


def append_weight_column(biased_data: pd.DataFrame, biasing_vars: List[str], all_data: pd.DataFrame) -> pd.DataFrame:
    # Get different combinations of biasing variables and how often they occur in biased data and all data
    biased_data_combinations = biased_data.groupby(biasing_vars, as_index=False, dropna=False).size()
    all_data_combinations = all_data.groupby(biasing_vars, as_index=False, dropna=False).size()

    # Merge on these biasing variables to add n and m columns
    m = biased_data.merge(all_data_combinations, on=biasing_vars, how='left')
    m.rename(columns={'size': 'n'}, inplace=True)
    out = m.merge(biased_data_combinations, on=biasing_vars, how='left')
    out.rename(columns={'size': 'm'}, inplace=True)

    out['weight'] = out['n'] / out['m']

    out.drop(columns=['m', 'n'], inplace=True)
    return out

