How to check for "statistical significance" of categorical feature in black box models Let's say we have a categorical feature $X_i$ and we have build a black-box classification model like xgboost with $X_i$ as one of many predictors. We'd like to ask a question: does $X_i$ affects the overall prediction and, if so, how much? 
In particular $X_i$ could be:


*

*a dichotomous variable

*a n-level variable where we are interested in the potential difference between two particular levels


In white-box models like linear regression we have tests to obtain statistical significance. But can we obtain statistical-significance-alike with black box models? Does any tool from explainable artifficial intelligence field is applicable to that? Or would it be better to just perform standard t-test on the output probabilities predictions?
 A: There is a simple, but computationally expensive solution here: permutation tests.


*

*Evaluate your classifier, with $X_i$ excluded, on your validation data and save the score, $s$.

*Shuffle $X_i$ to produce a surrogate variable $\bar X_i$.

*Fit your classifier with $\bar X_i$ included as a predictor, evaluate it on the validation data to obtain the score  $\hat s$, and the improvement $\bar \delta = \hat s - s$ in out-of-sample accuracy.


*

*Repeat this $n$ times to obtain a distribution of $\bar \delta$ values. This is the expected distribution of improvements under the null hypothesis that the predictor is no use.


*Calculate the actual improvement $\delta$ obtained using your original variable, $X_i$. 


*

*Compare $\delta$ to the distribution of $\bar \delta$ scores. If $\delta$ exceeds the 95th percentile of $\bar \delta$ (or 97.5th percentile for a two-tailed test), you can reject the null hypothesis at $p < .05$.


A: In most cases, if we are building a black box model, we do not care the variable importance too much. Because we can have thousands or even million features as input. Models like gradient boosting on trees can automatically select important features.
If you really want to know variable importance (not statistical significance), you may check following links.
http://docs.h2o.ai/h2o/latest-stable/h2o-docs/variable-importance.html
A: Black-box predictions can be estimated with regression analysis
Suppose that your black-box model is predicting a response variable $y_i$ given the variable $x_i$ and another set of variables $\mathbf{z}_i$.  Denote the unknown prediction function by $g$ and observe that the prediction is:
$$\hat{y}_i = g(x_i, \mathbf{z}_i).$$
For simplicity, let's suppose that the variable of interest is a categorical variable with a finite number of states, and denote the allowable states as $x_i = 1,...,m$.  Then we are effectively estimating a set of $n$ functions $g_1,...,g_m$ that apply for each of the allowable values of the variable of interest.  Writing $g_k(\mathbf{z}_i) \equiv g(k, \mathbf{z}_i)$ and using these functions we can write our black-box model as:
$$\hat{y}_i = g_{x_i}(\mathbf{z}_i).$$
Now, although the form of the functions $g_1,...,g_m$ is unknown (since the model is a black-box), we can generate as many predictions as we want.  Thus, we can estimate each of these functions by standard regression methods, with any number of data points we wish to generate.  The standard approach here would be to stipulate a broad functional form for the function $g$ with a number of unknown parameters, and with interaction terms for $x_i$ and each variable in $\mathbf{z}_i$.  Then you would generate predictions to use as data, and use this to estimate the parameters using standard regression methods, which gives you a regression estimate of the function $g$.  Since you can generate as many predictions as you want, this is effectively a regression problem with an unlimited amount of data.
Once you have estimated the functions $g$, you can apply standard statistical tests for whether or not the categorical variable $x_i$ actually has any effect in the function.  Ordinarily you would compare a regression model with this variable in it to an equivalent nested model with the variable excluded and do a comparison with a likelihood-ratio test.  Ultimately you will obtain a statistical test for the effect of the variable $x_i$ in the prediciton method, along with an estimate of the prediction function.  If you generate a very large number of predictions then you should be able to get quite a good estimate of the function.
