# How to extract relative importance of features from a tensorflow DNNRegressor model?

I followed these two posts to understand about restoring a saved model and then extracting variables from it:

But now I am unable to understand as to what do those variables mean exactly and how to understand the relative importance given by the model to the features in the learning process? Below is the output of tf.train.list_variables and tf.train.load_variables (respectively) from my code:

>> tf.train.list_variables('./dnn_fe_trial1/model.ckpt-1')

            [('dnn/hiddenlayer_0/biases', [50]),
('dnn/hiddenlayer_0/weights', [61, 50]),
('dnn/input_from_feature_columns/brand_embedding/weights', [10000, 16]),
('dnn/input_from_feature_columns/city_embedding/weights', [12, 3]),
('dnn/input_from_feature_columns/dow_embedding/weights', [7, 3]),
('dnn/input_from_feature_columns/l_cat_embedding/weights', [11, 3]),
('dnn/input_from_feature_columns/product_id_embedding/weights', [10000, 16]),
('dnn/input_from_feature_columns/type_id_embedding/weights', [10000, 16]),
('dnn/logits/biases', [1]),
('dnn/logits/weights', [50, 1]),
('global_step', [])]


>> tf.train.load_variable('./dnn_fe_trial1/model.ckpt-1','dnn/hiddenlayer_0/weights')

            array([[ 0.14350541,  0.18532775, -0.03176343, ..., -0.07279533,
-0.08580479, -0.07619692],
[ 0.16894072, -0.10593006,  0.06088932, ..., -0.01411209,
-0.26995516,  0.15667924],
[-0.10020741, -0.03164399, -0.14427225, ..., -0.02787848,
-0.15646952, -0.1361219 ],
...,
[ 0.15014522,  0.15378515, -0.05414914, ..., -0.16788298,
-0.14711154, -0.226382  ],
[-0.16823539,  0.2009476 , -0.271177  , ..., -0.10694946,
-0.22870012, -0.13458726],
[-0.13175508,  0.15535942, -0.18468232, ..., -0.1362714 ,
-0.27476427, -0.21606216]], dtype=float32)

• Can those who down vote this question tell us why? – quintumnia Apr 15 '18 at 8:33

You cannot see the relative importance of (input) features in your NN from just looking at its parameters.

Estimating the importance of features is a branch of research in itself. It is called Sensitivity Analysis.

In the case of neural network models, a lot of papers recently introduced tools to do (most of the time) local Sensitivity Analysis to understand the importance of each part of the input on the output.

Among them, one could cite the widely used LIME (Ribeiro et al., 2016), and the SHAP values (Lundberg et al., 2017) which are an improvement over the LIME method.

The importance of a feature-variable depends on the distributiuon of all the other feature-variables used, for the classification problem at hand.

Generally, when changing a feature-value $x_i$ can cause the pattern or feature vector ${\bf {\it x}}$ to become assigned a different class label, then that feature-value is of importance: Feature-value $x_i$ is said to have potential influence for the classification of feature vector ${\bf {\it x}}$. Feature-values that have potential influence for a large number of (different) pattern vectors, they are important ones for the classification task at hand.

A wrapper-approach to feature assessment involves removing each feature-variable, one-by-one, and compute the resulting decrease in classification performance. Hence, with $n$ feature variables, you need to train $n$ different classifiers, each with one less feature (basically, a 'leave-one-out' approach).

See the analysis of this problem in: [M. Egmont-Petersen, J.L. Talmon, A. Hasman, A.W. Ambergen. "Assessing the importance of features for multi-layer perceptrons," Neural Networks, Vol. 11, No. 4, pp. 623-635, 1998].