How to "highlight" an input feature of an artificial neural network?

Question

I'm trying to solve a binary classification problem by using an artificial neural network implemented in Torch. My neural network has 82 input features (=neurons).

After implementing a plain version that gives to the all the 82 input neurons the same importance, I need to design a new version of my algorithm in which a user can highlight / give more importance to one single feature among all the 82.

How could I do this in statistical tems? How could I edit my algorithm to give more importance to a single feature?

Answer 1

Try the wide & deep network architecture? Directly link the "important" features with the output neuron.

[1]. https://arxiv.org/abs/1606.07792

Answer 2

{1} explored one way to take prior knowledge on features into account when training a neural network. Abstract:

Different features have different relevance to a particular learning problem. Some features are less relevant; while some very important. Instead of selecting the most relevant features using feature selection, an algorithm can be given this knowledge of feature importance based on expert opinion or prior learning. Learning can be faster and more accurate if learners take feature importance into account. Correlation aided Neural Networks (CANN) is presented which is such an algorithm. CANN treats feature importance as the correlation coefficient between the target attribute and the features. CANN modifies normal feedforward Neural Network to fit both correlation values and training data. Empirical evaluation shows that CANN is faster and more accurate than applying the two step approach of feature selection and then using normal learning algorithms.

I didn't read the paper carefully, I am unsure how sound it is, and I'd be quite cautious. The same author published a few other papers on the same topic, e.g. {2}. Personally I rely on backpropagation to do the job.

Perhaps another way could be to change the weigh update rule and/or weight initialization rule for this feature, so as to bias the weights connected to your important feature to have an absolute value larger than the other weights connected to the other features.

A last idea would be to connect your most important feature to layers other than the first layer.

---

• {1} Iqbal, Ridwan Al. "Using Feature Weights to Improve Performance of Neural Networks." arXiv preprint arXiv:1101.4918 (2011). https://scholar.google.com/scholar?cluster=15075021269543299652&hl=en&as_sdt=0,22 ; http://arxiv.org/abs/1101.4918
• {2} Al Iqbal, Ridwan. "Empirical learning aided by weak domain knowledge in the form of feature importance." In Multimedia and Signal Processing (CMSP), 2011 International Conference on, vol. 1, pp. 126-130. IEEE, 2011. https://scholar.google.com/scholar?cluster=13856845400679996300&hl=en&as_sdt=0,22 ; http://arxiv.org/abs/1005.5556

Answer 3

In neural networks, the "importance" of each signal is established during the learning phase. It comes hard coded in the model, rather than expressed by a nice numeric parameter. I'm afraid you may not be able to manually alter the importance of a feature.

One way of forcing it, if you are using dropout, is to avoid it on the signal the user judges "important". Other than this, I really can't see how to force it. Please, notice that I'm using "forcing" because what you want to do is ... counter-intuitive for almost any machine learning classifier.

Answer 4

You might consider interpreting your neural network as a probabilistic graphical model. From "An Introduction to Variational Methods for Graphical Models", Jordan et al:

Neural networks are layered graphs endowed with a nonlinear "activation" function at each node (see figure 5). Let us consider activation functions that are bounded between zero and one, such as those obtained from the logistic function $f(z) = 1/(1 + e^{−z})$. We can treat such a neural network as a graphical model by associating a binary variable $S_i$ with each node and interpreting the activation of the node as the probability that the associated binary variable takes one of its two values. [...] The advantages of treating a neural network in this manner include the ability to perform diagnostic calculations, to handle missing data, and to treat unsupervised learning on the same footing as supervised learning. Realizing these benefits, however, requires that the inference problem be solved in an efficient way.

Later portions of the paper discuss how to do this efficiently. It would seem you could "highlight" a feature by changing the prior placed on its associated parameters.