# Does removing a feature with significantly large weight make the prediction worse?

Suppose my model have a feature with a significantly large weight. If I remove the feature, will my prediction get worse?

I think yes, because a large weight indicates that a feature is important to our prediction. Thus, if we remove it from our training model, of course our prediction will get worse.

Not sure if my reasoning is correct. I'd appreciate any help! Thanks

• A large weight does not imply more importance. Try in a linear regression to use two times the value of a feature instead the original values of the feature and observe the change in the coefficients. Commented May 12, 2020 at 8:01

You are right that deleting a feature which has a large weight will cause a decrease in performance. The weight does signify the importance of that feature. A highly positive weight signifies a strong positive correlation between the variables.

To understand this better, a simple equation of a neural network is given by,

$$Z=x1w1+x2w2+x3w3$$

where w1, w2 and w3 are the weights, x1, x2 and x3 are the features and Z is the output.

Higher the weight of one feature, more the output is influenced by it. This holds true when you use an activation function like sigmoid as well.

You can also check this paper out - http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.45.9756&rep=rep1&type=pdf. It has a few points that you might find useful.

• Curious why this answer got downvoted. Generally, if a feature has more importance compared to other features and the model you have is dense, with sufficient training, your model will give it more importance by optimizing weight matrices to account for that because we have partial derivatives in back propagation which calculate change in each connection, so it learns to give more importance to that feature on itself.
– Anuj
Commented May 12, 2020 at 12:55