0
$\begingroup$

Suppose my model have a feature with a significantly large weight. If I remove the feature, will my prediction get worse?

I think yes, because a large weight indicates that a feature is important to our prediction. Thus, if we remove it from our training model, of course our prediction will get worse.

Not sure if my reasoning is correct. I'd appreciate any help! Thanks

Improve this question
$\endgroup$
1
  • $\begingroup$ A large weight does not imply more importance. Try in a linear regression to use two times the value of a feature instead the original values of the feature and observe the change in the coefficients. $\endgroup$
    – javierazcoiti
    Commented May 12, 2020 at 8:01

1 Answer 1

Reset to default
-1
$\begingroup$

You are right that deleting a feature which has a large weight will cause a decrease in performance. The weight does signify the importance of that feature. A highly positive weight signifies a strong positive correlation between the variables.

To understand this better, a simple equation of a neural network is given by,

$$Z=x1w1+x2w2+x3w3$$

where w1, w2 and w3 are the weights, x1, x2 and x3 are the features and Z is the output.

Higher the weight of one feature, more the output is influenced by it. This holds true when you use an activation function like sigmoid as well.

You can also check this paper out - http://citeseerx.ist.psu.edu/viewdoc/download?doi=10.1.1.45.9756&rep=rep1&type=pdf. It has a few points that you might find useful.

Improve this answer
$\endgroup$
1
  • $\begingroup$ Curious why this answer got downvoted. Generally, if a feature has more importance compared to other features and the model you have is dense, with sufficient training, your model will give it more importance by optimizing weight matrices to account for that because we have partial derivatives in back propagation which calculate change in each connection, so it learns to give more importance to that feature on itself. $\endgroup$
    – Anuj
    Commented May 12, 2020 at 12:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.