0
$\begingroup$

Is there a variant of Feed Forward networks that admit co-variance between features in the input vector?

For instance, with binary input vectors of size 6 like v = [0 1 1 0 0 1] Suppose we know that v[1] and v[2] are strongly co-variant. I believe this to be deemed "intra-layer communication" in the DL literature. How would I encode this information a priori into the model? If there is no way, I'm interested to learn of NN-based techniques that would accept this input. Thanks.

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

I don't know of any such technique to encode this prior, but also, I want to claim that it isn't necessary. In linear regression, a high amount of covariance between input features is sometimes indicative of multi-collinearity, which can cause numerical issues (especially when inverting matrices), but this problem doesn't carry over to neural networks, because they are trained in an entirely different way. Other than that, there really is no need to encode this information into the model -- it wouldn't help performance.

If you really wanted to, I suppose you could apply PCA to the inputs before sending them through the network, which would severely reduce the amount of covariance between input features. However, in the literature, this rarely helps the performance of a model.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ I don't think it's true that multicollinearity affects NNs differently than linear regression because of model structure or training procedure. Linear regression can be trained the same way as NNs (e.g. using gradient descent), and is actually equivalent to a single layer NN with linear activation function. Rather, multicollinearity can present problems when we care about identifying 'true' parameter values (i.e. inference), but doesn't affect predictions or how well they fit the data. $\endgroup$
    – user20160
    Commented Aug 5, 2017 at 0:46
  • 1
    $\begingroup$ (continued) Multicollinearity comes into play differently for these models because we tend to care about inference in the context of linear regression, and about prediction (but not inference) in the context of NNs. $\endgroup$
    – user20160
    Commented Aug 5, 2017 at 0:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.