1
$\begingroup$

Deep learning for 1D/2D inputs usually assume some sort of local connections, whether it is using local filters, or recurrent connections etc. What if for our problem we think a connection between neighboring values/pixels is as likely as one between those far away? Suppose we get a signal with time values (or an image with pixel locations) completely randomized. Obviously this increases the complexity of the problem etc.

What kind of network topology would work for this? Has anyone addressed a problem such as this? I'd still like to use things like pooling at some point to try to decrease the complexity.

$\endgroup$
  • $\begingroup$ You can easily compute crosscorrelation between far away pixels as long as you specify the length scales you want. This can get very expensive for many different lengths $\endgroup$ – Alex R. Feb 16 '17 at 23:30
2
$\begingroup$

For feedforward networks, all-to-all connectivity seems to be what you're asking about. This is the standard/traditional way to construct a feedforward network (and is still in wide use). Local connectivity with shared weights (i.e. convolutional networks) are a more recent development. The local connections in convnets impose the assumption that there is local structure to exploit, and the shared weights impose the assumption that this structure is translation invariant. In return, the number of parameters is drastically reduced, which can help prevent overfitting. If your data don't satisfy these assumptions and you don't otherwise have a way to constrain the network architecture, then all-to-all connectivity may be your best bet. As you mention, the increased complexity may necessitate other kinds of constraints (e.g. regularization) to reduce overfitting if the network is large relative to the amount of data you're able to train on.

Recurrent networks can be thought of as feedforward networks that are unrolled in time, where temporally adjacent layers are connected via a shared set of weights, and each layer has its own input/output. 'Temporally global' connections don't make much sense when picturing the network in its recurrent form. But, there are architectures (e.g. LSTMs) where certain units act as memory buffers, feeding their values forward through time. I suppose you could think of this as achieving a similar effect to a 'temporally global' connection.

It should be mentioned: just because connections are constrained to be local doesn't mean that a network can only learn local structure. For example, early layers in convnets do, indeed, learn local features (it would be impossible for them to learn global features). But, higher layers--with only local connections to their respective inputs--learn compositions of these features, which tend to be more global with respect to the input space. For example, in image processing convnets, lower level layers learn features like local edge detectors, and higher level layers learn global features/concepts like faces, cats, and dogs, which span many pixels in the input space. Conversely, even all-to-all connected networks tend to learn local edge detectors in the early layers. In the temporal domain, recurrent networks can learn long timescale patterns despite connections being 'temporally local', because the recurrent connections propagate information forward in time. However, architectures with special memory units have tended to be more successful in learning very long timescale dependencies because they're better able to cope with the vanishing gradient problem.

Regarding your question about randomly permuting the input: If the input is something like images, then a feedforward network with all-to-all connectivity could work, and scrambling the input would have no effect compared to training the same network on the original input. In fact, early layers might very well learn local features in the original image space (which you could see if you unscrambled them). If the input is a time series, a recurrent network no longer makes sense because the temporal information has been destroyed. In this case, you might be able to treat individual sequences as input vectors to a feedforward network with all-to-all connectivity.

$\endgroup$
  • $\begingroup$ +1 and thank you for taking the time. This was helpful to start thinking about. $\endgroup$ – highBandWidth Feb 17 '17 at 0:42
  • $\begingroup$ @user20160 Can you please expand on your statement about feed-forward and all-to-all connectivity? I'm interested in models with intra-layer information beyond adjacency (like pixels)... thanks $\endgroup$ – Brian Dolan Aug 2 '17 at 23:26
  • $\begingroup$ @BrianDolan Sure. Which statement did you have in mind and/or what were you wondering? $\endgroup$ – user20160 Aug 2 '17 at 23:42
  • $\begingroup$ @user20160 "For feedforward networks, all-to-all connectivity seems to be what you're asking about" I have features which are highly correlated and I'm looking for newer methods that capture the correlations a priori. It seems most DL techniques want to develop that structure rather than take as data. For instance, in feature vector [1 0 0 1 1 0] suppose elms 4 & 5 have strong covariance. How can I use feed-forward for that? Is this a new question...? $\endgroup$ – Brian Dolan Aug 3 '17 at 15:55
  • $\begingroup$ @BrianDolan Yes, it sounds like posting a new question would be best $\endgroup$ – user20160 Aug 3 '17 at 16:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.