1
$\begingroup$

So i know feature crossing is a way to transform the data such that it can be linearly separated, which makes it useful for things like classification. But in a DNN, activation functions replicate non-linearity as well. So is feature crossing really useful in Deep Learning?

EDIT: this question is not asking about feature crossing with traditional ml methods, like linear-regression or SVM. I am asking about deep neural networks, which automatically transform features and generate non-linearity

$\endgroup$
12
  • 3
    $\begingroup$ Please define "feature crossing". $\endgroup$
    – Galen
    Jul 27 at 18:40
  • $\begingroup$ Related: developers.google.com/machine-learning/crash-course/… $\endgroup$
    – Galen
    Jul 27 at 18:55
  • 4
    $\begingroup$ That's usually called an "interaction", never heard the term "crossing" on this context before $\endgroup$
    – Firebug
    Jul 27 at 19:04
  • 2
    $\begingroup$ Some examples of hard-to-learn problems for vanilla FFC NNs stats.stackexchange.com/questions/349155/… $\endgroup$
    – Sycorax
    Jul 27 at 19:41
  • 1
    $\begingroup$ As usual, if you expect a certain characteristic, it will be more efficient to use that expectation that to make your model discover it. Why make the model have to discover that $x_1 x_2$ matters when you can tell it? Likewise, why apply a Fourier transform (wavelet, whatever) to a time series if a neural network is a universal approximator? I say that this is because you can get this transform by doing it explicitly, rather than having to spend parameters (overfitting risk) to discover that such a transformation is useful. // I am a huge fan of that answer linked by Sycorax! $\endgroup$
    – Dave
    Jul 27 at 19:50
7
$\begingroup$

Quoting my other answer regarding feature engineering in general

So while in many cases you could expect from the algorithm to find the solution, alternatively, by feature engineering you could simplify the problem. Simple problems are easier and faster to solve, and need less complicated algorithms. Simple algorithms are often more robust, the results are often more interpretable, they are more scalable (less computational resources, time to train, etc.) and portable. [...]

Moreover, don't believe everything the machine learning marketers tell you. In most cases, the algorithms won't "learn by themselves". You usually have limited time, resources, computational power, and the data has usually the limited size and is noisy, neither of these helps.

Yes, deep learning models can learn feature crosses (aka interaction terms) alike features by themselves. However by providing them by yourself, you simplify the problem to be solved, so you can expect it to converge faster. The fact alone that neural networks can learn from nearly raw data does not mean that we should drop all attempts for feature engineering.

A similar argument can be made about unsupervised learning algorithms: if we can learn without labels, why bother? We bother because learning from labeled data is easier, faster, needs simpler algorithms, less data, it's easier to debug, it's less tricky to train since by providing the labels you point the algorithm into the desired direction. The same applies to feature engineering.

A simple example is provided in my other answer referred above, it's learning XOR function from the data. With feature cross, it can be solved using a trivial model, while without you would need to build a much more complicated model (e.g. multi-layer network).

$\endgroup$
1
  • 1
    $\begingroup$ (+1) I strongly agree on considering the limitations on resources. While there are certain mathematical results that hold under certain assumptions, in practice the best model is the one that works... best in practice. In other words, empirically testing what works well for a given use case is important. $\endgroup$
    – Galen
    Jul 27 at 20:45

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.