I have always read the term interaction in the context of regression. Should we also consider interactions with different models e.g. knn or svm?
If there are $50$, $100$ or even more features and lets say $1000$ observations what is the usual way to find useful interactions? Try all combinations? Or use only combinations which makes sense?
Interactions are needed explicitly in regression models because the formula does not include any interactions per se. More precisely, a regression model will always be linear in its input, whereas an interaction $X_i * X_j$ is a nonlinear combination of the features.
The simplest way to see this is through the XOR-Problem, a regression model without any interactions cannot solve this, as it requires a nonlinear combination.
KNNs and SVMs on the other hand (and many other models also) are universal function approximators. This means, that they cannot only combine their inputs in a linear fashion, but also in any possible non-linear way. That is given enough layers or a suitable kernel, they can basically "create" their own interactions, exactly as they need them. If you know or expect specific interactions to be important, though, you can still use them as an input to guide the models in the right direction.
Similarly, tree-based models can be interpreted as only consisting of interactions. Basically, a split in a tree-based model creates a specific interaction with all previous variables.
So for deciding which interactions to use, for sufficiently "high-power" models (i.e. those which are universal function approximators), you don't need them and you can let the model do its own magic. For other models it depends. There are some techniques available to guide the decision, like CHAID or step-wise regression. CHAID also works with a large number of features, for step-wise regression it may get lost in the number of possible interactions. Given that if you have $N$ features, there are $2^N$ possible interactions (counting not only two-way but also higher order interactions).
No.
In fact, you can think SVM with polynomial kernel is adding all the (high order) interactions between all features. For example, if we have two features $(x_1,x_2)$, SVM with 2nd order polynomial is doing $(x_1^2,x_2^2,x_1x_2)$.
SVM is called Kernel Trick, because it is implicitly doing polynomial basis expansion with a lot less computational complexity. Think about 10th order polynomial expansion on 10 features, manually expand it will have $10^{10}$ columns. But using kernel trick, we can easily do it.
So, not only interaction has been widely used in other models. In adding to interaction, other models trying to more with feature engineering. Instead of multiplication of two columns, more complicated features are derived.
Interactions that improve adjusted R-squared, BIC for likelihood regression (alternatively AICc and others), VIF, and the F-statistic of ANOVA, the latter without individual parameters that are judged non-contributory using their partial probabilities.
Also very important, but not asked, is that reparameterization can markedly improve both the effect of individual variables and their interactions. However, BIC, AIC, and other likelihood quality measurements are not valid for comparing different repareterizations leaving adjusted R-squared, VIF, and the F-statistic of ANOVA for such purposes.
