Permutation of input features of SVM, simple logistic and KNN classifiers?

Question

I have come across a journal article, with an impact factor higher than 2.8, in which a very strange training procedure was performed. Since I consider myself a beginner, and since the article is published in what I presume to be a good journal, I questioned my assessment that the procedure performed is indeed wrong or strange.

The article tries to solve a [single class] classification problem for which there are less than 10 training instances. Each instance consists from 4 consecutive parts (e.g. A, B, C and D). So, the training dataset is in the format:

A_1  B_1  C_1  D_1
A_2  B_2  C_2  D_2
...
A_10 B_10 C_10 D_10

The strange thing is, in order to increase the number of the training instances, the order of the parts is permuted so each of the training instances above will produce 24 training instances (i.e.4!). One of the instances produced from the first instance, for example, will be:

C_1  D_1  A_1  B_1

This is strange and does not make sense at all to me since any feature interaction information will be lost. Am I right? If not, how would SVM, simple logistic and KNN cope with that?

Another issue I noticed is that the total number of input features is more than 1000 while they have at most 240training instance (after permuting the original training instances). I understand that it would not be a big problem for SVM, but would the other classifiers cope with such curse of dimensionality?

• UPDATE: this is the article in question: Source code and design conformance, design pattern detection from source code by classification approach.

Answer 1

Permuting inputs is entirely unacceptable and meaningless (some more suitable but less appropriate terms could be used too). I would avoid whichever journal published this like the plague.

Are you sure about the impact factor? 2.8 is very high for pure machine learning (only a handful have this IF and none would publish rubbish like this). Unfortunately, I've seen several occasions where comparable nonsense got published in more applied journals where the machine learning aspect is not the main interest, e.g. in bioinformatics.

Papers with major machine learning errors can slip through the net in application-oriented venues because the reviewer crew often does not include machine learning experts, but rather physicians, biologists, ... Naturally, non-machine learning savvy reviewers focus on other aspects, which may well be good.

Answer 2

Although I have not read the paper, permuting the features is not a weird idea. In the case of classifying sets of objects where the order does not matter, this is actually desirable. For example, we want to classify a sports team like a tennis duo, there is no clear order between the two although you would want exactly the same output regardless of which player you put in first. Shuffling the inputs is a bit of a hacky way of solving this however, but you can still learn non-linear relations between the objects, depending on the model. However, building a model that knows that the features are permutation equivariant or invariant is a better way of solving this issue. Here are two papers that are about this topic using neural networks:

https://arxiv.org/abs/1612.04530

https://arxiv.org/pdf/1703.06114.pdf

Answer 3

There is lots of bad analysis in papers, it should not happen, but it is very common, so don't consider everything published right. However, it's also a good idea to assume that other people knows what they are doing.

You are right that permuting features is weird. There is no reason to do that. Features are features for a reason, you cannot permute 'number of eyes' with 'weight' or anything like this. Just no.

Permutation can be used. You might permute target labels as way of significance testing or you might permute order of your samples if it matters for your training algorithm (e.g SGD).

There is a way creating additional data, sometimes small amount of noise is added, or you are imputing new in a space between other samples as in SMOTE or ROSE, however, that will not help you if you don't have enough data. It might help your model to behave nicely, but it will not give you more training data in a sense that you can get more information out of your training set.

Yes, feature interaction is lost and those algorithms will not cope with that, because it just doesn't make sense. However, since you are probably using linear models, you are not really looking at feature interactions anyway, e.g. you cannot see XOR interaction.

They don't have 240 instances they have 10!!! Whatever you do with your data, it's part of your model/training algorithm. So you have 10 training instances and youa re trying to learn some structure from your 10 instances, if you create synthetic data, they wills till be created only based on information from your 10 training data.

There is big difference between 'not enough data' and 'course of dimensionality.' They often appear simultaneously, but they bring different problems. IF you don't have enough data, regardless of they dimensionality, you cannot do a proper holdout, therefore you cannot really check if you are overfitting. You might have model with good performance just by chance. It is therefore important to have model with as little things to tune as possible, because you don't have independent set for parameters tuning and every tuning is just opportunity for a bias. This is why you would stick to very simple (possibly linear) models, because you don't have to tune them as much, or at all.

Course of dimensionality is a different problem. You might have billion pictures of size bilion*bilion pixels. It's a huge dimensionality problem, but you can have 300 milion images as your hold out testing set, and you can therefore do any crazy modelling schemes you want without a fear of overfitting, peeking, double dipping, biasing... With high dimensional space, relative distance between your samples is approximately same, therefore KNN will fail, other approaches will depends on your data, but now you can use kernells and hidden neurons.

In high dimensional space with small sample sizes all linear classifiers works approximately the same. You might choose a classifier based on other desired properties, for example GPX if you want to have probabilistic prediction, SVM if you don't want to wait ages for GPC and so on.