3
$\begingroup$

I'm relatively new to machine learning and I was wondering if I can use the correlation between features and the target value to manipulate the features.

i'm trying to do some manual feature construction by looking at correlation of each feature with target value, like it is used for feature selection but here i want to use this information to create a new feature, for example we have two features 'a' and 'b', by looking at the correlation we see that their correlation is a = 0.1 and b = 0.3, now am i allowed to give the something like a score like a = 1 and b = 2 and then create a new feature c = a+b.

my concern is that i'm looking at the target value to do this. Is it valid? is it considered falling in the trap of looking at the target value? if it is valid it should be done only on the training set, right?

$\endgroup$
1
$\begingroup$

In general, using target values isn't a very sound procedure. I'm guessing you'll have to rely on an unsupervised method to generate target values for your test data. So you can imagine that your solution will depend on the strength of your unsupervised method, which is obviously sub-par, otherwise you would have just used the unsupervised method in the first place.

However, in practice, it's something that people do all the time. For example, speech recognition strongly relies on an estimate of the boundaries of phonetic units for its classification. The boundaries are then updated and fed back into the classifier for another iteration. This procedure usually converges to something.

In conclusion, it's alright if you decide to incorporate target values in your features. However, be aware that your system will need target values in the test stage. In that case you'll probably end up with an iterative procedure of generating targets and updating them to get "better" target values.

$\endgroup$
  • 1
    $\begingroup$ no it is in fact a supervised method. maybe i didn't ask my question properly so i edited. my problem is exactly this sentence : ' In general, using target values isn't a very sound procedure' although i'm not actually using the target values itself just its correlation with features. $\endgroup$ – john d Jul 5 '18 at 7:44
  • 1
    $\begingroup$ If by "their correlation is a=0.1 and b-0.3" you mean, their correlation with the target is 0.1 and 0.3, my point still stands. Say you've trained your system with these features. How is the client (or user) going to be able to generate c to apply the model? Remember, in reality we don't have access to the target. $\endgroup$ – idnavid Jul 5 '18 at 22:54
  • $\begingroup$ yes but we do have 'a' and 'b' which are our binary features, and we know that for training the system we have assigned a=1 and b=2, so can't we just write an algorithm for the client to generate c by looking at features a and b? sorry if i'm misunderstanding something obvious $\endgroup$ – john d Jul 6 '18 at 7:13
  • 1
    $\begingroup$ yes exactly, a and b originally are binary (0 or 1), and i want to give them a higher value based on their importance in regards to target values, and then add them to transform them to only one variable, where if originally 'a' and 'b' both equal to 1 than 'c=3' and if only a=1 then c=1 and if only b=1 than c=2. so i really want to look at the correlation with target value when training the system and then for user to use it the default value of a and b in system will be 1 and 2. $\endgroup$ – john d Jul 9 '18 at 9:34
  • 1
    $\begingroup$ Thanks for clarifying. In the simple case that we've considered here, I don't see any problem of using correlation. Essentially, you're using your expert knowledge to weight your features. However, keep in mind, that approaches like this are generally not desirable, since when you increase the number of features (instead of a, b imagine a1, a2, ..., a1000) it gets out of hand pretty quickly. $\endgroup$ – idnavid Jul 9 '18 at 10:00
1
$\begingroup$

I think that you're thinking about a clustering-type process, where you extract important subsets of the input space. Usually this is represented with finding shapes or patterns in the data to exploit. These patterns might not always emerge just looking at descriptive statistics. If you want to cluster the input based on purely observed inputs, the principle components analysis procedure can help you construct new variables that are linear combinations of the original input dimensions. If you want to incorporate the output in the clustering approach, you could use an active subspace linear or quadratic model to simillarly identify important directions in your input space.

$\endgroup$
  • $\begingroup$ thank you for your answer, i'm actually saving the PCA as the last resort. sorry i edited my question for more clarity, i appreciated if you could give me an answer based on that. is active subspace a good approach for dimensionality reduction of binary variables? $\endgroup$ – john d Jul 5 '18 at 7:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.